Yes I watched the same exact video yesterday. I feel we are still at the feet of the mountain in terms of what’s next in AI, plateau is still not in sight.
Yes I watched the same exact video yesterday. I feel we are still at the feet of the mountain in terms of what’s next in AI, plateau is still not in sight.
I’m not an expert by any means, just someone who is interested and reads AI news, but lately it seems like optimisation and efficiency work better than increasing parameters to improve performance of LLMs. And research is also clearly pointing at different architectures, other than transformers, to improve performance. I’d be surprised if GPT5 , which is 2-3 years away, will be just a mere development of GPT4, i.e. a LLM with many more parameters. These statements from Bill seem a little bit short sighted and contradictory to the general consensus.
I am also aware of the Dunning-Krueger effect and how it may be tricking me into thinking I somewhat understand things I have no idea of lol
The plateau may be close, and GPT5 may not be that huge step forward everyone expects, but this implies that GPT5 will not change architecture, which is highly unlikely. GPT5 is at least 2-3 years away, and the recent rumors about Q* show that research in AI is actively looking elsewhere to boost capabilities. I will be utterly surprised if GPT5 will use the same architecture if GPT4 and if it will actually be a minor step forward - I give it 5% probably to Bill’s prediction to be accurate.
I don’t think Bill is right on this one, LLM may have achieved a plateau on performance with current architecture, but research is all about optimisation and efficiency, not mere parameter increase. Here’s a good example:
https://venturebeat.com/ai/new-technique-can-accelerate-language-models-by-300x/
Remember GPT 5 is still at least 2-3 years away. Plenty of time.