The future of generative AI will likely not be made of ever-larger models
(This article is a transcript in English of one of my french linkedIn posts)
The future of generative AI will likely not be made of ever-larger models.
In recent years, the dominant impression was that the only way to improve language model performance was to increase their number of parameters. This frantic "scale" race seems unsustainable in the long run.
Sam Altman himself stated last April: "Future progress won't simply come from bigger models." We have probably reached the end of the era of gigantic models.
There are several arguments that lead me to say we're entering an inflection point:
▶ We will hit two types of limits: costs and available data.
▶ There is currently a GPU shortage that prompts rethinking approaches.
▶ It does not align with history for cutting-edge AI to be the exclusive domain of a handful of players.
▶ Most current models are still highly suboptimal.
It is this last point I want to dig into here.
Recent work like Google DeepMind's "Chinchilla" research paper published in March 2022 shows many models are under-trained - meaning training them longer on more data could improve performance.
This also means smaller models trained on more data can reach the same performance level.
The "Chinchilla" researchers sought to determine the optimal parameter and data amounts for a given compute budget (FLOPS).
They conducted a vast study training 400 models with variations in size, training time, and data volume ingested.
One interesting paper result is it would take 20x more tokens than parameters for optimal results. Researchers built the "Chinchilla" model following this finding.
Chinchilla, with 70 billion parameters, outperforms GPT-3 which has 175 billion.
Chinchilla has fewer parameters but was trained on 1.4 trillion tokens, 20x the number of parameters, while GPT-3 was trained on 300 billion tokens, just 2x the parameters.
Meta's latest July Llama 2 model goes even further, with the 70B version trained on 2 trillion tokens - a ratio of 28.
Knowing this, there is an advantage to making smaller models, as they take up less storage space, require less GPU power for inference, are more easily deployable (and cheaper), and can be more accessible to the community.
I think in the future it won't stop there, we will certainly find even more optimized architectures. In my opinion, the future belongs to compact, efficient models.
It's only a matter of time before we can run highly capable LLM directly on our smartphones.
Sources: