The Era of Fast and Inexpensive LLM is Underway and Microsoft is Joining the Dance!

The Era of Fast and Inexpensive LLM is Underway and Microsoft is Joining the Dance!

This current trend of focusing on developing capable but smaller LMs is not displeasing to me because although recently unveiled models like Falcon 180B are impressive, I doubt many companies in France have use cases that would justify the cost of deploying and maintaining these models in the cloud (you need 8 40GB A100 GPUs just to load the model).

On the other hand, I am much more stimulated by a model like Microsoft's recently published 1.3B parameter Phi 1.3B because I think that today models between 1 and 13B parameters meet most of the needs of companies.

Today, the war is more than ever to reduce inference costs.

Until now, to make models more compact, we could turn to small LLM and use fine-tuning to specialize them on a specific task, thus avoiding the use of larger generalist models.

We can also use "quantization", i.e. reduce the model weight size by reducing the precision of the calculations (for example from 32 bits to 16, 8 or even 4...).

(There are other techniques like knowledge distillation or low rank factorization but they are a bit less accessible.)

Another lever we can act on is the neural network architecture.

Transformers that emerged in 2017 are still the basis for today's LLM.

The Transformer architecture has a small flaw: its inference complexity is linear with sequence length, i.e. O(N), due in particular to its characteristic attention layers.

Yesterday, Microsoft (again) published an article to suggest an alternative to Transformers: the "RetNet" and the results are promising! The inference complexity is constant, i.e. O(1). In particular, inference would be 8.4x faster than with Transformers while saving 70% of GPU memory.

To get concrete, today to load a 7B LLM on your GPU you need 14GB of memory (with 16-bit precision).

The longer the text sequence to process, the more additional GPU your model will consume, up to 45GB of RAM in the example.

With RetNet we stay constant around 15GB.

I'm excited and at the same time part of me invites me to remain cautious, others have already tried to replace Transformers, without success.

Wait and see!