Why bigger context windows won’t make RAG disappear.

We sometimes hear that RAG (Retrieval Augmented Generation) is destined to disappear because LLMs have an increasingly large context window and can therefore receive more and more words.

In other words, we'll soon be able to pass an entire database to the LLM in one prompt and let it figure it out to answer the question asked.

Personally, I don't think this is a good approach, and for several reasons:

The computational cost increases quadratically with the number of words. More simply put, the more words you send to an LLM, the more you tax the GPU memory.

If you host the LLM yourself on a GPU instance, the direct consequence is that you'll probably have to opt for a larger instance, so pay more.

If you go through an API, the consequence is also that it's more expensive because you pay per token. You'll also hit query rate limits faster.

With very long prompts, context loss or hallucinations are encouraged (see the "Lost in the Middle" publication).

And finally, I haven't been able to experiment with such solutions with better results than RAG.

From my point of view, it's more expensive, it takes longer, and it's generally not as good.

All of this reminds me a bit of deep learning a few years ago. The approach was to realize that we have mind-boggling data streams and extraordinary computing capabilities, so we might as well use them. Even if the data is not quality, the promise was that quantity would make up for it.

We’ve come back a bit from that narrative today by saying that we don't need that much data; it must, however, be qualitative.