RAG vs Finetuning - Complementary not Opposing

RAG and finetuning are often opposed: Should we fine-tune our LLM or do RAG? In reality, the approaches do not oppose each other. On the contrary, they complement each other.

RAG, which needs no introduction, allows querying an external knowledge base to enrich the context provided to the LLM. We thus add the relevant documents to the prompt. This approach is very interesting when dealing with evolving information because we update the database, not the model, which is much simpler and less expensive. The double advantage is that by constraining the model to rely on documents, we reduce its propensity to hallucinate.

The downside is that by adding both context, instructions, documents, thought processes (Chain of Thought), and finally few-shot examples to the prompt, the prompt becomes a whole little booklet.

Even though LLMs have increasingly impressive ingestion capacities (context window of 200k tokens for Claude 2 and 128k for GPT-4), they are designed such that they have attention loss when they have to "read novels". In particular, their focus is on the beginning and the end, with potentially lost information in the middle, hence the interest in staying concise (see the paper "Lost in the Middle").

Finetuning, on the other hand, allows adjusting the LLM by training it on your own dataset. This allows it to learn things specific to your field, jargon, tone, behavior, but also to align it with your values... This not only improves its relevance but also means that if you use RAG on top, there will be even less to make explicit in the prompt, thus giving the LLM the opportunity to focus on the key information.

A recent scientific publication has moreover quantified the performance gain of the finetuning + RAG method. (RAG vs Finetuning)

When possible, the best approach is to do both.