4 Simple Tips for Improving RAG Performance
Here are four simple tips for improving the performance of your RAG (Retrieval Augmented Generation):
- Understand the RAG: The RAG combines a "Retrieval" step to find relevant documents for a user query, and a "LLM" step where those documents feed a language model to generate the final response.
- You can't improve what you don't measure: Use experiment tracking with tools like MLflow. Log everything necessary to reproduce the experience and analyze it afterwards. This includes scores, prompt templates, queries, retrieved documents, hyperparameters like chunk size, chunk overlap... but also matplotlib visualizations etc. This will avoid the "It was better before but I don't know what I changed". There are still a lot of projects navigating by sight.
- Evaluation of response quality: How to evaluate LLM responses? What metrics? This is still a complex area, and for good reason it is an active subject of research. Simple metrics like ROUGE can be used but they are limited because they are based solely on the number of common n-grams between the LLM prediction and the target. There is also BertScore to consider, or using another LLM to evaluate the final response. It's up to you to choose your method.
- Evaluation of Retrieval: What we can do much more easily, however, is measure the ability of retrieval to find relevant documents. Prepare an evaluation dataset with queries and expected relevant documents. The idea is then to compare the documents retrieved by the retrieval to the relevant documents. The metrics to use here are similar to those of recommendation systems: Hitrate, Precision@k, Recall@k, MAP.
- Query optimization: The assumption of retrieval is the semantic similarity between documents to be retrieved and the user query. This is a very strong assumption that is often inaccurate. To improve this, explore strategies such as query transformation. You can use reformulation via an LLM or go through more classic NLP techniques like NER for keyword extraction, it's up to you. Also consider specialized embedders that have been specifically trained on question-answer pairs. Also pay attention to some embedders that have a fairly low token limit (128 tokens for some Hugging Face models).