RAG with LLMs: Easy to POC, Hard to Iterate On.

The paradox of LLMs is that they have made POCs (Proofs of Concept) incredibly easy to create, but at the same time, they have made iterating on the product more difficult. This is particularly visible with RAG (Retrieval Augmented Generation) systems.

RAG systems are initially very simple systems; they require relatively little effort to build with frameworks/libraries like LangChain or LlamaIndex. This generic approach generally works very well on a limited test subset.

Then, over time, we must inevitably complexify the solution to improve its performance and create custom code blocks for the use case, while keeping production constraints in mind. We often find ourselves simultaneously optimizing 5 opposing axes:

1) The relevance of the system's response (effectively provides the expected answer).
2) The behavior of the LLM (tone, output format in JSON/XML, etc.).
3) The system's latency (time between the message and the response).
4) The cost of a system response.
5) The complexity of the solution.

Optimize the relevance of the response and, with the sensitivity of prompts and the system, you end up altering the behavior of the LLM.

Separate the optimization step for relevance and behavior into two distinct independent phases so that one does not influence the other, and you now end up with higher latency and cost.

Use a smaller and faster LLM, you improve latency but increase the complexity of the solution by having to incorporate more context and examples into the prompts.

The "one size fits all approach" works very well and very quickly at the POC stage. To take it a step further, more creativity, more customization, and more thought about the data and overall system design will be required.