The AI Engineering Quest

RAG with LLMs: Easy to POC, Hard to Iterate On.

2024-06-04T22:10:26+02:00

The paradox of LLMs is that they have made POCs (Proofs of Concept) incredibly easy to create, but at the same time, they have made iterating on the product more difficult. This is particularly visible with RAG (Retrieval Augmented Generation) systems.

RAG systems are initially very simple systems; they require relatively little effort to build with frameworks/libraries like LangChain or LlamaIndex. This generic approach generally works very well on a limited test subset.

Then, over time, we must inevitably complexify the solution to improve its performance and create custom code blocks for the use case, while keeping production constraints in mind. We often find ourselves simultaneously optimizing 5 opposing axes:

1) The relevance of the system's response (effectively provides the expected answer).
2) The behavior of the LLM (tone, output format in JSON/XML, etc.).
3) The system's latency (time between the message and the response).
4) The cost of a system response.
5) The complexity of the solution.

Optimize the relevance of the response and, with the sensitivity of prompts and the system, you end up altering the behavior of the LLM.

Separate the optimization step for relevance and behavior into two distinct independent phases so that one does not influence the other, and you now end up with higher latency and cost.

Use a smaller and faster LLM, you improve latency but increase the complexity of the solution by having to incorporate more context and examples into the prompts.

The "one size fits all approach" works very well and very quickly at the POC stage. To take it a step further, more creativity, more customization, and more thought about the data and overall system design will be required.

Google’s Python Style Guide for Data Scientists

2024-06-03T17:55:21+02:00

Most Data Scientists have learned to code well on the job. Indeed, higher education generally focuses more on theory (mathematics, statistics, etc.) rather than software engineering, deployment, etc.

I don't know if it's a good or bad choice, the attraction to theory is probably also a bit cultural, and it's also what has produced some exceptional profiles that shine internationally along the way.

In any case, for the rest, the mere mortals who have to improve the quality of their code on a daily basis, Google has made available their best practices and their development style in Python (apparently, it's also a subject for them). We're talking about style because it's primarily a matter of preferences, and also because Python has its own usages and "tacit conventions", we're talking about coding in a "Pythonic" way.

This is a useful document for anyone who wants to improve in this area, especially since the choices are argued. This is important because sometimes certain recommendations or "Craftsman" rules can seem a bit sectarian :)

Link to the document:
https://google.github.io/styleguide/pyguide.html

The Role of Language in Reasoning and Conceptualization

2024-06-02T17:51:41+02:00

Yann LeCun's recent statements about language not being necessary for reasoning are as fascinating as they are divisive. Among other things, he emphasizes that animals or children acquire basic knowledge through their senses. He adds that sight allows for collecting a wealth of information much faster than one could by reading a book. Language would ultimately be the cherry on top, but one would first need to build the cake.

However, even if we read fewer words per second than we see images, language seems to be very efficient in compressing concepts into relatively few symbols. Something that struck me when I started getting interested in Stable Diffusion and Imagen, the famous models that generate images in response to a prompt, is that language plays a predominant role.

Indeed, these models are composed of two sub-models: the first encodes the text of the prompt into embeddings, which are then passed to the second diffusion model (UNet) to generate an image. In Google's publication on Imagen, one can read that the researchers found, quite surprisingly, that increasing the number of parameters in the text encoder contributed much more to improving the quality of the generated image than increasing the number of parameters in the UNet.

From this empirical observation, one is tempted to say that even if language is not a priori necessary for reasoning, it still seems to simplify many things in the process of creation and conceptualization.

The boundaries between AI, philosophy, and neuroscience are sometimes thin, and these fields intersect and influence each other in fascinating ways.

Managing a Mainstream Open Source Library is Tough

2024-06-02T17:32:39+02:00

Managing a mainstream open-source library is a full-time job that requires implementing an efficient workflow to avoid being overwhelmed.

This is particularly noticeable with Langchain, where at certain times of the day, you refresh the Pull Requests page on GitHub, and there's almost a new one every 10 minutes.

While I'm not a fan of all Langchain's features, I'm quite admiring of the daily workload of the dev teams: nearly 10,000 Pull Requests processed to date for ~2,400 contributors. To handle this flow of code to review, they have implemented the following workflow:

To contribute, you must follow the standard "fork and pull request" procedure, meaning you cannot push directly to their repo; you must make a PR from a fork you have modified.

Before coding anything, you must install the project and its various dependencies with poetry, which will also create a virtual environment for you and centralize all project configurations in the pyproject.toml file.

If you implement a new feature, you will need to document it, add unit tests (and integration tests if necessary), and provide a Jupyter notebook to facilitate understanding and provide concrete usage examples.

Then your code will have to pass a battery of tests, including the following technologies:

pytest for unit tests / integration tests.

ruff, a super-fast linter / code formatter written in Rust.

mypy, which will ensure that all the typing of your functions is consistent.

codespell will search for any typos that may have crept into your comments / docstrings.

(Incidentally, it's interesting to see that they don't use GitHub pre-commit).

Makefile is used to automate the commands to be executed, so you can run "make test", "make format", "make lint", etc., locally to check your code.

Finally, when you push your PR, a GitHub Actions CI will trigger to perform the same checks as mentioned above, but additionally, it will ensure code compatibility with Python versions 3.8, 3.9, 3.10, and 3.11.

If your proposal passes all tests and is deemed worthy of interest, it will be reviewed by a team member and potentially merged.

This will undoubtedly put off more than one person, but the idea is to guarantee and maintain a high level of quality. This approach will allow filtering, simplify the review, and decrease the time spent on it.

There are undoubtedly some good practices to be gained.

Is There a Future for On-Device Open Source LLMs?

2024-03-12T22:56:10+01:00

One obstacle to using open source LLMs is the cost of hosting them on GPU instances in the cloud. But perhaps there is a future for on-device open source?

From what I've seen, the overwhelming majority of AI products using LLMs are based on API calls to services like OpenAI, Bedrock or Vertex AI. And that makes sense - it's quick to set up, cheaper to run, and there's no heavy infrastructure to manage.

In a previous post I made a distinction between use cases requiring an instant response (streaming) like chatbots, and those that can be handled in batches like annotation, proofreading, etc. In the latter case, you just rent a GPU instance in the cloud for the duration of the task, not 24/7. It's an inexpensive way to benefit from open source LLMs.

I imagine that in the future, a new trend may emerge for open source models - on-device. For example, for a corporate chatbot, instead of designing cloud infrastructure and a web UI for employees, the work would consist of directly (or partially) downloadable applications that run from laptops (also bringing technical and practical challenges).

When you think about it, LLMs are increasingly optimized and the chips embedded in computers more and more powerful. There are also massive investments happening in the space right now.

Obviously with all this, Apple MacBooks with their M-series chips are ahead and seem like natural candidates to run LLMs locally, even if AMD and Intel are working hard playing catch up.

I wonder if for some time now Apple has been secretly conceiving its own on-device LLM that will interact with the entire macOS ecosystem.

Why bigger context windows won’t make RAG disappear.

2024-03-11T22:53:37+01:00

We sometimes hear that RAG (Retrieval Augmented Generation) is destined to disappear because LLMs have an increasingly large context window and can therefore receive more and more words.

In other words, we'll soon be able to pass an entire database to the LLM in one prompt and let it figure it out to answer the question asked.

Personally, I don't think this is a good approach, and for several reasons:

The computational cost increases quadratically with the number of words. More simply put, the more words you send to an LLM, the more you tax the GPU memory.

If you host the LLM yourself on a GPU instance, the direct consequence is that you'll probably have to opt for a larger instance, so pay more.

If you go through an API, the consequence is also that it's more expensive because you pay per token. You'll also hit query rate limits faster.

With very long prompts, context loss or hallucinations are encouraged (see the "Lost in the Middle" publication).

And finally, I haven't been able to experiment with such solutions with better results than RAG.

From my point of view, it's more expensive, it takes longer, and it's generally not as good.

All of this reminds me a bit of deep learning a few years ago. The approach was to realize that we have mind-boggling data streams and extraordinary computing capabilities, so we might as well use them. Even if the data is not quality, the promise was that quantity would make up for it.

We’ve come back a bit from that narrative today by saying that we don't need that much data; it must, however, be qualitative.

Padding strategy matters when Fine tuning LLMs

2024-03-10T22:49:13+01:00

A poor padding strategy can disrupt an LLM after fine-tuning, and the reasons are quite subtle. This is especially true since there is no pad_token in the Llama2 or Mistral tokenizers.

A neural network is a sequence of matrix multiplications. When working with text and variable sequence lengths, padding is required to normalize them to a common length to allow matrix representation.

This padding simply consists of filling the sequences with a special token until they reach the length of the longest sequence in the batch.

It turns out that most LLMs were pre-trained without padding. As decoders, they predict the next word based on previous words. It is then enough to split the training corpus into equal sequence lengths and train the model on it (even if sometimes the splitting is not very graceful). No need for padding, besides in many models like Mistral or Llama2, the padding token doesn't even exist in the tokenizer.

When fine-tuning these models on specific instructions, we provide the instructions and expected responses in a single block (we concatenate both). There is no reason for the sequences to be the same length. To process all of this in batches, padding is then required. Having no padding token, one way to get by is to use the eos (end of sequence) or unk (unknown) token instead.

Then, what do we do? Right or left padding?

With eos, I'm personally not very comfortable doing left padding: the model is asked to stop the sequence even though it hasn't started. Doing right padding, why not.

With , doing left or right padding should be fairly equivalent.

On the other hand for inference, you have in my opinion no choice but to do left padding. Simply because the decoder will rely on your last token to predict the next one; if it's padding, the quality of the response will be altered.

One last approach that seems quite elegant and effective for fine-tuning is to do as in initial training: no padding, concatenate the entire question-answer dataset and slice it into identical chunks of N tokens.

Curious to know your padding strategies.

Navigating the Jungle of LLM Quantization Formats, GGUF, GPTQ, AWQ and EXL2 - Which One To Pick?

2024-03-09T22:38:11+01:00

Quantizing LLMs reduces calculation precision and thus the required GPU resources, but it can sometimes be a real jungle trying to find your way among all the existing formats.

GGUF, GPTQ, AWQ, EXL2... Which quantization format to choose and why?

There are so many standards and tools because quantization is not just about rounding values. There are in fact more advanced strategies and different philosophies. It's a bit like images with JPEG, PNG, SVG, WEBP formats...

Take GGUF, the format popularized by llama.cpp, its goal is to reduce precision while optimizing calculations from a CPU perspective, with a particular focus on Apple hardware. It remains possible to offload some of the weights to the GPU for more speed. To run an LLM locally, it's therefore a good candidate, especially if you have a Mac.

GPTQ is oriented towards GPU inference performance. It quantizes the weights by minimizing error thanks to a calibration dataset. This allows comparing the outputs of the full precision and quantized model. At inference, GPTQ dynamically "dequantizes" the weights in float16 to limit loss of precision, all with low impact on GPU memory.

AWQ is based on the observation that ~1% of weights contribute significantly to model performance; these must therefore be treated with special care by applying per-channel scaling. AWQ uses a dataset to analyze activation distributions during inference and identify critical weights. AWQ is faster at inference than GPTQ and also seems to have better perplexity but requires slightly more VRAM.

EXL2 uses the GPTQ philosophy but allows mixing weight precisions within the same model. Some critical weights thus retain high precision, with the rest being more quantized to optimize performance. EXL2 probably offers the fastest inference, at the cost of slightly higher VRAM consumption than AWQ.

To finetune a quantized LLM with QLoRA, you’ll only be able to do it with GPTQ (or with bitsandbytes for on-the-fly quantization).

If you want to avoid quantizing an LLM yourself, the user TheBloke on Huggingface has close to 4000 models ready to use in GPTQ, GGUF or AWQ formats.

But things are moving so fast, it’s hard to stay abreast of the latest advances and it wouldn't be surprising to see new standards emerge in the coming months.

RAG vs Finetuning - Complementary not Opposing

2024-03-08T22:33:42+01:00

RAG and finetuning are often opposed: Should we fine-tune our LLM or do RAG? In reality, the approaches do not oppose each other. On the contrary, they complement each other.

RAG, which needs no introduction, allows querying an external knowledge base to enrich the context provided to the LLM. We thus add the relevant documents to the prompt. This approach is very interesting when dealing with evolving information because we update the database, not the model, which is much simpler and less expensive. The double advantage is that by constraining the model to rely on documents, we reduce its propensity to hallucinate.

The downside is that by adding both context, instructions, documents, thought processes (Chain of Thought), and finally few-shot examples to the prompt, the prompt becomes a whole little booklet.

Even though LLMs have increasingly impressive ingestion capacities (context window of 200k tokens for Claude 2 and 128k for GPT-4), they are designed such that they have attention loss when they have to "read novels". In particular, their focus is on the beginning and the end, with potentially lost information in the middle, hence the interest in staying concise (see the paper "Lost in the Middle").

Finetuning, on the other hand, allows adjusting the LLM by training it on your own dataset. This allows it to learn things specific to your field, jargon, tone, behavior, but also to align it with your values... This not only improves its relevance but also means that if you use RAG on top, there will be even less to make explicit in the prompt, thus giving the LLM the opportunity to focus on the key information.

A recent scientific publication has moreover quantified the performance gain of the finetuning + RAG method. (RAG vs Finetuning)

When possible, the best approach is to do both.

Assessing the Criticality of AI Failures

2024-03-07T12:27:00+01:00

What is the criticality of a bad prediction from your algorithm? In my opinion this is one of the first questions to ask before considering a generative AI or AI project in general.

It's encouraging to see public knowledge about AI progressing. However, I think some aspects still require awareness-building, like choosing and framing use cases.

With the emergence of generative AI services (ChatGPT, MidJourney, etc.) and their democratization, business ideas are blossoming. I sometimes hear project ideas or startup ideas around me that make me wince because they don't take into account cases that could go wrong.

Machine learning solutions come from statistical methods, so a prediction will never be 100% reliable. What’s worse, in some cases they’re not even deterministic – they can vary from one run to another. We have to work with that and therefore choose use cases accordingly.

With that in mind, a question that naturally arises is: when the algorithm is wrong, what will the consequences be?

For a programming assistant solution like Github Copilot, criticality is almost zero. If the code suggestion is bad, it’s not serious, you’re there to correct it, there’s a human in the loop.

For an autonomous vehicle solution, a failure can be very critical – in the worst case, it can take a life.

In any case, whatever the project, I think it's always healthy to think about cases that can go wrong and their consequences. What happens if my solution discriminates against users? If it misinforms because of hallucinations? If it generates offensive content? Dangerous content? If it gives access to sensitive information? If it causes users to lose money?

All this to say that if the slightest prediction error is critical for your business, AI is probably not the right path.

In my opinion, the "good" use cases are those that tolerate a certain error rate and for which failures do not have serious consequences. In particular, use cases with a "human in the loop" are generally relatively safe.