The importance of monitoring LLMs performances in production

Is ChatGPT becoming less capable over time?

(This article is a transcript in English of one of my french LinkedIn posts)

This is what a recent Stanford study suggests, which caused quite a stir over the past few days:

While the study is debated, it raises a very interesting point.

The need to have a system to evaluate the performance of LLMs (Large Language Models) in production.

Whether you use different models via an API, compare open source models, or explore fine-tuned versions of the same model, you will need a point of comparison.

The authors proceeded as follows:

They created sets of questions covering 4 tasks: solving math problems, answering sensitive questions, generating functional code, and formulating visual reasoning.

What is interesting is that one reason for choosing these types of problems is that they have objective answers and are therefore easy to evaluate. Another noteworthy point is that through the ability to answer sensitive questions, they quantified the model's propensity to give answers it should not (crucial for evaluating its ethics).

Regarding the metrics used:

• Accuracy (rate of correct answers or executable code portion depending on the task)
• Verbosity (length of response)
• Overlap of responses between two versions of the same model

Measuring overlap may seem surprising at first, but it is also interesting! It allows quantifying to what extent two versions of the same model (here GPT-3.5 in March and June 2023) give the same response to the same question. In other words, it measures the "continuity" between two versions. I think we could, for example, reuse this approach to compare two differently fine-tuned versions of the same model.

Finally, they concluded that between two versions of GPT-4, performance significantly declined.

In any case, I think there are quite a few interesting ideas to leverage...

In case you woudlike to reuse the datasets of questions used in the paper, here is the link to the project:
https://github.com/lchen001/LLMDrift