Are LLM Benchmarks Reliable? Not Such a Simple Question…
Are LLM Benchmarks Reliable? Not Such a Simple Question...
"When a measure becomes a target, it ceases to be a good measure” - Charles Goodhart.
Let's take Hugging Face's Open LLM Leaderboard, very useful for comparing LLMs. It uses 4 benchmarks including the most popular "MMLU".
The MMLU is a set of question-answering to assess "multi-task accuracy" across 57 tasks (math, history, computer science...).
I see 3 points of attention:
1/ The first is that MMLU implementation is not universal. In the LLama 1 scientific paper, the 65B version achieves an MMLU score of 0.637 while at the time, on HuggingFace's leaderboard, it only had 0.488 (before HuggingFace synchronized the leaderboard score with the publication).
So why?
First, it's important to understand an LLM predicts the next word based on its probability.
In the original MMLU implementation, used by the LLama team, the model must, when faced with a question, select an answer among 4 choices (A, B, C or D). The score does not take into account the entire LLM vocabulary, but is limited to these four letters. So even if the expected answer is "B" and the most probable word in the model's vocabulary is "Voyage", followed by "B", the model is deemed correct because consideration is limited to the probabilities of A, B, C or D.
In the HuggingFace Leaderboard implementation that uses the "Harness" variant, we calculate the probabilities the model associates with the full answer sequences, for example "A: This is answer A”, “B: This is answer B", etc. The score is still limited to the 4 given answers but considers the full sequence.
So two MMLU scores from different sources may not be directly comparable depending on implementation.
2/ Some models may have been trained or tuned directly on the benchmarks, causing a data leak from the evaluation set. This is obviously a form of cheating, sometimes intentional. The Hugging Face team normally does what is needed to regularly remove contaminated models from the ranking but some remain. One sign is when "small models" have scores comparable to very large ones. In this case, HuggingFace could request access to the dataset for verification.
3/ Generic metrics like MMLU are probably not relevant for your much more specific use cases (summarizing, interacting with humans etc.)
Benchmarks should be used as a guide and starting point but not absolute truth. Do your own evaluation for your use case and it's very likely the results will surprise you.