AI benchmark

An AI benchmark is a standard test for AI models: a fixed set of questions or tasks with known answers, so that different models can be scored the same way. The tables in which a vendor shows its new model beating a competitor are benchmark scores. For a business they help decide which models to try, although your own tasks may differ from those in the test.

How does a benchmark work?

A benchmark combines examples with known answers, a task and a way of scoring, and the results often end up on a public leaderboard. MMLU (Hendrycks et al., 2020) asks multiple-choice questions in 57 subjects, from elementary mathematics to law, and scores the share answered correctly. HumanEval (Chen et al., OpenAI, 2021) has 164 programming problems and counts a solution as correct only if it passes unit tests. Its pass@k score is the share of problems solved when the model gets k attempts.

Some benchmarks use people instead of an answer key. On Chatbot Arena, started at UC Berkeley in 2023, visitors put a question to two anonymous models and vote for the better answer, and the votes are turned into a ranking (Chiang et al., 2024). The project was renamed LMArena and, in January 2026, Arena.

Which benchmarks come up often?

• GSM8K: 8,500 maths word problems at primary-school level.
• SWE-bench: 2,294 real GitHub issues from 12 Python projects; the model must change the code so that the issue is resolved.
• GPQA: 448 multiple-choice questions in biology, physics and chemistry, on which experts with a PhD score 65% and skilled non-experts with web access 34%.
• Humanity's Last Exam: 2,500 expert questions, made because large language models score over 90% on popular tests such as MMLU.

Where does it go wrong?

A benchmark stops separating models once they all score near the top, which is why harder ones such as Humanity's Last Exam keep appearing. Test questions can also leak into training data. Zhang et al. (2024) had 1,205 new problems written in the style of GSM8K (GSM1k) and saw accuracy drop by up to 8%. Several model families showed signs of having partly memorised the original test, a form of overfitting on the benchmark itself; the leading models showed little of it.

Leaderboards can be gamed as well. The Leaderboard Illusion (Singh et al., 2025) found that some providers test several private versions of a model on Chatbot Arena and publish only the best score; Meta tested 27 variants before the release of Llama 4. The same risks are a reason to keep a small test set of your own, for example real customer questions with the answers you would accept, and to run it against each model you consider (agent evaluation works the same way).

Example

In March 2023 OpenAI's GPT-4 technical report gave GPT-4 a score of 86.4% on MMLU, against 70.0% for GPT-3.5, both with five worked examples in the prompt. It also reported a score of 298 out of 400 on a simulated bar exam, around the 90th percentile. Martínez (2024) re-examined that percentile: against first-time test takers he estimated the 62nd, and against people who passed the exam the 48th.