LLM benchmarks explained

Every model launch arrives with a chart of benchmark scores, and every leaderboard claims to know which model is best. This guide explains what the major benchmarks actually test, which ones still separate frontier models as of October 2026, how human-preference arenas rank models, and how to build an evaluation that predicts performance on your own work.

Why benchmarks exist

A benchmark is a fixed set of tasks with an agreed scoring method, so different models can be compared under the same conditions. Researchers use benchmarks to track progress, labs use them to catch regressions, and buyers use them as a first filter when building a shortlist.

The catch is that a benchmark is a proxy. It measures performance on one sample of questions, in one format, under one set of settings. A model that aces graduate-level physics questions can still fumble your invoice format. Treat public scores as a screening tool and your own evaluation as the deciding vote.

Benchmarks also have a life cycle. A new test starts out hard, labs optimize for it, scores climb toward the ceiling, and the field moves on. Knowing where a benchmark sits in that cycle tells you how much its scores can still reveal.

What the major benchmarks measure

The table covers the benchmarks you'll meet most often in model cards and launch posts. Status reflects published results and maintainer statements as of October 2026, so check the source before relying on any single score.

Two patterns stand out. Multiple-choice knowledge tests have mostly topped out, so the signal has moved to benchmarks that require doing something: shipping working code, using tools correctly, or adapting to an environment the model has never seen. And the newest tests deliberately target what models still do badly, which is why their scores look low.

Long-context results deserve special care. The RULER study found that only half of the models it tested, all claiming windows of 32,000 tokens or more, performed satisfactorily at 32,000 tokens. The context windows guide explains why advertised and usable context differ.

Contamination and saturation

A benchmark is saturated when top models score so close to the ceiling that differences are mostly noise. Small test sets make this worse. GPQA Diamond has 198 questions, so each one is worth about half a percentage point, and a 2-point lead is just four questions. AIME results usually cover a year's two 15-problem exams, so a single problem moves the score by more than 3 points.

Errors in the answer key cap scores too. If 3 percent of a test's answers are wrong, a flawless model tops out at 97 percent, and a model that scores higher may have memorized the mistakes.

Contamination happens when test questions or answers leak into training data, usually because benchmarks are published on the web that models learn from. The model then recalls instead of reasoning. OpenAI's February 2026 audit of SWE-bench Verified is the clearest recent case. Every frontier model it tested could reproduce some original fixes or task details verbatim, and 59.4 percent of the 138 hard problems it reviewed had flawed tests or descriptions.

OpenAI concluded that recent gains, from 74.9 to 80.9 percent over six months, increasingly reflected exposure rather than skill. Benchmark makers respond with private held-out test sets, fresh questions written after training cutoffs, contests that refresh every year, and interactive tasks like ARC-AGI-3 that can't be answered from memory.

Human-preference arenas

Arenas take a different approach. A user submits a prompt, two anonymous models answer side by side, and the user votes for the better response. The best known is LMArena, which began as the academic Chatbot Arena project and now operates under the Arena brand. Its 2024 paper described more than 240,000 votes, and the platform has gathered far more since.

Votes become ratings through a Bradley-Terry model, which estimates each model's strength from all pairwise results at once. The project originally used chess-style Elo updates, which depend on the order of games; the Bradley-Terry fit removes that effect and yields confidence intervals. Ratings sit on an Elo-like scale where a 50-point gap means the higher-rated model wins about 57 percent of votes, and a 100-point gap about 64 percent.

Arenas capture something static tests miss: what real people prefer on real prompts. They also carry known biases:

• Style over substance. Voters tend to reward longer, well-formatted, confident answers. Arena publishes style-controlled rankings that adjust for length and Markdown, so compare both views.
• A particular prompt mix. Votes reflect what visitors ask, which leans toward coding, quick questions and puzzles rather than long documents in your domain.
• Unchecked answers. Voters rarely verify facts or run code, so a fluent wrong answer can win.
• Uneven access. The 2025 paper "The Leaderboard Illusion" reported that some providers privately tested many variants before release, 27 for Meta ahead of Llama 4, and that Google and OpenAI each received about a fifth of all arena data. Arena has since open-sourced its ranking code, which reweights models with fewer battles.

The practical lesson: an arena rating is a useful read on general chat quality, but a lead of a few points says little about your workload.

How to read a leaderboard skeptically

Whether you're looking at a lab's launch chart, a third-party tracker or our LLM leaderboard, run through these questions before drawing conclusions.

Composite indexes that blend many benchmarks, such as Epoch AI's Capabilities Index, smooth out single-test noise but inherit their ingredients' blind spots. Use them to shortlist, then line up finalists with side-by-side comparison and your own tests.

Building your own evaluation

Start with a golden set

A golden set is a collection of real inputs with known-good outputs or clear pass criteria. Fifty to two hundred examples is enough to start. Pull them from production logs or realistic drafts, cover your task types in proportion to real traffic, and deliberately add hard cases and past failures. Keep the set private, and add each new failure you find.

Know your margin of error. With 200 cases and a pass rate near 85 percent, the 95 percent confidence interval is roughly 5 points either way; with 100 cases at 90 percent, it's about 6. A model that scores 87 against another's 84 on 200 cases hasn't clearly won, so rerun before deciding.

Write rubrics, not vibes

For open-ended outputs, break quality into yes-or-no checks: "cites the policy section," "under 150 words," "no invented order numbers." Binary criteria are applied more consistently than a 1-to-10 scale, and they tell you what to fix. Score exact matches and formats with code wherever possible, since it's cheaper and deterministic.

LLM-as-a-judge pitfalls

Using a strong model to grade outputs, called LLM-as-a-judge, makes evaluation cheap to scale. The 2023 study that popularized it found GPT-4's verdicts matched human preferences over 80 percent of the time, about as often as humans agree with each other. It also documented systematic biases:

• Position bias: judges favor an answer because of where it appears. Run pairwise comparisons in both orders and count only consistent wins.
• Verbosity bias: longer answers look better. Put length expectations in the rubric.
• Self-preference: models tend to favor outputs like their own. Use a judge from a different model family when you can.
• Limited checking: judges miss math or code errors they can't verify. Supply a reference answer or execute the code.

Before trusting a judge, have a person label 50 to 100 examples and measure how often the judge agrees. If agreement is low, fix the rubric before scaling up.

Cost and latency are metrics too

Suppose Model A passes 92 percent of your tests at $0.012 per task with a 9-second p95 latency, and Model B passes 89 percent at $0.002 with 2 seconds. At 100,000 tasks a month, A costs $1,200 and B costs $200. But if a person reviews every failure, B's extra 3 percent means 3,000 more reviews; at 50 cents each that's $1,500, which makes the "expensive" model $500 a month cheaper overall.

Record cost per task, cost per correct answer, and p50 and p95 latency next to accuracy. The LLM cost calculator handles the token arithmetic, and the prompt engineering guide shows how to rerun the same set after every prompt change.

Common mistakes

• Choosing a model from one number. A single headline score hides the settings, cost and task fit behind it.
• Mixing settings. Comparing one model's tool-assisted, high-effort score with another's plain score is meaningless.
• Trusting saturated tests. MMLU and HumanEval no longer separate frontier models.
• Evaluating on a dozen examples. At that size, noise looks like signal.
• Leaking the test set. Examples you use in few-shot prompts or publish online shouldn't also be in your golden set.
• Running each case once. Outputs vary, so repeat runs at your production settings.
• Evaluating only at launch. Rerun after every prompt edit and model update, and pin model versions in between.

Next steps

Check the LLM leaderboard to see how current models rank on quality and value, then line up two or three finalists in side-by-side comparison. The guide to choosing an LLM shows how to combine benchmark results with cost, privacy and latency constraints.

If you'd like help designing a golden set or an automated evaluation pipeline, you can work with an LLM evaluation specialist.