How to Read a Benchmark Number

1. The claim

Nobody in this story is cheating. That is what makes it worth writing down.

A benchmark number is produced by a pipeline: a set of tasks, an answer key that decides what counts as correct, a policy for how many attempts a model gets, and a harness that decides what the model can see and do. A capability difference between two models is one input to that pipeline. The claim is that the other four inputs move the output by more than the model gap you are trying to read, and that they are almost never published alongside the number.

So when you compare 62 against 55, you are not necessarily comparing two models. You may be comparing two attempt policies, or one contaminated model against one clean one, or two harnesses, and the arithmetic will look exactly the same in every case.

2. The answer key can be wrong

This is the one that should change how you read every chart you see, because it happened on the benchmark everybody quotes.

SWE-bench Verified is the standard for agentic coding. OpenAI audited 138 of its problems, roughly 27.6 percent of the set, concentrating on ones models often failed. At least 59.4 percent of the audited problems had flawed tests. Forty-nine tests were too narrow and rejected functionally correct submissions. Twenty-six were too wide and demanded behaviour the issue never asked for. OpenAI’s conclusion was to stop evaluating against it and publish why.

Read the direction of that failure carefully. Tests that are too narrow reject correct work, which means the unsolved pile was never entirely a pile of model failures. Some fraction of every “the model could not do this” was “the grader would not accept it”. Separately, a study found more than 15 percent of Verified instances carry incomplete test patches that let wrong or partial solutions through, so the errors run in both directions at once. Work on test adequacy suggested leaderboard scores may be inflated by six to seven points on that basis alone.

Six to seven points is larger than the gap that decides most procurement arguments.

3. The model may have seen the answer key

SWE-bench Verified draws on public GitHub issues, and its 500 tasks and their resolutions have been sitting in public repositories for years. Any model trained on GitHub data after mid-2024 has plausibly read them, solutions included, and contamination has been reported across frontier models generally rather than at one lab.

This is not a fixable oversight so much as a structural property of building benchmarks from public data and then training on public data. A benchmark’s usefulness decays from the day it is published, and the decay is invisible in the number.

The same shape shows up in speech recognition. When reference transcripts contain errors, models that have optimised against the benchmark reproduce the erroneous transcript rather than what the audio actually says. Part of a leaderboard lead is memorising the key rather than doing the task, and from the outside those two look identical.

4. The headline may be several attempts pooled

This one is the easiest to check and the most commonly missed.

Aikido ran ten models three times each against 32 freshly disclosed CVEs, asking each to rediscover the vulnerability from source. DeepSeek V4 Pro’s headline 28 of 32 is the union of three runs, not one. The write-up’s own finding is that running a cheaper model a few times and pooling reliably beats a single pass of a stronger, pricier one, and that DeepSeek V4 Flash reached 24 of 32 across three runs at more than ten times less cost than a frontier competitor.

That is a genuinely useful result about how to spend money. It is a different result from “this model finds 28 of 32”, and the second sentence is the one that travels.

The question to ask of any score is how many attempts bought it, because pass at three and pass at one are different quantities with the same units. If your production path gives the model one shot, a pooled number is not a forecast of anything you will experience.

5. Why this persists

Because every party is behaving reasonably.

The one genuinely encouraging thing in this paper is that OpenAI published the audit that made its own strong results on that benchmark unusable. That is the behaviour you want, it is rarer than it should be, and it is the source I would trust most in the whole argument precisely because of who it costs.

6. Reading one properly

Four questions, in order of how much they change the answer.

If you cannot answer any of the four, you have a number and no idea what it is a number of, and the correct move is to treat the leaderboard as a shortlist rather than a ranking. Take the top three, then run your own thirty tasks in your own harness, which is the same prescription as 5.17 and 5.18 and is starting to look like the only prescription this site has.

I said in the abstract that this is more expensive than it sounds. Building thirty representative tasks with a defensible answer key is genuinely a week of somebody’s time, and the reason everyone reaches for the leaderboard is that the leaderboard is free. The argument is that a decision worth a year of inference spend deserves a week of somebody’s time, and that almost nobody is making that trade consciously.

7. The strongest objection

Benchmarks improved this field enormously, and a paper that makes people distrust them may do more harm than the flaws do. A noisy shared yardstick beats everybody privately claiming their model is best, which is the world benchmarks replaced and which was much worse.

I accept that and it is why the four questions in Section 6 are questions rather than a recommendation to ignore scores. What fails here is measurement quoted without its conditions, which is a reporting problem and a reading problem, and both are fixable without burning the practice down.

8. What this paper does not claim

It does not claim SWE-bench is bad work. Auditing your own widely adopted benchmark and publishing the flaws, as its ecosystem did, is how this is supposed to go.

It does not claim contaminated models are not useful. A model that memorised solutions to real GitHub issues has memorised something genuinely valuable. It just is not the thing the benchmark says it measured.

And it does not put a number on the total distortion. I drew Figure 1 without a scale on purpose, because assigning sizes to those four bars would be inventing the exact kind of figure this paper is complaining about.