The benchmarks are broken — and the audits are arriving late

Note before you read: the findings summarised here come from published audits. Check each against the primary source before repeating the numbers — including these.

The dominant evaluation finding of 2026 isn't about models. It's that the benchmarks themselves contain errors serious enough to change conclusions.

Three findings worth sitting with

Curated is not verified. SWE-bench Verified was built specifically to be the trustworthy subset — the original had known problems, so a version was filtered with human review. A 2026 audit reported that on the hardest tasks, a majority had tests that would pass even when the underlying bug was never fixed.

Read that again. The grader could not distinguish a fixed bug from an unfixed one. Every model evaluated on those tasks received credit for work it did not do, and none of it was visible from the leaderboard.

Repair is worth points. A 2026 re-examination of a workplace agent benchmark corrected scoring bugs, ground-truth errors, and prompt issues — then held the model completely fixed and re-scored it on both versions. The same model went from 49% to 57%.

Eight points. Same weights, same day. Eight points of the benchmark becoming fairer, sitting in the record as model progress.

Most failures may not be the model's. An audit of a data-engineering agent benchmark examined every task the agent failed and asked whose fault each was. Three categories dominated: evaluation scripts too rigid to accept correct-but-differently-formatted output, ambiguous specifications, and ground-truth values that were simply wrong. The large majority of audited failures contained at least one benchmark-attributable error.

The agents were being marked down for being right.

What these have in common

Every one of those findings came from someone auditing a benchmark after it shipped — often years after, long after its numbers had entered the literature and shaped what people believed about model progress.

That work is valuable. It's also the wrong end of the pipeline.

Here's what should be uncomfortable: the checks that catch these problems in an audit are the same checks you could run before publishing.

• Can the known-correct answer earn full marks against this grader?
• Can a plausible shortcut beat it?
• Does the environment actually contain what the task requires?

None of that is expensive. It's just usually optional — and anything optional gets skipped under a deadline.

Two checks, before anything ships

Check one: the oracle must score 100%. Write the answer a genuine expert would give and run it through your own grader. If the reference answer can't earn full marks against your own rubric, the task is broken — your rubric is asking for something the correct answer doesn't do.

Check two: deliberate cheats must score 0%. Write plausible-looking shortcuts. The empty answer. The one that restates the question in confident language. The keyword stuffer. Run those through. If a shortcut scores anything at all, the grader is broken — and it will reward exactly that behaviour in every model you test from now on.

Both failures are common. And here's the part that makes them dangerous: neither is visible from the score. A broken task and a good task produce numbers that look identical.

What this means for your internal evaluation

If published, peer-reviewed, widely-used benchmarks contain these errors, your internal suite almost certainly does too. Not because your team is careless — because everybody's does, and the only way to find out is to audit failures rather than count them.

A practical starting point: take fifty failures from your last run and read them. Not the summary — the transcripts. For each one, ask whose failure it was. Model, task, grader, or environment?

Teams that do this exercise for the first time are routinely surprised by how much of their "model failure" is specification failure. That's not bad news. Specification failures are cheaper to fix and they stay fixed.

Saturation and contamination, briefly

Two structural problems worth naming, because they cannot be fixed by being careful.

Saturation. A benchmark stops working long before people stop publishing it. When the frontier clusters at the top, the differences between models are smaller than the noise in the measurement. Report intervals rather than point scores so overlapping models look overlapping. Watch the hardest slice, where separation survives longest. And be willing to retire a benchmark when it stops separating — which is hard, because the effort that went into building it makes retiring it feel wasteful. It isn't. A benchmark everyone passes has finished its job.

Contamination. Published tasks get scraped into training data. No one has to cheat. The defences all involve giving up public inspectability of the tasks in exchange for trustworthiness of the result: held-out sets never published, tasks generated after a model's cutoff, or private sets where results are shared but tasks never leave. Knowing you're making that trade is most of the skill.

The one-line version

An audit finds the damage. A gate prevents it.

The difference between the two is a few hours per task, spent before anyone believes the number.