Fourteen questions to ask a vendor about their benchmark

A vendor shows you a benchmark result. Here's what to ask before you believe it.

None of these are hostile. A good vendor will enjoy them, because they're the questions that separate real measurement from marketing — and a vendor who has done the work wants you to know.

About the tasks

1. How many tasks, and how were they chosen?

Fifty tasks gives roughly ±14 points of resolution. If they're claiming a three-point win on fifty tasks, they're reading noise.

2. What's the difficulty distribution?

Ask specifically for the share of clean cases versus exceptions and adversarial cases. Suites are routinely 90% clean, because clean cases are easiest to write — and clean cases are where every serious model already succeeds.

3. Who wrote the correct answers, and what were their qualifications?

"A domain expert" is not an answer. Which domain, how senior, and did more than one of them write an answer for the same case? If two experts never independently answered the same task, nobody has tested whether the specification is unambiguous.

4. Can I see the rubric?

Then read it looking for one thing: are there criteria a reasonable person could apply two different ways? "Reasonably clear," "appropriate tone," "good quality" are not criteria.

About the grading

5. Is the grading automated, human, or model-based?

All three are legitimate. The follow-up matters more than the answer.

6. If model-based — what's the agreement with human experts, on what sample size?

And is that raw agreement or corrected for chance? Raw agreement is inflated by the base rate. If 85% of cases are passes, a judge that says "pass" to everything scores 85%.

7. Is that a point estimate or a lower bound?

A point estimate from forty cases is mostly luck. Ask what the confidence interval is.

8. Did you test the grader for position, verbosity, and self-preference bias?

Three cheap experiments. If they haven't run them, they don't know whether their judge prefers longer answers or its own model family — and if the judge is from a vendor whose model is being evaluated, that's a conflict of interest inside the methodology.

9. Does the known-correct answer score 100% against your own grader?

This is the single highest-yield question on the list. If the reference answer can't earn full marks, the task is broken. Published, widely-used benchmarks have failed this check.

10. Did you write deliberate cheats, and do they score zero?

If a plausible shortcut scores anything, the grader rewards that behaviour in every model tested.

About reproducibility

11. What was pinned — model version, scaffold, environment, judge?

"Latest" is not a specification. If the model version isn't recorded with the result, the result can't be reproduced or defended six months from now.

12. What's the run-to-run variance?

Ask for the spread across repeated runs of the same configuration. If they haven't measured it, no difference they report can be interpreted.

13. Does the benchmark touch the live network during a run?

If yes, two runs are not comparable and any change over time confounds the model with the world.

About the result

14. Could a third party reproduce this number?

Not agree with it — reproduce it. This is the question that separates a measurement from a claim, and very few published results survive it.

How to read the answers

You're not looking for perfection. You're looking for whether they've thought about it.

A vendor who says "we haven't measured run-to-run variance, but here's our sample size and here's the interval" is being straight with you and is probably competent.

A vendor who has an answer for everything and no caveats anywhere is either exceptional or hasn't looked. The base rate favours the second.

Three answers should genuinely worry you:

• They can't show you the rubric. Either it doesn't exist in a form that would survive being read, or the criteria are subjective and they know it.
• They built the benchmark, ran it, and reported it, with no external check. One party as rule-maker, examiner and candidate. Not necessarily dishonest — structurally unfalsifiable, which is worse, because nobody inside can tell the difference either.
• The benchmark is public. Then contamination is a live possibility, and the score may be partly recall rather than capability.

The question underneath all fourteen

Whatever they show you, the real question is whether it's about your work.

A model that tops a general leaderboard is generally capable. That is worth knowing and it narrows a shortlist. It says nothing about whether it can do the specific job at your company, under your policy, with your systems, on the cases that make your version of the job hard.

Ask them to run it on twenty of your cases. The answer to that request — not the benchmark they arrive with — is usually the most informative thing in the whole meeting.