How to run a model bake-off that produces a decision
Most vendor evaluations produce a slide. Here's the protocol that produces a defensible choice.
Note: the figures in this post are illustrative examples, not measured results. The bake-off numbers below show the shape of the decision, not a recommendation for any model.
You have three or four candidate models and a decision to make. Most bake-offs at this point produce a slide with some percentages and a recommendation that reflects whoever ran it. Here's how to run one that survives scrutiny.
Before you touch a model
Write down what you'll decide. "We will deploy the winner to the claims triage workflow in Q3" is a decision. "We will understand model capabilities" is not, and an evaluation scoped to it will never end.
Write down the threshold. What pass rate would make you ship? What would make you not ship? Decide this before you see any numbers. The version of you who sets it now has no favourite yet.
Write down the comparisons. Which models against which, on which slices. Anything you test later that isn't on this list is a hypothesis, not a result — that's the multiple-comparisons problem, and pre-registration is the only cheap defence.
Fix everything except the model
This is the part that gets skipped and it's the part that makes the comparison mean anything.
| holds constant | why |
|---|---|
| the task set | different tasks = different exams |
| the rubric and graders | a changed rubric is a changed benchmark |
| the judge, and its version | judge drift reads as model difference |
| the scaffold | the harness often outweighs the model |
| the environment image | a different tool surface is a different test |
| the budget cap | an unbounded model isn't competing fairly |
| temperature and seeds | or at least, report them |
Change exactly one variable. If you upgrade the scaffold for one candidate because "it works better that way," you're no longer running a bake-off — you're comparing two systems and attributing the difference to the wrong component.
If a vendor insists their model needs their harness, that's a legitimate position. Run it as a separate entry, labelled as a system rather than a model, and say so in the result.
Include the entries people forget
- Your current stack. Without it you can measure which candidate is best and not whether any is better than what you already run. This is the single most common omission and it's the one that matters commercially.
- A cheap model. If the cheap one is within noise of the expensive one on your workflow, that's the finding, and it's worth more than the ranking.
- A deliberately weak baseline. If your benchmark can't separate a small old model from a frontier one, it isn't sensitive enough to separate the frontier ones either.
Run it properly
Multiple seeds per model. One run per candidate tells you nothing about whether the gap is real. Three minimum.
Pin versions by date, not by alias. Record the exact version string with the result.
Log everything. Every tool call, every judge verdict, every refusal. The transcripts are where the actual findings live — the scores are just an index into them.
Cap the budget identically. A model that scores two points higher by using four times the tokens has not won.
Read it honestly
Intervals, not points. If the intervals overlap, you have not shown a difference. Say so. "We cannot yet distinguish A and B" is a legitimate and useful finding — it moves the decision to cost, latency, and support, which are things you can actually resolve.
Look at the failure map, not just the ranking. Two models at the same score can fail completely differently. One misses exceptions; the other invents figures. Those are not equally acceptable, and the score cannot tell them apart.
Check the hard slice separately. Overall scores saturate. The exception and adversarial tiers are where the candidates actually separate, and where your risk lives.
Count the cost. Tokens, tool calls, retries, latency, escalation rate. Put them in the same table as the pass rate. A decision made without them isn't a decision, it's half of one.
What the output should look like
Not a winner. A recommendation with its reasoning exposed:
Model B, at 87% [83–91] against our current 76% [71–81], within budget at 1.4× current cost per case.
A scores 89% [85–92] but at 3.1× cost, and the interval overlaps B's.
B's remaining failures cluster in two named patterns — 61 cases of X, 44 of Y — both addressable with roughly 1,300 examples of expert-worked data.
Re-run scheduled on every provider version bump. Rollback to current stack is a config change.
That paragraph answers all three questions a board will ask. A percentage does not.
Two traps
The demo trap. A vendor will offer to run the evaluation for you, or to help tune their entry. Politely decline both. The moment one candidate gets optimisation the others didn't, you're measuring engagement effort rather than capability.
The moving-target trap. Bake-offs take weeks, and during those weeks someone will want to add a newly released model. Adding it is fine. Re-running everything against a rubric you've since revised is not — that's two benchmarks. Freeze the fingerprint at the start; new entrants run against the frozen version or they run in round two.
The thing worth keeping
The winner is the least valuable output of a good bake-off.
The task set, the rubric, the calibrated grader and the environment are reusable for the next model release, the next vendor, and the next workflow. The decision expires in six months. The instrument doesn't.