The number nobody publishes

Here are two numbers about the same model on the same benchmark.

91%. And 34%.

Neither is wrong. They answer different questions, and almost every published result is the first one.

What the two numbers are

Run each task five times instead of once. Now, for each task, you have five outcomes rather than one, and there are two obvious things to count.

Count a task as solved if at least one of the five attempts worked. That's pass@k — with k=5 here. It's a measure of capability: is the answer inside this model at all?

Count a task as solved only if all five attempts worked. That's pass^k — read "pass-hat-k". It's a measure of reliability: can I depend on it?

If you only ever run each task once, you're reporting pass@1, and you cannot compute either of the interesting numbers. Most leaderboards are pass@1. Some report pass@k with a large k, which is the most flattering number available.

Very few report pass^k. It's the one your deployment depends on.

Why the gap is the interesting part

A model at 91% pass@5 and 34% pass^5 has a specific, diagnosable condition: it knows how to do most of these tasks, and it does them about half the time.

That gap is not noise. It's variance in the model's own behaviour, and it has causes you can act on — an ambiguous instruction it resolves differently on different samples, a tool it sometimes forgets to call, a retry path it sometimes doesn't take. High capability with low reliability is a prompt, scaffold, and guardrail problem, and it's usually fixable without changing models.

The opposite shape — pass@k barely above pass^k, both low — is a different diagnosis entirely. The model is consistent and consistently wrong. No amount of scaffolding fixes that. You need a different model, a different decomposition of the task, or a human in the loop.

One number cannot distinguish those two situations, and they call for opposite responses. That's the argument for reporting both.

The number your users actually experience

Think about what a user meets in production.

They don't get five attempts and keep the best. They get one attempt. If the workflow runs a thousand times a week, they meet the distribution, not the maximum.

A 91% pass@5 headline invites the reading "this works 91% of the time." It doesn't. On a single attempt, it works at whatever the per-attempt rate is — and pass^k tells you how often all attempts land, which is exactly the shape of a multi-step workflow where every step has to work for the run to be useful.

For an agent that chains six tool calls, per-step reliability compounds ruthlessly. A step that works 95% of the time, six times in a row, is a workflow that works 74% of the time. Reliability is the only number that composes.

Rules that keep these honest

Below k=2, neither number exists. Not zero — absent. pass@1 and pass^1 are both just the pass rate, and reporting them under a k-name implies a measurement you didn't make. If a task has only one rollout, the honest output is null and a note saying how many more runs are needed.

Report k, always, even when the rate is null. "pass^3 = 0.34" and "pass^12 = 0.34" are wildly different claims. A rate without its k is uninterpretable, and the k is the cheap half.

Use the minimum k across tasks, not the average. If one task ran twice and the rest ran ten times, your aggregate is a pass^2 claim. Averaging the k's produces a number that describes no task in the set.

Vary the seed, not the prompt. The point is to sample the model's own variance. If each rollout gets a slightly different prompt you've measured prompt sensitivity, which is also interesting and is not this.

Report per-task, not just the aggregate. The aggregate hides the shape. Ten tasks at 100% and ten at 0% average to the same place as twenty at 50%, and the first is a coverage problem while the second is a reliability problem.

The related trap: a benchmark that separates nothing

While you have the per-task numbers in front of you, look for a second failure.

A task every model passes tells you nothing. A task every model fails tells you nothing. Both belong in a benchmark about as much as a question every candidate answers correctly belongs in an exam.

If most of your task set sits at 0.0 or 1.0 across the whole model panel, your benchmark is not measuring — it's confirming. The headline pass rate will still move around, driven entirely by the handful of tasks that actually discriminate, and you'll draw conclusions about a model from a sample of four items.

The caution here is against over-correcting. A task that went 0-for-3 has a point estimate of 0.0 and a confidence interval reaching most of the way to 0.7. Refusing that task is refusing on three samples. Use an interval, not a point estimate, and only rule out tasks whose interval sits entirely at one end.

Which brings the two ideas together: you need multiple rollouts per task to say anything about reliability, and you need them to say anything trustworthy about whether a task discriminates. The same k buys both.

What to do on Monday

1. Pick your five most important tasks. Run each one five times, same prompt, different seeds.
2. Compute both numbers. Write them side by side with the k.
3. Look at the gap. Large gap → work on scaffolding. No gap and low → work on the model or the task decomposition.
4. Look at the per-task table for anything at exactly 0/5 or 5/5. Those tasks are costing you slots and returning no information.

This costs five times the compute of what you're doing now, on a set small enough that five times nothing is still nothing.

The one-line version

pass@k tells you what the model can do. pass^k tells you what it will do.

You are shipping the second one. Publish it.