Say what you're predicting before you know if you're right

Eventually somebody senior asks the question that all of this was for:

Did the benchmark predict anything?

The model that scored highest — did it actually work better once it was doing the job? Or did you spend two quarters building an elaborate apparatus for producing numbers that correlate with nothing?

It is the right question. Most evaluation programmes cannot answer it, and the reason is not statistical.

The statistic is the easy part

Rank your models by benchmark score. Rank them by whatever production outcome you care about — resolution rate, escalation rate, correction rate. Compute a rank correlation. It's twenty lines of code and an afternoon.

Use a rank correlation rather than a linear one, incidentally: neither quantity is normally distributed, both are bounded between 0 and 1, and the claim you're testing is monotone ("higher score, better outcome"), not linear. You do not care whether the relationship is a straight line. You care whether it's in the right order.

That's the whole method. It is not the hard part.

The hard part is that you have to commit first

Here's what happens without discipline.

Six months of production data accumulates across a dozen workflows. Someone pulls the benchmark scores, tries the correlation against workflow A — nothing. Workflow B — weak. Workflow C — 0.82! That goes in the deck.

That number means nothing whatsoever. With a dozen candidate outcomes and a handful of models, a strong correlation somewhere is close to guaranteed by chance alone. You have not measured predictive validity; you have measured your own search process. And the search is invisible in the result — the slide says 0.82 and doesn't mention the eleven you tried first.

The fix is the one experimental science settled on a century ago and software has mostly not adopted: write the claim down before the outcome exists.

When you certify a benchmark result, record — as part of the result, not beside it — which production workflow this score claims to predict, and the date the claim was made. One workflow. Named in the customer's own vocabulary. Timestamped.

Then, when the production data arrives, there is exactly one correlation to compute and no choice available to you. The number means what it appears to mean.

Put it inside the signature

If your results are signed or hashed — and for anything going in front of a regulator or a board they should be — the prediction claim belongs inside the signed payload, not attached to it.

The difference matters. A claim inside the signature is one you can prove you made before you knew the answer. A claim in a spreadsheet next to it is one you could have edited last Tuesday. The entire value of pre-registration is that the claim provably predates the outcome, and only the signature establishes that.

This has a design consequence worth flagging, because it catches people: you cannot retrofit it. A result certified without a prediction claim can never acquire one honestly — adding it later is precisely the thing pre-registration exists to prevent. So the field has to be there from the first certificate you ever issue, even though you won't be able to use it for a year.

Two implementation notes that keep it clean:

• It's optional, and absent by default. A result making no predictive claim should carry no empty one. An empty claim invites someone to fill it in later.
• Omit it from the hash when absent, so results that make no claim hash identically to those issued before the feature existed. Otherwise adding the capability retroactively invalidates every signature you've ever produced, and your tamper detector starts alarming on honest history.

Rules that keep the pairing honest

Only pair periods that follow the claim. Correlating a score against outcomes that predate the moment you made the claim is measuring hindsight. This sounds obvious and is violated constantly, usually by a join that ignores dates.

Only pair the named workflow. A certificate that claimed workflow A does not get quietly matched against workflow B because A's data is thin.

Below five paired periods, report nothing. A rank correlation over three points is a picture of three points. The honest output is null and a note saying how many more periods are needed — which is actionable, unlike a spurious 0.9.

Don't report a p-value. Deliberate. At single-digit sample sizes a p-value is a false precision that will get quoted out of context and defended in a meeting. Report the count and show the scatter. Anyone qualified to interpret a p-value there can compute it, and everyone else is better served by seven dots on a chart.

Never sign the correlation itself. The claim is signed at issue time; the correlation is computed later from data that didn't exist yet, and it changes every reporting period. Signing a number that will change is how a signature stops meaning anything.

What a weak correlation actually tells you

Assume you do this and it comes back at 0.2. Is the programme a failure?

Usually not — but it is information, and there are three quite different explanations you now have to distinguish:

The benchmark measures the wrong thing. Your tasks are lookups and the production difficulty is ambiguity resolution. Fix the tasks.

The production metric measures the wrong thing. Resolution rate is dominated by queue routing and staffing, and the model's contribution is a small term inside it. Fix the metric, or find a nearer one.

Both are right and the relationship is genuinely weak. Model quality above some threshold stops being the binding constraint on the outcome. That's a real and valuable finding — it says stop optimising the model and go work on the surrounding system.

Any of those three is worth more than not knowing. And you can only tell them apart if the claim was specific enough to be wrong.

Be honest about the timeline

This ships as a mechanism, not a result. Nothing can be measured until you've certified a benchmark and then accumulated five reporting periods of production telemetry on the same workflow. For a programme starting today that is quarters away.

Build it anyway, and build it first. The claim has to be recorded at issue time, and a result issued without it can never be repaired.

The one-line version

The only evidence that your evaluation predicts anything is a prediction you made, in writing, before the outcome existed — and a correlation chosen after seeing the data measures nothing but your own choosing.