Evaluation doesn't end at launch

Most evaluation effort goes into deciding what to ship. Then the system ships, and the instrument that made the decision is quietly retired.

That's backwards. The decision was a one-time event. The risk is continuous.

Offline and online answer different questions

Offline asks: can it do the job, under controlled conditions? Fixed tasks, known answers, repeatable.

Online asks: is it doing the job, on real traffic, today?

Offline gives you comparability — same tasks, same graders, any time — which is the only way to rank models or measure a change. What it cannot tell you is what your users are actually sending, and that drifts constantly.

Online gives you reality and almost no ground truth. Nobody knows the correct answer for most live requests, and you can't rank two models on production traffic because they never saw the same inputs. Comparing them is comparing different exams.

They're not rivals; they have a division of labour. Offline decides what to ship and whether a change helped. Online tells you whether the offline picture still describes the world — and when it stops, that's your signal to update the offline suite.

The failure mode is picking one. Teams with only offline evals ship confidently into a distribution they've stopped tracking. Teams with only online monitoring can see something is wrong and have no way to test a fix.

Put the benchmark in CI

An evaluation that runs when someone remembers to run it is an evaluation that runs after the incident.

• A fast subset on every change — a few dozen tasks, minutes not hours, catching obvious breakage while someone still holds the context.
• The full suite nightly, where cost and time block nobody.
• A hard failure below an agreed threshold. Not a warning. Not a dashboard.
• The threshold set once, in advance, by someone not under deadline pressure.

That last one carries most of the value. A threshold you can lower in the moment is not a gate — it's a suggestion with extra steps. The point of writing it down in advance is that the version of you who set it had no shipping date attached.

One practical note: gate prompt changes like code changes. Most teams govern their code and leave prompts entirely open, even though a prompt edit can move behaviour further than a dependency bump ever will.

The day the provider updates the model

Your code didn't change. Your prompts didn't change. Your behaviour did.

A provider updated the model behind the alias you were calling. It will happen, and it will not be announced in a way that reaches your on-call engineer at the moment it matters.

Four defences, all boring, which is why they work:

• Pin to a dated version, never a floating alias. "Latest" is not a specification — it's a promise to accept whatever arrives.
• Re-run the full suite on every version bump, treating it as what it is: a dependency upgrade with unknown behavioural consequences.
• Keep the previous version live until the new one clears your threshold, so rolling back is a config change rather than a project.
• Record which version produced which result, in the result itself. Six months later, when someone asks why the numbers moved in March, that field is the only thing that will answer them.

The mental model that fixes all of it: treat a model version like a dependency. You'd never let a library upgrade itself in production without tests, and this is a far larger surface than any library you depend on.

Shadow first, then canary, then everyone

Three stages, and one of them is chronically skipped.

Shadow. The new model runs on real production traffic and its output affects nothing. Nobody sees it. Nothing is written.

Canary. It handles a small share of real decisions, monitored tightly enough that you'd notice within minutes.

Full. Everyone, with the rollback still warm.

Shadow is the underused stage, for a reason easy to miss: it is the only place in your entire pipeline where the old and new model see identical real inputs. Offline, inputs are identical but synthetic. In canary and beyond, both models are live and handling different requests, so any comparison is confounded.

So shadow gives you a head-to-head on the actual distribution. Where do they disagree? On which kinds of case? That disagreement set is also the best possible human-review queue — those are precisely the cases where the choice of model changes the outcome.

Run shadow long enough to cover a full business cycle. Whatever is unusual about month-end, you want to have seen it before the canary does.

You can't label production. You can sample it.

Nobody reviews every live decision — that's the point of automating them. Sample, in four streams:

• A small random sample. The only one that gives an unbiased estimate of true production quality. Boring, essential, and the one people skip because it feels wasteful.
• Everything the agent escalated. Cases it flagged as beyond its confidence — they tell you where the boundary actually sits.
• Everything a human later overrode. The most underrated data in the building. Someone looked at the output, disagreed, and acted differently. That's a free label, generated by normal operations, and most teams discard it entirely.
• Everything the judge was least sure about. Where the rubric is still ambiguous.

The last three are cheap and information-dense, but you need the random sample alongside them — the other three are all selected, so they'll show you problems and never tell you the base rate.

Then feed all four back into the offline suite.

Write the postmortem into the benchmark

Something went wrong in production. You investigated, found the cause, shipped a fix, wrote the postmortem. Standard, good engineering.

You are not finished.

If that failure doesn't end up as a permanent task in your evaluation suite, you've fixed one instance and learned nothing that will protect you next quarter, after the model changes and the fix quietly stops applying.

Extract two things from every incident: the failing case becomes a task — real inputs, correct handling, in the suite forever — and the rule it violated becomes a criterion, if it wasn't one already, because the incident is proof that the rule was load-bearing and unstated.

Now consider what that does over two years. A benchmark that only grows from imagination drifts away from your business; it accumulates cases somebody thought of in a workshop. A benchmark that grows from incidents converges on your actual risk surface, because every addition is something that provably went wrong here, with real consequences.

That's a very different asset — and it isn't buildable by anyone else, because they don't have your incidents.