The economics of evaluation

Evaluation gets funded badly because it's usually justified badly — as risk reduction, which is real but unfalsifiable, or as best practice, which is not an argument.

Here's the case in numbers you can actually construct.

What it costs

Four line items, and the shape surprises people.

Expert time. The irreducible cost. A structured specification interview, plus writing known-correct answers for the hard cases, plus annotation of a calibration set. For one workflow this is typically days of one senior practitioner, not months — but it must be your best practitioner, which is what makes it feel expensive. It isn't the hours; it's whose hours.

Task construction at volume. Generating and gating hundreds of cases against a frozen specification. Mostly tooling and compute.

Compute for runs. Every model × every task × every seed. Straightforward, and cheaper than people fear until you add seeds — which you should.

Engineering. Environment, harness, integration with your systems. The one that overruns, because environments always do.

What it saves

Three savings, in increasing order of size.

Avoided model spend. The cheapest thing an evaluation ever tells you is that the expensive model isn't better on your workflow. That's a per-transaction cost difference, forever, and it usually pays for the evaluation on its own.

Avoided data spend. The larger one. The default data purchase is a volume — fifty thousand examples of roughly the right shape, aimed at nothing, because at purchase time nobody knows what's missing. A failure map converts that into three specific asks totalling a fraction of the volume. Undirected data spend is the largest wasted line item in enterprise AI, and it's downstream of not having a ruler.

Avoided rework. The largest and the least visible. Shipping a system that fails in a way nobody tested for costs a rollback, an incident, remediation of whatever it did while live, and — the expensive part — the internal credibility of the whole programme. Organisations that have one bad AI incident don't run one fewer project. They slow everything down for a year.

The number that actually matters

Not the cost of the first workflow. The cost of the second.

If your second workflow costs roughly what the first did, you bought a project. If it costs a fraction, you built an instrument — and the difference compounds.

What carries over: the environment, the harness, the graders and calibration machinery, the provenance and audit layer, and — most valuably — the organisational knowledge of how to run a specification interview and get a usable rubric out of it. That last one is a skill, and the second time is dramatically faster than the first.

What doesn't carry over: the specification, the correct answers, the domain judgement. That's the irreducible per-workflow cost, and it's the expert-hours line.

So the shape is a large fixed cost and a small marginal one. Which means the entire economic question is: how many workflows will you evaluate? If the answer is one, the arithmetic is genuinely marginal and you should be honest about that. If it's five, it isn't close.

Framing it for a finance conversation

Three things make this land, and none of them is "we should measure things."

Give the ranked data asks. "We need a data budget" is a request. "Three asks, ~1,600 examples total, estimated to move pass rate from 76% to the high 80s, ranked by expected lift" is an investment case with a hurdle rate.

Quote the cost of a wrong decision. Every workflow worth automating has one — a wrong approval, a missed exception, a mis-coded claim. Get the number from the business, not from engineering. Then the accuracy discussion is denominated in the same currency as everything else on the page.

Show the regression gate as insurance. Providers update models. Without a gate, you find out from users. With one, you find out from CI. That's not an accuracy argument, it's an operational continuity argument, and it lands with a different set of people.

Two honest caveats

Evaluation cannot make a bad use case good. If the workflow is a poor fit for automation, a rigorous evaluation will tell you so faster and cheaper than a deployment will. That's a real saving and it doesn't look like one on any dashboard.

There's a point of diminishing returns, and it arrives sooner than enthusiasts admit. Precision improves with the square root of your sample: going from 400 tasks to 1,600 quarters your uncertainty and costs four times as much. Decide the size of difference that would change a decision, buy exactly that resolution, and stop. Precision nobody will act on is just cost.

The one-line version

Evaluation is a fixed cost that buys a variable one down. It's worth it exactly when you'll do it more than once — and almost every organisation deploying AI will do it more than once, whether or not they've admitted that to themselves yet.