You're six months into an AI programme. Is it working?

Note: the figures in this post are illustrative examples, not measured results.

Here's a position an enormous number of companies are in right now.

You're six months into an AI programme. The pilot went well. A team is using it daily and they like it. Somebody produced a slide with a percentage on it.

And nobody in the building can actually tell you whether it's safe to widen.

Why that happens to competent teams

Look at what the evidence consists of.

The pilot measured enthusiasm — did people find it useful, did they keep using it. That's a real signal about adoption and it tells you nothing about accuracy.

The percentage came from cases somebody selected, and they selected reasonable ones, which means it's a measurement of the easy path.

And nobody has systematically looked at what it gets wrong, because there was never a mechanism for that.

None of this is a failure of the team. Every step was sensible. The gap is that enthusiasm, anecdote and a curated demo are the only instruments most organisations have — and none of them answers the question an executive has to answer.

The gap isn't capability. It's evidence.

The three questions that arrive

When an AI system moves from a team to a company, three questions show up. From a board, a risk committee, an auditor, or a large customer's procurement team — and they're remarkably consistent.

1. How often is it wrong, and how wrong?
2. What happens when it is?
3. How will we know if it gets worse?

Notice what's not on that list. Nobody asks which model you're using. That's the question technical teams prepare for, and it almost never comes up — because the people asking have correctly worked out that it isn't the risk.

All three need a number and a record. None is answerable with a demo, however impressive, because a demo is a sample of one that you chose.

The uncomfortable version: if you can't answer these, the honest position is that you don't know whether the system is safe to widen. Most teams in that position widen anyway, because the alternative is admitting the gap.

What "end-to-end evaluation" actually covers

The phrase gets used loosely. Concretely, it means evaluating the system, doing the job, in your environment. Five parts:

• Your documents, in the state they actually arrive. The bad scan, the missing page, the form somebody filled in wrong. Not clean fixtures.
• Your systems of record, reachable and writable, so the agent has to navigate them rather than being handed a tidy summary.
• Your policy as the definition of correct. Not a general standard — yours, including the overlays and exceptions that make your version different.
• Your edge cases, weighted by consequence rather than frequency.
• The scaffold you actually intend to ship — the retries, the tools, the memory. Frequently the scaffold does more work than the model.

Change any one of those and you're evaluating something different from what you'll deploy. The gap between those two things is exactly where post-launch surprises live.

What the output looks like

This is the part worth being concrete about, because the difference from a score is the entire point.

Not "84%". This:

Four sentences. Each carries three things.

A named cause. Not "it struggles with hard cases" but a specific behaviour somebody can picture.

A volume, so you can rank. The exception failures are nearly three times the rationale failures — that ordering is a budget decision.

An owner. The retrieval failure belongs to engineering. The escalation failure is probably a prompt or policy-access problem. The rationale failure is a scaffold change. The exception failure likely needs data.

Four different fixes, four different teams, one measurement. That's a project plan, and it fell out of an evaluation.

You cannot fix "84%". You can fix these.

Expert data: the fix you can actually buy

Once a failure has a name and a volume, it has a price.

Take the 83 missed policy exceptions. What fixes that isn't more data generally — it's examples of that exception, worked correctly, by someone who understands why the exception exists.

Same for the rationale failures: rationales written the way your auditors expect to read them, which is a house style rather than a general capability. Same for the edge cases: situations your policy manual never explicitly covered, where experienced people apply judgement that was never written down.

Notice what all three have in common. This data does not exist to be bought. It has to be produced by people who do the job. That's precisely why it's worth something, and precisely why generic datasets don't move your number.

It's also why sequencing matters. Buy data first and you're guessing. Measure first and every hour of expert time you commission is aimed at a failure you've already counted.

What you own at the end

Six things:

• A written definition of correct for this workflow, reviewed and signed by your own experts. Most organisations have never had that written down anywhere.
• A test environment that mirrors your systems, so future evaluations don't start from zero.
• Hundreds of cases with known-correct answers — your cases, your policy.
• A grader proven to agree with your people, so future runs don't need them.
• A ranked list of what to fix and what each fix is worth.
• A signed result you can hand to a board, an auditor, or a customer.

Here's the part that changes the economics. The first four are not single-use. They're reusable for every model you evaluate after this, every vendor who pitches you, and every version of your own system. The second workflow you measure costs a fraction of the first, because the machinery already exists.

You're not buying a report. You're buying an instrument — and the report is what it produces the first time you use it.