Should you build your evaluation in-house?

"Build or buy" is the wrong shape for this question, because an evaluation practice isn't one thing. It's six, and the right answer differs for each.

The six components

1. The specification — what "correct" means for your workflow.

2. The task set — the cases, with known-correct answers.

3. The environment — the documents, systems, and tools the agent works in.

4. The graders — deterministic checks plus model-based judging.

5. The harness — running episodes, sandboxing, logging, replay.

6. The record — provenance, versioning, audit trail, attestation.

Now, honestly, for each.

What you must build

The specification is not outsourceable. No vendor, tool, or model can decide what correct means at your company. Your overlays, your exceptions, your escalation boundary, your documentation standard — these exist because someone in your organisation learned something, usually expensively. A vendor can facilitate, structure, and challenge the specification. They cannot originate it.

This is the irreducible cost of the whole discipline, and it's smaller than people expect: a few hours of your best practitioner, done properly, in a structured interview.

The correct answers are also yours. Someone who could do the job has to say what the right outcome is on your hard cases. Again, facilitation helps enormously. Substitution doesn't.

That's the build side, and it's mostly measured in expert hours rather than engineering.

What you probably shouldn't build

The harness. Sandboxing, deterministic replay, per-episode version pinning, byte-identical reruns, cost accounting. This is real distributed-systems work with no domain content in it, it takes a small team months, and it looks finished long before it is. The failure mode is subtle: a harness that mostly works produces numbers that mostly mean something, and you won't know which ones.

Judge calibration infrastructure. Not the judgement — the machinery. Annotation campaigns, agreement statistics with proper confidence bounds, stratified sampling, drift monitoring, floor enforcement. Every team that builds this builds the same thing, badly, twice.

The record layer. Hash-chained trails, signing, offline verification, licence enforcement on export. Nobody's competitive advantage lives here and getting it wrong is expensive in exactly the moment you can least afford it.

What genuinely depends

The task set at volume. The first eleven cases must be yours. Scaling to four hundred against a frozen specification is generation plus gating, and that's tooling. The question is whether you'd rather spend your experts' scarce hours writing case four hundred or auditing a sample of four hundred. The second is a better use of them.

The environment. If your workflow lives in common systems, this is largely configuration. If it lives in a thirty-year-old internal platform with no test instance, you're building regardless — and that build has value beyond evaluation.

Graders. Deterministic checks are cheap and yours. Model-based judges are cheap to write and expensive to trust, and the trust is the part that isn't cheap.

The test that actually decides it

Not "can we build this?" You can. Ask instead:

Will the second workflow cost less than the first?

If yes, you're building an instrument, and the investment amortises across every model, vendor and workflow after this one.

If each new workflow costs roughly what the last one did, you're not building a platform. You're doing consulting work with your own staff, at your own fully-loaded cost, without the option of stopping.

Most in-house builds fail this test — not on the first workflow, which usually goes well, but on the fourth, when it becomes clear that nothing generalised and three engineers are now permanently maintaining a bespoke harness that only their company uses.

The independence problem

One consideration that has nothing to do with cost.

If you build it, run it, and report the result, one party fills three roles: rule-maker, examiner, candidate. That's fine for internal decisions where everyone shares an interest in the truth.

It stops being fine the moment the result has to travel — to a regulator, an acquirer, a board that's sceptical, or a customer's procurement team. At that point "we measured ourselves and we're pleased" is not evidence, however rigorous you actually were. Nobody outside can distinguish an honest self-assessment from a flattering one, and neither can you.

The practical shape most organisations land on: build the specification and the judgement, buy the machinery, and get the attestation from somewhere that isn't you.

The one-line version

Build what only you know. Buy what everyone needs. And whatever you do, make sure the second workflow is cheaper than the first — because if it isn't, you didn't build a capability, you completed a project.