Evaluating an agent is a different discipline

Grading a chatbot is comparatively simple. One question, one answer, compare against what should have come out.

An agent is different in kind. It takes many steps. It uses real tools. The world changes underneath it as it works. And there are usually many legitimate routes to the same correct outcome.

Almost every practice that worked for model evaluation breaks somewhere in that gap.

Four things need grading, not one

Did it reach the right outcome? The obvious one, and the only one most evaluations measure.

Did it take a legitimate route? An agent that reached the correct lending decision by opening a record it had no permission to open has failed. Outcome-only grading calls it a pass. In production it's an incident.

Did it stay inside the rules? Guardrails — the things that must never happen — are different objects from quality criteria and need different scoring. More on that below.

What did it cost? Two agents both score 90%. One averages 8 tool calls and 40 seconds. The other takes 60 calls and 11 minutes. The scoreboard says these are the same result. Your infrastructure bill disagrees.

Grade only the final answer and you're blind to three of the four. You'll happily pass an agent that got the right result by doing something your compliance team would never allow.

You're not grading an answer any more. You're grading work.

The scaffold is doing more work than the model

This is the finding that should change how you read every agent benchmark result you see.

The same agent benchmark, run on the same tasks, can produce results tens of points apart depending on the harness — whether the agent gets a full tool stack, retries, a planner, memory, or none of it. The spread between a bare model and a well-built scaffold on identical tasks is frequently larger than the spread between frontier models.

Which means a headline agent number is a claim about a system, not a model. And the system is usually the part nobody describes.

Three consequences:

• Pin the scaffold the way you pin the model version. It's just as load-bearing.
• Report both, because a number without its harness is uninterpretable.
• Change one at a time. Upgrade the model and the scaffold in the same release and you've learned nothing about either.

And for buyers: when a vendor shows you a benchmark result, a large share of what you're seeing may be their engineering rather than the model you're being sold. That's not necessarily bad — engineering is real value. You should just know which one you're buying.

"Model X scores Y on this benchmark" is an incomplete sentence.

The environment is part of the benchmark

Swap the browser. Change the document parser. Add one internal API the agent didn't have before. Upgrade a library that changes how a tool returns errors.

You now have a different benchmark. The tasks look identical. Last month's score is not comparable.

This is obvious when stated and almost never handled, because the environment feels like infrastructure rather than methodology. It isn't. An agent that can't parse a PDF will fail a task about a PDF, and that tells you nothing about the model.

Declare the tool surface explicitly as part of the specification. Pin and fingerprint the environment image. Treat any tool change as a version change to the benchmark itself.

Cost is a score, and an early warning system

Four things worth recording per episode: tokens in and out, tool calls with retries counted separately, wall-clock time, and how often the agent gave up and escalated.

That last one is sneaky. An agent that escalates aggressively can look safe and score well while quietly doing very little. You won't notice until someone checks how much work actually got automated.

There's a diagnostic benefit too. Retry loops, circular reasoning, and an agent thrashing against a tool it doesn't understand all show up in cost numbers long before they show up in the pass rate.

Long-horizon tasks break ground truth

For a short task, "correct" is a single end state. For a task that runs for hours across many steps, there are usually many acceptable end states — and genuinely expert people disagree about which count. Not because anyone is wrong, but because professional judgement contains legitimate variation.

Three things help: grade checkpoints rather than only the ending; specify the required end state rather than the path; and — the one people skip — measure your own experts' agreement with each other before you build anything.

That number is a ceiling. If three qualified experts agree with each other 60% of the time on a class of case, no grader can meaningfully exceed 60% on it. A judge reporting 90% isn't being better than your experts. It's being more consistent than the concept is, which means it has quietly picked a side and stopped telling you.

Sandboxing isn't optional

An agent that can reach the live internet during an evaluation is an agent whose results cannot be reproduced — the world moved between your two runs, so you can never separate a model change from a world change. It's also an agent whose actions cannot be undone, which matters the first time an evaluation writes to a system that mattered.

Four requirements: no network egress during the episode, writes to a copy rather than production, identical bytes on a repeated run, and every tool call logged in enough detail to replay.

Notice that reproducibility and safety turn out to be the same engineering problem. You isolate the agent so the world can't move, and the same isolation means the agent can't move the world. One piece of infrastructure, two requirements satisfied.

If you can't replay it, you didn't measure it. You watched it.