The 30-day evaluation

Most evaluation programmes die in planning. Here's a month that produces something real, in an order where each week is useful even if you stop.

Assumes one engineer at maybe half time, and access to one senior practitioner for a few hours.

Week 1 — Write the standard

Day 1–2: the consequence interview. Sit with your best practitioner and ask one question: what is the mistake we would have to explain to someone? Not "what can the model do." Write down four to six answers. Those are your criteria seeds.

Day 3–4: draft the rubric. Turn each into something checkable. "The documentation is reasonably clear" is not a criterion. "All four required documents are present" is. Expect to discover that two of your seeds can't be made measurable as stated — that's the exercise working, not failing.

Day 5: the disagreement test. Take three real cases. Have two practitioners independently write the correct answer, blind to each other. Where they differ, your rubric is underspecified. Fix it.

End of week 1 you have: a written definition of correct that two people applied and agreed on. Most organisations have never had this. If you stop here you've still gained the most valuable artefact in the list.

Week 2 — Twenty cases

Day 6–8: collect twenty real cases. From actual operations, in the state they actually arrived — the bad scan, the missing page. Weight them deliberately: roughly six clean, six thin, six exception, two adversarial. Resist the pull toward clean cases; they're easiest to write and they're where every model already succeeds.

Day 9–10: write the oracles. Your practitioner writes the correct answer for each. This is the expensive part and it's irreducible. Twenty cases is a few hours, not a week.

End of week 2: twenty cases with known-correct answers, weighted toward where the risk lives.

Week 3 — Make it run, and check it

Day 11–13: automate the run. Whatever harness you have. Get all twenty executing against your current system, with the full transcript logged — not just the final output. The transcripts are where the findings live.

Day 14: the two checks. This is the day that separates a real evaluation from a plausible one.

• Run each oracle through your own grader. Does it score full marks? If not, your rubric is asking for something the correct answer doesn't do. Fix the rubric.
• Write three cheat probes per task — the empty answer, the confident restatement, the keyword stuffer. Do they score zero? If any scores above zero, your grader will reward that behaviour in every model you ever test.

Expect failures here. Everyone has them. Published, peer-reviewed benchmarks have them.

Day 15: measure your noise floor. Run the identical configuration three times. The spread is the number that makes every future comparison interpretable, and almost nobody has it.

End of week 3: a running suite whose grader has been checked, and a known noise floor.

Week 4 — Learn something, then decide

Day 16–18: read fifty failures. Not the summary — the transcripts. For each, ask whose failure it was: model, task, grader, or environment. Teams doing this for the first time are routinely surprised by how much of their "model failure" is specification failure.

Day 19–20: cluster and count. Group the model failures by which criterion refused them, not by topic. Now each cluster names its own cause, is countable, and maps to a line of your rubric — so you can tell whether it's a data problem, a prompt problem, a tool problem, or a spec problem.

Day 21–22: run one comparison. Your current system against one alternative. Same tasks, same rubric, same everything else. Report intervals, not point scores. If they overlap, say so — "we can't yet distinguish them" is a legitimate finding that moves the decision to cost and latency.

Day 23–25: write it up. Four things: the failure clusters with volumes, the comparison with intervals, the noise floor, and what you'd need to resolve the largest cluster.

What you have after 30 days

• A written, tested definition of correct
• Twenty verified cases weighted by consequence
• A grader that has passed the oracle and cheat checks
• A measured noise floor
• A ranked failure map with volumes
• One defensible comparison

That's not a mature practice. It's stage 2 going on 3, and it's enough to make a decision defensibly and to know what the next investment should be.

What to do in month two

Whichever of these your month one made obvious:

• Failures are mostly specification problems → another rubric round, more cases per criterion.
• Failures are mostly model problems → scale the case set for resolution, then commission expert data against the largest cluster.
• Your noise floor is too high to see the differences you care about → more tasks, lower grading temperature, more seeds. Fix the instrument before running more comparisons.
• Human grading is now the bottleneck → calibrate a model-based judge against the humans you already have.

The one-line version

You don't need a platform to start. You need one practitioner for a few hours, twenty real cases, and the discipline to check that your own grader accepts the right answer and rejects a cheat.

Everything else in this discipline is scale.