Running human annotation that doesn't fall apart

Every evaluation eventually depends on people saying what correct looks like. That's the ground truth everything else is calibrated against — and it's the part most likely to be done casually.

Annotate blind, always

If annotator two can see annotator one's verdict, you have one annotator with extra steps. Agreement becomes anchoring and the number you compute from it is meaningless.

Blind annotation costs nothing and it's the single most common corner cut, usually because the tooling made it easy to see previous verdicts.

Same applies to the model's answer. If your expert sees what the system produced before deciding what correct looks like, you're measuring plausibility, not correctness. Write the oracle before showing anyone the output.

Two annotators minimum, and use the disagreements

One annotator gives you a label. Two give you a measurement of how well-specified your task is — which is more valuable, because it's the thing you can act on.

When two qualified people disagree, you have exactly two possibilities:

• The rubric is ambiguous. Fix the criterion. This is the common case and it's free information.
• The judgement is genuinely contested. Legitimate professional variation. Then say so in the specification and score it accordingly — perhaps as a range of acceptable answers rather than one.

What you must not do is have a third person break the tie and move on. That converts a signal about your specification into a number, and buries the finding.

Measure the ceiling before you chase it

Before building anything, have three experts independently handle a sample of cases and compute their agreement with each other.

That number is a ceiling. If three qualified people agree with each other 65% of the time on a class of case, no automated grader can meaningfully exceed 65% on it. A judge reporting 90% isn't better than your experts — it's more consistent than the concept is, which means it has quietly picked a side and stopped telling you.

Teams that skip this step spend months trying to push judge agreement past a wall that was never a judge problem.

Spend the hours where the judge is unsure

Random sampling is statistically clean and enormously wasteful. Most randomly-drawn cases are ones the judge was already confident and already right about — you paid a senior practitioner to confirm something you knew.

Stratify by the judge's own uncertainty. Sample thinly where it was confident and agreements are dense; heavily where it was uncertain or where judges disagreed. Weight the results back and you get the same statistical guarantee for a fraction of the effort.

The compounding benefit: the cases you oversample are exactly the ones that reveal where your rubric is still ambiguous. Stratified sampling doesn't just calibrate more cheaply — it generates a better queue of specification problems.

Keep a small random sample alongside it, though. Everything else is selected, and selected samples will show you problems while never telling you the base rate.

Design the session, not just the task

Annotation quality decays measurably within a session. Some practical constraints that cost nothing:

• Cap sessions. Ninety minutes is generous. Agreement in hour three is not the same as hour one and you won't be able to tell from the data.
• Randomise order across annotators, so any fatigue effect doesn't align with any particular subset of cases.
• Seed known-answer cases through the set — a handful where the correct verdict is unambiguous. If someone's accuracy on those drops, you know when to discard the tail of their session.
• Give them somewhere to say "the rubric doesn't cover this." Without that escape hatch, ambiguity gets resolved silently by whichever interpretation the annotator picked, and you never hear about it. That free-text field is often the most valuable output of the whole campaign.

Record who, when, and against what

Every verdict should carry the annotator, the timestamp, and the rubric version it was made against.

That last one matters more than it looks. Rubrics change. A verdict made against version 0.3 is not evidence about version 0.4, and if you can't tell them apart your calibration set silently becomes a mixture of standards. Re-certify against the new version or mark the old verdicts as superseded.

Treat annotation as a recurring cost, not a project

The instinct is to do one big campaign and be finished. It doesn't hold, for three reasons: your input distribution shifts, your rubric evolves, and your judge drifts as providers update models underneath it.

Budget a small, regular re-annotation — a monthly or quarterly sample — rather than a large one-off. It's cheaper in total, it catches drift while it's small, and it keeps the muscle alive. Teams that annotate once discover eighteen months later that nobody remembers how, and the rubric has three undocumented interpretations.

The one-line version

Your annotators aren't producing labels. They're producing the standard everything else is measured against — and the disagreements between them are worth more than the agreements.