The place for trusted benchmarks and quality data.
One sentence in → a verified, reproducible agent environment, benchmark, and training data out. You supply the sentence and one expert’s time; a structured interview does the specifying. Twelve steps, each gated. Everything signed — including the instant the world was frozen at.
How it works, end to end.
One enterprise question — can an agent underwrite a mortgage? — carried the whole way, a stage at a time: interview, spec, critique, freeze, build, calibrate, run, review, iterate, certificate. Two of those stages are refusals, and they are not failure states. They are the reason the signed result at the end means anything. It plays on its own; the same twelve stages are written out below.
Worked example · end-to-end mortgage underwriting · all figures are specimen data from a dogfood run
Twelve steps. No shortcuts.
Every engagement runs the same spine, E0 through E11. Each step must clear its gate before the next one opens.
- E0BriefOne sentence: the task, the domain, what the model must do.
- E1InterviewStructured questions turn intent into measurable requirements.
- E2SpecRubric, tiers, tool surface, and budget — drafted as a contract.
- E3FREEZESubjective rubric terms are refused. The spec hashes to a sha256 fingerprint.
- E4BuildTasks, oracle solutions, and graders generated against the frozen spec.
- E5CertifyEvery task's oracle must earn full marks — and its grader must resist cheating.
- E6CalibrateLLM judges must clear Wilson-bounded agreement floors against human annotations.
- E7RunModels execute inside sandboxed environments; every step is logged.
- E8ReviewHumans audit transcripts and verdicts in campaign workbenches.
- E9AlignJudge and human verdicts are reconciled; floors re-checked on holdout.
- E10ExportTraining data and reports, sliced and fingerprinted.
- E11CertificateA signed attestation anyone can verify — even offline.
The refusals are the product.
Most platforms promise quality. Gradia refuses to proceed without it. These are real responses from the platform, not illustrations of intent.
Refused: 2 subjective terms in rubric.
A spec that can be argued about cannot be frozen. Fix the rubric, then freeze.
Run refused: budget does not cover the estimate.
- estimated_cost_usd
- 142.50
- budget_cap_usd
- 100.00
- shortfall_usd
- 42.50
The exact shortfall, before a single token is spent. Never a surprise invoice.
Wilson 95% lower bound vs. human annotations.
wilson_lb_exact 0.805 ≥ floor 0.70 ✓ — judge activated
Below the floor, the judge does not run. Agreement is proven, not assumed.
Oracle must score 100%. Cheats must score 0%.
- oracle solution
- 24/24 · 100%
- cheat probes
- 0/6 · 0%
- verdict
- certified
If the reference answer can't earn full marks, or the grader can be gamed, the task never ships.
A task may not see past the instant its world was cut at.
- as_of
- 2026-03-14T21:00:00Z
- latest record
- 2026-03-15T08:12:00Z
- blockers
- record_after_as_of
Eleven hours of hindsight is the difference between a benchmark and a crystal ball. The build stops.
The licence permits evaluation. It does not permit redistribution.
- certify
- allowed
- export · list
- refused
- blockers
- redistribution_forbidden_for_export
A merkle root proves what ran without moving a byte. Licensed data stays where its licence says it stays.
LLM judging that earns the right to scale.
Humans annotate in campaign workbenches. Judges shadow them, disagree, and are revised against human reasoning — until holdout agreement clears the floors. Only then does judging scale.
Proof, on paper.
Every certified run ends in a signed attestation. Change one digit of the pass rate and the signature no longer verifies.
What one number is made of · specimen data
Benchmark Attestation
cert_9f3ba21c · specimen with demo data
- sha256:7c1e9a04d2…f31b
- wilson lb 0.805 · κ 0.86 · n 200
- run_5d2c…a9
- rep_c4f1…7e
sealed
Incident response, CI/CD pipelines, and infrastructure work in a live shell.
Earnings analysis, credit memos, and portfolio work against real documents.
Contract review, discovery response, and compliance workflows.
Environments that can do anything.
Agents work in real tool surfaces — shells, editors, browsers, full computers, databases, and your own enterprise systems.
The tools a spec declares enter its fingerprint — swap one and the benchmark is a different benchmark. Custom enterprise integrations are certified against offline stubs, so nothing touches production while the tasks prove themselves.
Benchmarks with a date on them.
The questions enterprises actually want answered are time-indexed. Given this morning's filings and the current curve, does this credit still clear our overlays? A synthetic fixture can't ask that credibly, and a live API call during the episode destroys reproducibility outright.
Both are avoidable, because capture and replay are separable. The network is touched exactly once — by us, in the authoring plane, at a named instant. The agent reads a frozen capture through an ordinary tool and has no egress at all. The world is as frozen as it ever was. What's new is that it has a date, and the date is signed.
Did the model get worse, or did the world get harder?
- recipe_hash
- identical · 4a91…c07d
- vintage_digest · mar
- e21f…88b3
- vintage_digest · jun
- 9c40…1de6
- pass rate
- 0.90 → 0.81
- attributable to
- the world, not the model
Hold the recipe identical, move the capture forward, and a score change decomposes. Every buyer with an agent in production has this question and nowhere to take it.
- V1No live egress at run time — the agent reads the capture, never the network.
- V2No lookahead — not one record may postdate the instant the world was cut at.
- V3Point-in-time faithfulness — a restated source is not what was knowable then.
- V4Provenance and rights — unlicensed data may be certified, never exported.
- V5Replay determinism — read the vintage twice; the bytes must match.
Each is a 409 with named blockers, refusing at build, at export, and at the listing door. Each was mutation-verified: stubbed out in testing to prove a named test catches its absence.
Four answers, off one ruler.
A benchmark is not the deliverable — it's the instrument. What an engagement actually produces is a decision, a diagnosis, a shopping list, and a way to check whether the fix worked. It can start from a single file.
One file → an ecosystem · specimen data from a dogfood run
Which model should I use?
Bake-offEvery candidate runs the identical fingerprint. Per-episode model pins, confidence intervals, one ruler.
Where does the winner still fail?
Failure mapClusters by the gate that refused, plus a behavioral taxonomy of what the agent actually did wrong.
What data would fix it?
Ranked asksThe failure clusters become a prioritized data request, not a purchase by the pound.
Did the fix hold?
Honest deltaRe-run the same fingerprint after your fine-tune. Same ruler, so the difference is the model.
Undirected data spend is the largest wasted line item in enterprise AI, and it is downstream of one thing: you cannot aim without a ruler.
Built for two kinds of builders.
Your benchmark, your data, verified end-to-end.
- —Evaluate the tasks your business actually runs, against your own systems.
- —Every artifact fingerprinted, every event on the hash-chained audit trail.
- —Deploy where you need it: laptop → VM → bring-your-own compute cells.
Author, certify, publish — and sell eval access.
- —Build benchmarks on the same spine enterprises use, with the same gates.
- —Buyers run evals and get signed results — the test set never leaves the platform.
- —Your certificate is your storefront: provenance, calibration, and floors on display.
Frontier models on the hardest tier of Agents’ Last Exam, an expert-authored evaluation spanning 55 professional subfields.
Snorkel AI, 2025 · not our benchmark
The gap between a demo and a job is not a rounding error. When people who actually do the work write the questions, the models that clear consumer benchmarks stop clearing anything — which is the whole argument for expert-authored evaluation, and it is not ours to claim.
Breadth across 55 subfields is a different bet than the one we are making. Ours is depth and provenance in one: your job, your data, your experts, frozen at a named instant, with every refusal on the record and a signature over the result. A number that travels across a whole profession tells you the frontier is far away. It does not tell you whether an agent can underwrite your mortgages — and no benchmark written by strangers ever will.
Agents went to production. The rulers didn't.
Evaluation was a research chore while agents were demos. It became a procurement problem the moment they started touching real files, and the instruments didn't move with them.
- —Agents are being handed real work — underwriting files, discovery sets, production incidents — and approved on numbers borrowed from generic public benchmarks.
- —Published test sets enter training corpora within months. A leaderboard position is a claim nobody outside can falsify.
- —Procurement and audit have started asking for evidence of eval quality. There is no accepted format to hand them.
- —So models get picked on vendor reputation and training data gets bought by the pound.
- —Gradia does not train foundation models. The venue cannot also be a contestant.
- —Refusal can't be retrofitted. A tool that won its users by scoring anything cannot start telling them no.
- —There is no bypass — no skip-gate flag, no manual issuance. A certificate means the same thing whoever is holding it.
- —Certificate history, judge-agreement records, and registry roots committed before a model shipped are accumulated, never copied.
And when a pilot can't clear a calibration floor, what ships is a deterministic-gate-only benchmark and a rubric revision plan. The certificate date moves; the bar doesn't. A permissive platform in the same position ships the bad number.
Trust, itemized.
A certificate is only as strong as the weakest way around it. Gradia has no way around it — no override flag, no manual issuance, no negotiated floor. That is an architectural commitment, not a settings default, and it is the only reason the certificate can mean anything to someone who wasn't in the room.
Fingerprints on every spec, dataset, and eval harness.
An append-only audit trail links every event to the last.
Judges activate only above statistically-proven agreement floors.
A holdout's contents are provable from hashes alone — nothing is published.
Certificates check out anywhere — no call home required.
No skip-gate flag, no manual certificate issuance, no negotiated exception.
Gradia does not train foundation models. The venue cannot also be a contestant — which is why a lab can buy an honest map of its own weaknesses here, and why an enterprise's ranking isn't a vendor's marketing.
Every gate is stubbed out in testing and must be caught by a named test before it's restored. The gates aren't merely present — their absence is detectable.