The release gate
No update ships unless it beats the last one.
Four tests stand between a candidate and your people. A tie keeps the version you have.
Release gate
Release 13 against release 12, sealed test set
Live, release 12100.0
Candidate, release 1398.6
Forecast error, indexed to the live version. Lower is better.
Illustrative data. The live version only changes when a release beats it.
- Sealed test items per team, a pilot default
- 200+
- Trials for every item
- 3
- Held-out days in Benchmark 001, never seen in training
- 243
- To roll back to the last good release, by design
- 60 s
The four tests
In order, every time.
- 01Sealed testsReal tasks, scored by your own experts, kept apart from training. 200 or more items per team, three trials each.
- 02Held-out weeksTested on weeks the candidate has never seen, so it cannot have memorised the answer.
- 03Paired testBetter quality, proven with paired statistics. No worse on speed, cost or safety.
- 04Shadow runRuns beside the live version on real requests before anyone relies on it.
- ThenPromote or holdA tie keeps the live version.
Pilot defaults, by design. The scoring harness runs today, on our benchmarks.
Measured on every update
Four numbers decide.
- Task accuracy
- p95 latency
- Cost per request
- Refusal rate
Three outcomes
Released, held or rolled back.
- ReleasedBetter on every count. It goes live and the record gains a line.
- HeldNot better, or a tie. Nothing changes.
- Rolled backThe owner goes back to the last good release.
Back to the last good version.
Any time, by the owner, without a call to us. Rollback in 60 seconds is a design target.
Held
Our own gate holds our own work.
Benchmark 001 cut forecast error against its base model but is not yet ahead of a simple last value baseline. At the gate, that version would be held. We publish it anyway.
Start
Ask us how your test set would be built.
Thirty minutes with a founder.