How a release earns its place
A new version of your model goes live only when it beats the one you use today. This is the test it has to pass.
Most software ships when it is ready. A model of your own ships when it is better. That sounds like the same thing. It is not, and the difference is the whole product.
Every Eigna model learns from the work your people approve. When enough approved work comes in, we train a candidate: a new version that might be better than the one you run today. Might. The release gate decides.
Nothing learns without approval
The loop starts with people, not with data. Your team uses its expert, reviews the drafts and corrects them. A correction is not training data until someone approves it, and a training run starts only with at least 50 approved examples for the task.1At least 50 approved examples per training run: a pilot default, by design. It is slow on purpose. The model learns your work as your people judge it, not whatever happened to pass through it.
A test it has never seen
Before a candidate is trained, we seal a test set with you: real tasks from your work, scored by your own experts and kept apart from training. Our design default for a pilot is more than 200 sealed items per team, each tried three times.2Pilot defaults, by design: 200 or more sealed items per team, three trials per item, a paired test. Measured on every update: task accuracy, p95 latency, cost per request and refusal rate. The candidate and the live version answer the same items. Nobody tunes against them.
Release gate
Release 13 against release 12, sealed test set
Forecast error, indexed to the live version. Lower is better.
Illustrative data. The live version only changes when a release beats it.
To pass, the candidate must be strictly better on quality, item by item, in a paired statistical test, and no worse on speed, cost per request or refusals. Better has to be shown, not assumed.
A tie keeps the current version.
Held is an answer too
When a candidate does not pass, it is held. The live version stays, nothing changes for your team, and the record says why. Held is not the system failing. It is the system working.
Our own first gate shows it. In October 2026, three candidates of our first client’s forecasting model met the gate. The first cut forecast error 17.8% against its base model, yet still trailed a simple last value baseline, so it was held. A calibrated version and a newer version were held too. Release 1 was not issued.
Candidate 3
23 October, shrinkage head
HeldCandidate 2
14 October, calibrated
HeldCandidate 1
14 October, LoRA, 4 runs
HeldRelease 1
Not issued
No release
Released, with a record
A candidate that passes runs in shadow next to the live version first, then it is promoted. It gets a line in the release record and a line on the Model Deed: the version, the date and the fingerprint of the weights. If it disappoints later, the owner rolls back to the previous release. Our design target for that is 60 seconds.3Rollback in 60 seconds is a design target, not yet a measured figure.
What runs today
The scoring harness, with pinned data, pinned seeds and sealed tests, runs today on our benchmarks. The gate described here is designed around it. Learning in production, with the loop running for a client, is planned. When it runs, the first learning report will show what the loop tried, released and held.
A release has to earn its place. It is the promise we make, and the one we test ourselves against first.
Notes
- At least 50 approved examples per training run: a pilot default, by design.
- Pilot defaults, by design: 200 or more sealed items per team, three trials per item, a paired test. Measured on every update: task accuracy, p95 latency, cost per request and refusal rate.
- Rollback in 60 seconds is a design target, not yet a measured figure.
