Methods and corrections

1. 01Name the metricIn plain words, with its unit, and whether lower or higher is better.
2. 02Name the baselineThe simplest one first. Beating the base model is not enough.
3. 03Date itThe day of the run and the day we read it again.
4. 04Publish itWhether it wins or loses, next to the win, at the same size.

CRPS

The whole forecast distribution scored against what happened. Lower is better.

MAE

The miss of the central forecast. Lower is better.

Coverage

The share of outcomes inside the 10 to 90 band. The target is 80%.

Direction accuracy

The share of forecasts that call the direction of the move right. Secondary.

Rank IC

How well forecasts order the outcomes. Secondary.

QLIKE

A standard loss for volatility forecasts. Lower is better.

• The test is fixed before the model sees it: pinned data, windows, scoring code, samples and seeds.
• Every checkpoint is sealed and hashed before it is scored.
• Test days are held out whole and never used for training.
• Scores are compared day by day, in pairs; blocks of days are resampled 10,000 times; the level is 5%.
• The harness is deterministic. The model samples, so scores vary by run, and we report N.
• Replays use different windows and are never pooled.

1. 01Golden setReal tasks, scored by the client’s experts, sealed: 200 or more items per team, three trials each.
2. 02Held-out weeksA split in time, so the test comes after the training data.
3. 03Paired testStrictly better on quality; no worse on speed, cost or safety. A tie keeps the live version.
4. 04ShadowThe candidate runs next to the live version before it is promoted.
5. 05Promote or roll backEvery release recorded, and reversible. Rollback in 60 seconds is a design target.

The gate for client models is a design; the scoring harness behind it runs today on our benchmarks.

Harness code

ff2dbe2ce0f0

Data snapshot

ffad6145ab0d

Held-out windows

063b9a8d30ff

Weights lock

8f70a83226e9

Base revision

901c26c13326

Runs

28d016c2f209, cb9610f490f2, 8300770339a0, 4046a3ab9842

Fifth run

5ece4e970df3

Calibration gate

81ddea423d0d

Paper rules

760ee0f0b719

Bootstrap

10,000 resamples, mean block 3 days; replay seed 42

Same inputs replay byte for byte. The model samples, so scores vary by run.

• Two assets, one test period: an internal diagnostic, not a product claim.
• Same inputs replay byte for byte. The model samples, so scores vary by run.
• Weights stay private until a version passes the gate. Results are public.
• Our first client shares founders with Eigna.
• Paper trading only; no live session has run. Not trading advice.

• 24 October 2026. Benchmark 001 read again at benchmarks.eigna.ai. No newer replay; the 243-day table is unchanged.