The Proof Ladder: what we test next

One benchmark is a result, not a body of evidence. To claim that an owned model does real work better, we have to show it for more than one kind of model, and for more than the model alone: with tools, with a decision model, as a whole chain, and while it keeps learning. The Proof Ladder lays that out.

Six rungs

Each rung compares two things on the same pinned test: the open base, and the Eigna version of it.

1. The model alone. Base model against Eigna expert, on the same pinned test set.
2. With tools. Base plus tools against expert plus tools.
3. The decision model. Before and after tuning: whether to act, how much, or when to ask a person.
4. Expert and decision model together. The base pair against the tuned pair.
5. The whole chain. From model to decision to action, with tools.
6. The whole chain, learning. Retrained when new data arrives, to show what each retrain adds.

Nine families, 54 cells

We run the ladder for nine families of models: time series, language, voice, image, video, robotics, the typed router, 3D and guardrails. Six rungs times nine families is 54 cells.

One cell is done: the time-series model on rung a, which is Benchmark 001. One is partial: the same model on rung b, where paper trading ran on the 243 test days but the live paper session has not, because our compute quota for that machine is zero.1 The other 52 cells have no Eigna result yet.

We publish the empty cells on purpose. They are an honest map of what we still have to prove.

What runs next

The next experiments come straight from Case 001, each with a numeric target. First: calibrate the bands on untouched data, to 76 to 84% coverage with forecast error no worse. Second: a paired, significant win over last value and GARCH(1,1), the simple baselines the model still trails. Then five or more runs on more markets, from 2 to 15.

Rung c comes after the calibration step: a decision adapter on Qwen2.5-7B-Instruct, trained on 972 records a run, has to beat the fixed trading rule on paper. The public suites follow: fev-bench, with 100 tasks, and Chronos Benchmark II, with TimesFM 2.5, Chronos-2, TiRex-2 and Toto 2.0 added to the 243-day table once their licences are checked.

Beyond time series

Each family starts from named open bases: gpt-oss-20b, Qwen3.5 9B and Mistral Small 3.2 for language; Whisper large-v3, Parakeet TDT and Kokoro-82M for voice; FLUX.2 klein 4B, Qwen-Image and Z-Image-Turbo for image; Wan 2.2 and CogVideoX-2B for video; SmolVLA, pi0 and GR00T N1.6 for robotics; Laya 0.42B and Qwen2.5-3B for routing; TripoSG for 3D. Only Kronos-small’s licence, MIT, is verified so far. Every other base is checked before use.

When a cell fills, it fills in public at benchmarks.eigna.ai, with its losses. The ladder cell by cell is in our Proof Ladder report.