Benchmark 001, losses included

On 14 October 2026 we published our first benchmark. It is small: one model, one task, two assets. We published it in full anyway, because a company that asks you to trust its models should show you how it tests them, including when the result is not the one it hoped for.

The task

The model is a time-series forecaster built for our first client, which trades digital assets. Our first client shares founders with Eigna, and we say so up front. From 512 hours of hourly price history for BTC and ETH, it forecasts the next 24 hours as a range, not a single guess: 64 samples per forecast, so it can be scored as a distribution.

We started from an open base model, Kronos-small: 24.7 million parameters, MIT licence, frozen. On top of it we trained small adapters, four runs with different seeds, on one rented H200 GPU.1

The test

The test was fixed before the model saw it: pinned data, windows, scoring code and seeds. It covers 243 held-out days, 5 February to 12 October 2026, that the model never trained on. Days are scored in pairs, and blocks of days are resampled 10,000 times to see whether a gap is real.2 The main measure is CRPS, which scores the whole forecast distribution against what happened. Lower is better.

The win

Against its base, the model improved in every run. CRPS fell from 1.750% to 1.439%, which is 17.8% lower, with a day-level paired p of 0.006. The average miss, MAE, fell from 2.150% to 1.872%, 12.9% lower. Each of the four runs beat the base on both.

The losses, in the same breath

Three simple baselines still score better. Last value, which assumes tomorrow looks like now, scores 1.392%. EWMA scores 1.398% and GARCH(1,1) 1.407%. Ours scores 1.439%. The gaps are small and not significant, and they are not a win.

The forecast bands are too narrow. A band from the 10th to the 90th percentile should hold about 80% of outcomes. Ours held 57.3%: the model is too sure of itself. A quick calibration fix overshot to 88.3%, outside the band we had written down in advance.

And better forecasts did not make better trades. With the same pre-registered paper rules on the same 243 days, the base model returned +15.0% and ours +4.4%. The difference is not significant, and it is not a win either.

Out of sample, the picture is closer. In replay 018, 308 windows from 3 to 22 October, a newer version with a shrinkage head ranked 3rd of 9 forecasters, just behind Toto 2.0 and GARCH(1,1). None of the gaps was significant. A tie is not a win either.

What the gate did

Our release rule says a new version goes live only when it is strictly better, and no worse on calibration. This one was not, so it was held at the gate, and so were the two versions after it. Nothing untested reached our client’s desk. Case 001 tells that story in full.

What we learned

• Domain training moves a small base. Two hundred adapter steps cut forecast error 17.8%, in all four runs.
• Beat the baseline, not only the base. Three one-line rules still win. Compare with the simplest first.
• Calibration is part of accuracy. Bands that hold 57% of outcomes overstate certainty.
• Better forecasts are not better trades. The rule that turns a forecast into a position needs its own test.

Why publish a loss

Secure, private, enterprise grade: everyone says it, and buyers stopped believing adjectives. What a bank, a hospital or a ministry can check is a metric, a baseline and a date. If we only showed the 17.8%, you would have no reason to trust it. Next to the baselines it still trails, you can see exactly what it is worth.

This is an internal diagnostic on two assets, not a product claim. Every number, the run list and the digests to reproduce it are public at benchmarks.eigna.ai, and our report lays it out page by page.

First on the list now: beat the simple baselines. Benchmark 001 does not yet. When a version does, you will read it here, with the losses that remain.