Is it a fit?Book a call

Test report

Benchmark 001

17.8% lower forecast error than its base model. Not yet ahead of a last value baseline.

Held at the gateRun 14 October 2026. Read again 24 October 2026.
  • 17.8%Lower forecast error than its baseCRPS, day-level paired p 0.006
  • 0 of 3Simple baselines beatenLast value, EWMA, GARCH(1,1)
  • 57.3%Band coverageTarget 80%
  • +4.4%Paper tradingBase model +15.0%
01

At a glance

Model
Eigna TLM, a time-series model built for our first client, which shares founders with Eigna
Task
Forecast BTC and ETH 24 hours ahead from hourly history, as a range
Base
Kronos-small, 24.7M parameters, MIT licence, revision 901c26c13326, frozen
Test set
243 held-out UTC days, 5 February to 12 October 2026: 486 windows
Runs
N = 4, finished 13 and 14 October 2026; a fifth single run reported apart
Published
14 October 2026, benchmarks.eigna.ai; read again 24 October 2026

Our first client shares founders with Eigna, and we say so up front. An internal diagnostic on two assets, not a product claim.

02

Sealed first, then scored on days it never saw

The test was fixed before the model saw it: pinned data, windows, scoring code, samples and seeds. Four training runs, a fifth reported apart.

  1. 01SealEach checkpoint is sealed and hashed before scoring.
  2. 02Hold out243 whole days, 5 February to 12 October 2026: 486 windows, BTC and ETH.
  3. 03Forecast512 hours of history in, 24 hours ahead out, 64 samples.
  4. 04ScoreCRPS, MAE and band coverage; direction and rank IC as secondary.
  5. 05TestDay by day, paired; blocks of days resampled 10,000 times; 5% level.

CRPS scores the whole forecast distribution against what happened. MAE is the miss of the central forecast. Coverage is the share of outcomes inside the 10 to 90 band. Lower error is better. Methods

03

Against its base: 17.8% lower

On 243 held-out days, each of the four runs beat Kronos-small on both measures. CRPS 1.439% against 1.750%: −31.1 bp (29.4 to 32.8), 17.8% lower, day-level paired p 0.006. MAE 1.872% against 2.150%: −27.8 bp, 12.9% lower, p 0.017.

Base model, Kronos-small1.750%
Our first training run1.439%
CRPS, lower is better.
Kronos-small, the base2.150%
Eigna TLM1.872%
MAE, lower is better.

Per run: 1.427, 1.443, 1.442 and 1.445% against 1.743, 1.744, 1.745 and 1.769% for the base.

Direction accuracy 54.7% against 51.9%, rank IC 0.095 against 0.086: the intervals overlap, so we do not claim them.

04

Not yet ahead of the simplest baselines

Last value, EWMA and GARCH(1,1) still score lower. In replay 018, an out-of-sample window from 3 to 22 October, a newer version with a shrinkage head ranks 3rd of 9; no gap is significant (p 0.46 to 0.53).

Last valueSimple baseline1.392%
EWMASimple baseline1.398%
GARCH(1,1)Simple baseline1.407%
Eigna TLM, mean of 4 runs1.439%
Kronos-small, the base1.750%
CRPS on 243 held-out days, lower is better. Last value leads ours by 3.5 bp (95% interval −2.7 to 9.5 bp), GARCH(1,1) by 2.0 bp (−4.3 to 8.1). Not significant, and not a win.
RankModelCRPS
1Toto 2.00.748%
2GARCH(1,1)0.751%
3Eigna TLM, 23 October version0.754%
4TimesFM 2.50.765%
5TiRex-20.770%
6Kronos-small0.802%
7Kronos-base0.813%
8Chronos-20.861%
9Last value1.000%
Replay 018, 3 to 22 October 2026, 308 windows, Coinbase and Kraken BTC and ETH spot, CRPS. Replay 016 (1 to 21 October, an earlier version): 0.789%, 5th of 8. Different windows, never pooled.
05

The bands are still too narrow

A 10 to 90 band should hold about 80% of outcomes. Ours held 57.3%: the model is too sure of itself. Split-conformal calibration, fit on 121 days and tested on 122, overshot to 88.3% (BTC 88.9%, ETH 87.7%), outside the pre-registered 76 to 84%, and lost to GARCH(1,1) on volatility (QLIKE +0.486 against −6.493).

Target band 76 to 84%, target 80%
Kronos-small, the base43.2%
Eigna TLMRuns 56.0 to 57.8%57.3%
EWMA76.5%
Last value83.3%
GARCH(1,1)86.6%
Eigna TLM, calibratedOvershot the band88.3%
Share of outcomes inside the 10 to 90 band, 243 held-out days. Target 80%, pre-registered band 76 to 84%. The calibration fix was fit on 121 days and tested on 122.
06

Better forecasts. Worse trades, so far

With the same pre-registered rules on the same 243 days, the base model made more than ours. The rules: $10,000 paper start, half-Kelly sizing capped at 1.0×, 6.5 bps a side, a 3% daily loss stop, a 20% drawdown stop, daily rebalancing.

Kronos-small with the rules+8.5 to +21.5+15.0%
Eigna TLM with the rules−20.2 to +28.9+4.4%
Paper trading on the same 243 days, mean of 4 runs, with 95% intervals. Difference −10.6 points (−31.9 to +10.7): not significant, and not a win. Paper only.
07

Held at the gate. Not released

The release rule: strictly better on the sealed test and no worse on calibration; a tie keeps the current version. This candidate trailed the simple baselines and failed calibration, so it was held, and so were the two after it. No release was issued.

Release gate

Candidate 1 against the best simple baseline

Held
Best simple baseline, last value1.392%
Candidate 1, mean of 4 runs1.439%

CRPS on 243 held-out days. Lower is better. Held: 3.4% higher error.

Eigna TLM, Case 001
Release record

  • Candidate 3

    23 October, shrinkage head

    Held

  • Candidate 2

    14 October, calibrated

    Held

  • Candidate 1

    14 October, LoRA, 4 runs

    Held

  • Release 1

    Not issued

    No release
08

What we learned

  1. 01Domain training moves a small base17.8% lower forecast error from 200 adapter steps, in all four runs.
  2. 02Beat the baseline, not only the baseThree one-line rules still win. Compare with the simplest first.
  3. 03Calibration is part of accuracyBands that hold 57% of outcomes overstate certainty. A quick fix overshot to 88%.
  4. 04Better forecasts are not better tradesThe decision rule needs its own test before money depends on it.
  5. 05Bigger is not the shortcutKronos-base, four times the size, scored worse than Kronos-small in replay 018.
09

Every number can be checked

How to quote this benchmark

Our first training run cut forecast error 17.8% against its base model. It is not yet ahead of a simple last value baseline, and we publish that too.

Reproduce it

Harness code
ff2dbe2ce0f0
Data snapshot
ffad6145ab0d
Held-out windows
063b9a8d30ff
Weights lock
8f70a83226e9
Base revision
901c26c13326
Runs
28d016c2f209, cb9610f490f2, 8300770339a0, 4046a3ab9842
Fifth run
5ece4e970df3
Calibration gate
81ddea423d0d
Paper rules
760ee0f0b719
Bootstrap
10,000 resamples, mean block 3 days; replay seed 42

Limits

  • Two assets, one test period: an internal diagnostic, not a product claim.
  • Same inputs replay byte for byte. The model samples, so scores vary by run.
  • Weights stay private until a version passes the gate. Results are public.
  • Our first client shares founders with Eigna.
  • Paper trading only; no live session has run. Not trading advice.
benchmarks.eigna.ai

Notes

  1. LoRA adapters: rank 16, alpha 32, dropout 0.05, 200 steps, learning rate 2e-4, bf16, on one rented H200 GPU.
  2. Errors as a share of the price at forecast time. Mean of N = 4 runs, 14 October 2026.
  3. Coverage per run 56.0 to 57.8%; the calibration gate was evaluated on 14 October 2026.
  4. Per run, ours: +26.74, −3.11, +1.83 and −8.00%; the base: +18.99, +17.90, +10.80 and +12.14%.

Test Lab

Your proof, on your work.

Four weeks, one task, a sealed test on your own data, and a report you keep, whether the expert wins or loses.

Request a Test LabBook a call