Public benchmarks flatter open models. Your own test does not.

On 16 October, Epoch AI published a look at cyber skills. The best open-weight model, GLM-5.3, scored 160.1 on its cyber index: ahead of a frontier model from March, behind the newest ones. Epoch puts open models six to seven months behind on this kind of work.1

The more useful finding is older and quieter. Comparing 17 benchmarks earlier this year, the gap between open and closed models was 4 to 6 months on public tests and 8 to 10 months on private ones.2 One likely reason: public test items leak into training data, or models get tuned towards them. A test that is published can be learned.

What this means for a company

If you choose a model by its public score, you are choosing partly on what it has already seen. That is fine for a chat assistant. It is not fine for a model that will sort your claims, forecast your demand or draft your filings.

How we test instead

• Sealed before training. The test set comes from your own past work, scored by your own experts, and is kept away from training from the first day.
• Fixed in advance. Data, windows, scoring code and seeds are pinned before a model sees anything.
• Against a baseline, not only a base. A model has to beat what you do today, including the simplest rule that works.
• Published, losses included. Our own first test shows a 17.8% gain on the base model and a loss to a last value baseline, side by side.

That last line is Benchmark 001. It is the same discipline a Test Lab brings to your task: one task, about four weeks, a sealed set agreed up front, and a report you keep either way.