Ai2 releases StateBench, a four-level benchmark for long-context state tracking; frontier models fail on harder levels as inputs grow

Ai2 released StateBench, a benchmark that tests how well models keep track of a changing state over long inputs, at four difficulty levels (r-trivial, aperiodic, soluble, general finite). It comes with a tech report, data and code. By Ai2's own figures, open models such as Olmo 3 and Olmo Hybrid do well on the easiest level, lose accuracy on the second, and stay at chance on the hardest. Frontier models (Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol and Luna) do well on the first two levels but drop to 13% or lower on the longest hardest problems. In separate tests, 275M-parameter models trained from scratch on sequences of up to 256 assignments were tested out to 64,000. Gated DeltaNet and hybrid designs generalised almost perfectly on the first three levels, while transformers degraded on longer sequences.

This mainly helps researchers and developers choosing or designing model architectures. It gives a way to see which kinds of state tracking a design can handle, which is relevant as recurrent and hybrid models like Gated DeltaNet become more common. For anyone relying on frontier models to follow changing state across very long documents, it is a reminder that reliability on the hardest cases is poor. The impact is limited for now, because this is a research benchmark and not a product change, and it does not tell you how often real workloads look like the hardest level.

All the numbers are Ai2's own, and no independent group has reproduced them yet. The frontier-model failures are on synthetic puzzles at the extreme end, so they should not be read as a general claim that these models cannot track state in everyday tasks. The architecture comparison uses small 275M models trained on one task, and it may not carry over to full-scale models.

1. 01Independent reproduction of the headline results, especially the 13% or lower frontier scores on general finite problems, using the released code and data
2. 02Whether other labs report StateBench scores for their models, or whether the benchmark is added to common evaluation suites, over the next few months
3. 03Whether new or updated frontier models improve on the hardest level, and whether hybrid or recurrent models trained at larger scale keep their long-length generalisation advantage over transformers