StateBench measures models' limits with long documents
Allen Institute for AI unveils StateBench, a benchmark evaluating language models' ability to track state changes in long texts. Results show a clear gap between recurrent, hybrid, and transformer architectures.
Allen Institute for AI presented StateBench on November 3, 2026, a benchmark designed to measure how language models track state changes in long documents. The concrete challenge is to determine whether a model correctly keeps track of variables, objects, or characters as information accumulates. In Ai2's tests, recurrent and hybrid models hold up better over length, but transformers, including several state-of-the-art models, falter as dependencies become complex.
Suivre un état, ce n’est pas juste lire jusqu’au bout
The core problem is simple to state but harder to solve: a model can consume a long text without retaining what has changed inside. StateBench distinguishes four difficulty levels, from immutable assignments to updates by reference, where one variable takes the value of another. This hierarchy reflects theoretical distinctions from Krohn-Rhodes theory, an algebraic framework describing how state-tracking problems decompose into simpler primitives.
The researchers aimed to avoid a common bias in evaluations: testing only the hardest cases. Their benchmark thus keeps the same number of variables, distinct values, and update positions, changing only the operation type. The result: observed gaps better reflect the real difficulty of state tracking, not just prompt form. It’s not glamorous, but it’s precisely the kind of detail that makes or breaks a serious evaluation.
Des formats différents, des écarts qui restent
Ai2 tested pre-trained open models, including Olmo 3, a 7-billion-parameter transformer, and Olmo Hybrid, which adds Gated DeltaNet layers. None had seen contexts beyond 64,000 tokens during training, and tests often went well beyond that. In this setting, almost all models succeed on r-trivial tasks, decline on aperiodic as length increases, then remain at chance level on general finite tasks.
One detail matters greatly: problem presentation changes scores. On some formats close to RULER’s, small hybrid or Gated DeltaNet models approach ceiling, while the same logic presented as table edits drops results. In other words, a good score in one format does not prove general state-tracking ability. The benchmark was designed precisely to avoid this pitfall.
Pourquoi les modèles de pointe plafonnent-ils sur les cas durs ?
Tests on Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna show the same trend. All perform well on r-trivial and aperiodic tasks, but their performance drops on soluble and general finite variants as sequences lengthen. On the longest general finite cases, accuracy falls to 13% or less according to Ai2.
The likely reason lies in the problem’s sensitivity: the longer the update chain, the more a single omission distorts the final result. Transformers have a known weakness here, related to difficulty faithfully carrying state through long dependencies. Recurrent or hybrid models retain an advantage on this front when trained directly for it.
Quand l’entraînement direct change la donne
To separate architecture from pretraining effects, researchers trained six 275-million-parameter models on StateBench: two transformers, two hybrids, and two Gated DeltaNet variants. Training sequences had at most 256 updates, but tests reached up to 64,000, 250 times the maximum training length.
In this controlled setting, hybrids and Gated DeltaNet generalize almost perfectly on r-trivial, aperiodic, and soluble levels. Transformers learn easy cases, then degrade beyond their comfort zone, never mastering general finite problems. On the hardest task, the Gated DeltaNet variant with more flexible memory update stays near full score up to about ten times training length. The message is clear: some failures stemmed from pretraining, but not all.
Ce que la publication laisse encore ouvert
Ai2’s paper also presents evidence for a Gated DeltaNet variant with non-negative eigenvalues, capable of solving all aperiodic tasks and some periodic cases, including parity. Parity involves determining if an input repetition is even or odd. However, its exact place in the hierarchy remains open, as does its ability to solve all soluble problems or the effect of coupling with attention layers.
Another acknowledged limitation: StateBench assumes a fixed number of variables. Real documents and conversations constantly add new names, objects, or references to track. Thus, the benchmark measures well the maintenance of a known state but not the arrival of new entities mid-way. For assistants juggling long files or extended exchanges, this nuance matters greatly.
Key points
- StateBench was published on November 3, 2026, by Allen Institute for AI.
- Four levels test increasingly difficult state updates.
- Tested models handled up to 64,000 assignments.
- Frontier models drop to 13% or less on the longest cases.
- Olmo 3 and Olmo Hybrid are among the evaluated open models.
- The benchmark includes text, code, and spreadsheet formats.
By the numbers
- 4 levels — r-trivial, aperiodic, soluble, general finite; StateBench.
- 64,000 assignments — maximum tested length, according to Ai2.
- 13% — accuracy or less on the longest general finite cases.
- 68,000 lines — published dataset size, about 1.05 GB.
- 250 times — gap between training and test length on some models.