Research on LLMs ages faster than the models themselves

A Stanford study shows that 44% of 50 widely cited results on LLMs are already obsolete. Researchers advocate for more dated and better reproducible publications.

Research describing large language models progresses more slowly than the models themselves, according to a Stanford study published on November 3, 2026. Owen Queen and James Zou tested 50 widely cited claims about LLMs against 20 recent models, and 44% of them are already "expired," meaning outdated. The signal is clear: in this field, a laboratory truth can age before it even finds its reader.

Quand une découverte dure à peine un an et demi

Among the 22 results for which the researchers could assign an expiration date, the average lifespan is only 15.7 months. After one year, 16% of the results were already obsolete; after two years, the share rose to 39%. Meanwhile, 34% of the examined conclusions significantly weakened, 16% still hold, and 6% even strengthened.

The evaluated models were published between July 2024 and September 2026 by eight different developers. This diversity does not prevent marked disparities: within the same release window, DeepSeek V4 Pro has 11 expired results, compared to 30 for GPT-6.1 Sol. Neither publication date, publisher, nor model size reliably predicts what will hold up over time.

Ce qui résiste, ce qui s’effrite

Claims about what models "cannot do" wear out the fastest. Five of nine results related to reasoning are considered expired, and none have held or gained solidity. The cited examples are telling: counting the letters of "strawberry," performing arithmetic in unusual numeral systems, losing information in long contexts, or abandoning the correct answer when the user insists.

Conversely, observations about preferences and trends hold up better. Three of eleven results still valid or strengthened concern models as evaluators: they prefer their own answers, rate accepted and rejected articles similarly, or are influenced by how a medical summary is phrased. Others relate to flattering the user, a left-leaning political tendency, nearly identical answers to open questions, and a non-uniform distribution of random numbers.

Les hallucinations de citations ont chuté, mais pas partout

As a case study, Queen and Zou examined fabricated scientific references. In a 2023 study, GPT-3.5 fabricated 55% of its citations. In 2026, GPT-6.1 Sol drops to 0% and Claude Sonnet 5.5 to 1% in the same test. But Qwen3.7-Flash still shows 69% fabricated references, reminding us that overall progress does not erase disparities between models.

The authors also warn against overly optimistic reading. On so-called two-hop questions, where the model must combine two known facts, Claude Sonnet 5.5 still fails in about 10% of cases when it correctly answers the two separate questions. The "Reversal Curse" and contradictory answers to paired time questions have lessened but not disappeared completely. In short, the problem shifts but does not vanish.

Ce que Stanford demande aux chercheurs

The conclusion is as much methodological as scientific: empirical results on LLMs must be presented as observations valid for a specific model, at a specific time, and not as general laws. Researchers recommend indicating the model, its version, and the measurement date, then publishing prompts and analysis code to allow subsequent verification.

This requirement matters as much for the market as for research. Companies, integrators, and product teams often rely on academic papers to choose a model, set safeguards, or justify an architecture. If findings expire on average in 15.7 months, technical documentation must become more dynamic, or it ends up in the museum of good ideas. And in this field, the museum fills up quickly.

Key points

  • 44% of the 50 tested results are already expired.
  • 15.7 months: average lifespan of dated results.
  • DeepSeek V4 Pro: 11 expired results, compared to 30 for GPT-6.1 Sol.
  • GPT-3.5: 55% fabricated citations in the 2023 study.
  • Qwen3.7-Flash: 69% fabricated references in the same test.
  • Owen Queen and James Zou studied 20 models.

By the numbers

  • 50 claims — tested by Stanford on LLMs (publication November 3, 2026).
  • 20 models — evaluated between July 2024 and September 2026.
  • 44% — results already "expired" according to the study.
  • 15.7 months — average lifespan of dated results.
  • 8 developers — represented in the analyzed sample.

To read