Stanford study finds 44% of 50 widely cited LLM findings no longer hold on 2024-2026 models

Stanford researchers Owen Queen and James Zou re-tested 50 widely cited claims about LLM behaviour on 20 models from eight developers, released between July 2024 and September 2026. They report that 44% of the findings have expired, 34% have weakened significantly, 16% still hold and 6% have strengthened. For the 22 findings where an expiry date could be estimated, the average lifespan was 15.7 months. Fabricated citations are one example: they fell from 55% (GPT-3.5, 2023) to 0% for GPT-6.1 Sol and 1% for Claude Sonnet 5.5, while Qwen3.7-Flash still fabricated 69%.

Anyone who relies on published LLM weakness findings, such as researchers, developers and enterprise risk teams, should treat them as observations about specific models at a specific time. Reasoning weaknesses expired fastest, so older claims in that area are the least safe to reuse. Evaluator bias, sycophancy and left-leaning political bias persisted, so those risks still need testing. The practical advice is modest and sensible: record model, version and date, and publish prompts and code. The impact is on research and evaluation practice rather than on what products can do.

The "development is outrunning research" framing is an interpretation, not a measured result. The 15.7-month lifespan covers only the 22 findings with an estimable expiry date, not all 50. The 50 claims were chosen by the authors, so the percentages may not generalise to LLM research as a whole. The paper is not linked in this coverage, so these figures come second-hand from The Decoder and I could not check them against the paper. "Expired" also depends on how the authors defined it and which models they tested.

1. 01Whether the paper is posted or peer reviewed, with the full list of 50 claims, the definition of 'expired' and the code, so the 44% and 15.7-month figures can be checked independently.
2. 02Whether other groups replicate the re-testing with different claim selections or models, especially the persistent findings on sycophancy, evaluator bias and political bias.
3. 03Whether major venues, benchmark maintainers or model cards adopt the recommended reporting of model, version and date, with prompts and code, over the next 6 to 12 months.