EverydayBench released: 77-task live-web benchmark for AI agents; Claude Opus 5.5 and Fable 5.1 tie for top strict pass rate at 54.55%
Edited by Elliot Marsh, Editor · written with AI from the linked sources (how)
A new benchmark, EverydayBench, was introduced with 77 human-authored tasks across eight everyday life and work domains. Success criteria are set before execution, and scoring combines deterministic constraint checks, verification of claims against agent-cited webpages retrieved at evaluation time, and rubric-based deliverable quality assessment. Across 23 agent-model configurations, Claude Opus 5.5 and Claude Fable 5.1 tie for the highest strict pass rate at 54.55%. A GPT-6 Sol effort sweep ranges from 29.87% (low) to 45.45% (max). Per-task-type leaders differ: GPT-5.6 Sol on lookup, Fable 5.1 on planning, Opus 5.5 and Fable 5.1 tie on comparison, GPT-6 Astra and Sonnet 5.5 tie on customized deliverables.
Entities: EverydayBench, Claude Opus 5.5, Claude Fable 5.1, Claude Sonnet 5.5, GPT-6 Sol, GPT-5.6 Sol
0 primary
What happened
A new academic paper (arXiv:2611.25146v1) introduces EverydayBench, a benchmark of 77 human-written tasks across eight everyday life and work areas. Success criteria are fixed before each run. Scoring combines automatic constraint checks, checking the agent's claims against the web pages it cited (fetched at evaluation time), and a rubric for the quality of the final deliverable. Across 23 agent-model setups, Claude Opus 5.5 and Claude Fable 5.1 tie for the best strict pass rate at 54.55%, which is 42 of 77 tasks. A GPT-6 Sol effort sweep runs from 29.87% (low) to 45.45% (max). The best model differs by task type: GPT-5.6 Sol on lookup, Fable 5.1 on planning, and ties elsewhere.
Why it matters
For teams choosing or building web agents, this is a useful extra data point. It tests whether an agent's answer is backed by pages it actually cited, and whether the result is usable, rather than only whether it finished. The headline finding is that even the best setups fail roughly 45% of everyday requests under strict scoring, so full unsupervised delegation is still risky. The task-type split suggests no single model is best across the board. Impact is limited for now: it is one preprint with a small task set and no sign yet of independent replication or adoption.
What is noise
A tie at the top of 77 tasks is not a meaningful ranking, as one or two tasks would change the order, and per-task-type leaders rest on even fewer tasks. The claim that existing benchmarks lack this combination is the authors' own framing. Live-web evaluation also makes results harder to reproduce, because pages change. The paper's authors designed the criteria and rubrics, and we do not know how much of the grading uses model judges.
Watch next
- 01Whether the authors release tasks, rubrics and grading code, and whether independent groups reproduce the 54.55% figures within a few points when run on later dates against changed live pages.
- 02Whether the benchmark expands well beyond 77 tasks, with confidence intervals reported, so that rankings between top models can be separated from noise.
- 03Whether model labs or agent vendors start citing EverydayBench scores in release notes, and whether the leaders change when newer model versions are tested.
Coverage
1 storyNewsletter
Get the no-hype AI news straight to your inbox
Weekly. No spam, unsubscribe any time. Privacy
More capability signals
Full feed →- Deepseek releases V4.1-Flash, an open-source model that sharply cuts KV cache memory and input-processing compute for AI agents10 Sept 202682
- Mistral releases Mistral Large 4, a ~1T-parameter open-weight multimodal MoE model, with day-0 Hugging Face support29 Oct 202680
- Meta releases Muse Spark Open, an open-weight ~816B-parameter multimodal mixture-of-experts model, with Hugging Face support15 Oct 202680
- OpenAI discloses sandbox-escape and credential-leak incidents, confirms pause on tool-use for its most capable models26 Sept 202680