GPT-6 Astra gets conflicting benchmark scores from Epoch AI and Artificial Analysis, but posts human-level move-efficiency on ARC-AGI-3, prompting Chollet to move up his AGI forecast

OpenAI's GPT-6 Astra was benchmarked by three independent evaluators with conflicting results: Epoch AI ranks it first overall among 267 models, while Artificial Analysis scores it level with its predecessor (Sol) and behind Anthropic's Claude Fable 5.1. On ARC-AGI-3, Astra scored 62.7% on ARC Prize's own harness versus 99.9% on OpenAI's harness, a discrepancy ARC Prize explicitly says is not a fair comparison. It also matched or beat median human move-efficiency on 96% of solved ARC-AGI-3 levels, solved 2 of 68 open FrontierMath Erdős problems, and costs 2.5x more per token than Sol while showing regressions on GDPval-AA v2, SciCode, banking support tasks and long-context handling.

For developers and enterprises choosing models, this is genuinely useful signal: concrete cost, token-efficiency and task-specific performance data that affects procurement and deployment decisions right now. But the contradictory rankings between Epoch AI and Artificial Analysis mean no one evaluator's verdict should be taken as definitive, and the documented regressions suggest Astra is a mixed upgrade rather than a clean win over Sol or Claude Opus 5.

The framing around François Chollet moving up his AGI timeline is speculation stacked on a measurement, and the ARC-AGI-3 "human-beating efficiency" headline leans on the 62.7% figure while glossing over the 99.9% harness discrepancy that ARC Prize itself flagged as not comparable. The "human-level" and symbolic reasoning language is doing more narrative work than the underlying numbers support, especially given the benchmark disagreement and quiet regressions.

1. 01Whether independent labs (not just OpenAI's own harness) replicate the 62.7% vs 99.9% ARC-AGI-3 gap, and whether ARC Prize publishes a normalised cross-vendor comparison
2. 02Artificial Analysis and Epoch AI updates or reconciliations of their conflicting rankings over the next 2-4 weeks, and whether other evaluators (e.g. LMSYS, Scale) weigh in
3. 03Whether OpenAI or third parties address the documented regressions (GDPval-AA v2, banking support, SciCode, long-context) and whether Astra's 2.5x cost premium changes enterprise adoption or procurement decisions