Researchers release Agent4Sci, a 20-benchmark testbed showing frontier coding agents improve published ML-for-science results mainly via engineering tricks

A research team published an arXiv paper introducing Agent4Sci, a testbed of 20 machine learning benchmarks drawn from scientific problems across 11 domains. Code and data are public on GitHub and Hugging Face. The authors report that frontier coding agents beat the previously published best result on every benchmark, with Claude Opus averaging a 10.4% relative improvement and Codex 4.4%. Adding plug-in skills for ideation, experiment management and checklists lifted those figures to 20.1% for Claude Opus and 8.1% for Codex. The authors attribute the gains mainly to ML-engineering tricks, not new scientific insight.

This gives researchers a shared, open way to measure whether coding agents can do more than tune pipelines. That matters for anyone judging claims about "automated researchers". The finding that gains come from engineering rather than insight is a useful corrective. Practically, it suggests agents can already squeeze extra performance out of existing ML-for-science baselines, and that scaffolding such as structured skills can double the gain. Nothing has been deployed, so the near-term impact is limited to the research community and teams building agent tooling.

The "beat the published best on every benchmark" line sounds dramatic, but the margins are modest (a 4.4% mean for Codex) and published baselines may be weakly tuned, so beating them may say more about baseline effort than agent skill. The Claude-versus-Codex gap is a single-paper result, probably with different scaffolds and run budgets, and shouldn't be read as a settled ranking. The claim that gains reflect "engineering tricks, not insight" is the authors' interpretation and is hard to verify.

1. 01Independent replication of the headline numbers (10.4% and 20.1% for Claude Opus, 4.4% and 8.1% for Codex), including run-to-run variance, compute budgets and whether the baselines were freshly re-tuned.
2. 02Whether other labs and agent developers adopt Agent4Sci and report results on it over the next few months, and whether newer model releases move the mean improvement beyond the plug-in-skill figures.
3. 03Any benchmark entries or follow-up work where an agent produces a result that domain experts judge to be a genuine scientific insight and not an engineering gain, which would test the paper's central caveat.