Study finds coding-agent harnesses change SWE-bench pass rates about as much as reruns, while raising token costs up to threefold

Two Shandong University researchers, Liu and Han, published a paper called "What does a harness buy? Mostly tokens." They tested five models in three coding-agent harnesses (Claude Code, OpenCode and mini-SWE-agent) on SWE-bench Verified (the 45 hardest tasks plus 447 others), with a small Terminal-Bench 2.1 cross-check. Differences between harnesses were mostly within rerun noise, which was up to about 6 points. Claude Code and mini-SWE-agent were under 2 points apart on the large pool, while OpenCode trailed by up to 9 points. Cost per task varied by up to 3x, mainly because of system prompt size: about 16,600 tokens for Claude Code, about 7,000 for OpenCode and under 1,000 for mini-SWE-agent. The authors say they released data and analysis scripts on GitHub, though the coverage we have does not link the paper or repo.

Teams choosing a coding-agent harness have a reason to weigh token cost more heavily, because a leaner harness appears to match a heavier one on pass rate at a fraction of the prompt overhead. It also gives buyers and researchers a practical rule: any claimed harness-driven benchmark gain, such as Zhipu's 2.4-point lead over Claude Code, should come with a rerun of the same setup to show the gap exceeds noise. The impact is moderate. This is a measurement study of one benchmark family with a limited set of models, not a deployed change, and it says little about real codebases.

The headline "not better" overstates the result: OpenCode trailed by up to 9 points, so harnesses are not interchangeable. Pass rate on SWE-bench is also a narrow measure, so it does not show that harness features such as tooling, permissions or workflow support have no value. The coverage is secondhand, from The Decoder, and we have not checked the paper's statistics ourselves.

1. 01Independent replication of the rerun-noise finding, or a published rebuttal, on SWE-bench Verified or newer benchmarks with different models and tasks.
2. 02Whether vendors or leaderboard maintainers (for example Zhipu, Anthropic and the SWE-bench team) start reporting repeated-run variance and token cost alongside pass rates in the coming months.
3. 03Whether the GitHub data and scripts match the paper's figures when others inspect them, especially the claim of up to 3x cost differences driven by system prompt size.