Alibaba-led researchers release Co-Design Bench; best of ten coding agents fully completes only 26.4% of real-world change requests

Researchers from Alibaba and several Asian universities published Co-Design Bench, a coding-agent test built from real user change requests to existing web apps, including images, audio, video and documents as inputs. Ten models were tested. The best, GPT-6 Astra, met about 70% of task requirements on average but fully solved only 26.4% of tasks (about 41% of multimodal tasks and just under 18% of repository tasks). Claude Opus 5 and Qwen3.8-Max followed, and Grok 4.6 came last. Scores fell as tasks combined three or four types of change, and the same model scored very differently depending on the harness (Qwen3.8-27B: 75 vs 56 percent on larger overhauls).

Teams choosing a coding agent get a useful warning: strong results on older benchmarks, which are saturated above 99%, do not mean an agent finishes real change requests, and most output still needs human cleanup. The harness swing suggests buyers should test model and tooling as a pair on their own tickets rather than trust a model-only leaderboard. The impact is limited, though. This is a research benchmark with no deployment, and it does not tell you how much cleanup effort each task actually needs.

The headline framing of "nearly three out of four tasks" needing cleanup is a restatement of the strictest full-completion measure; the best model met about 70% of requirements, so "mostly unfinished" overstates it. Grading relies on AI judges testing apps in a browser, with only a second AI judge re-checking positives, so the 26.4% figure carries unquantified grader error. The benchmark is also written by a company whose own Qwen models are among those ranked, and the extraction found no paper link to verify the numbers.

1. 01Publication of the paper and the benchmark code or task set, so the 26.4% figure, the judge setup and the harness settings can be independently reproduced.
2. 02Whether other labs or independent groups (for example METR) report Co-Design Bench results, and whether the gap between full completion and average requirement coverage holds up on their runs.
3. 03Whether scores on this benchmark rise meaningfully with the next releases from the top vendors, and whether vendors start publishing model-plus-harness results rather than model-only numbers.