Controlled study finds no clear performance advantage for vendor-native AI coding harnesses over alternatives, with cost comparisons undermined by missing usage data

A new arXiv paper ran a controlled experiment isolating the effect of an AI agent's "harness" (the tool/prompt/control-flow scaffolding around a model) from the underlying model itself, using a private set of 256 coding tasks and 792 completed paired runs. It found no statistically resolved advantage for vendor-native harnesses (e.g. Claude Agent SDK, OpenAI Codex SDK) over a generic alternative (deepagents) on average, though there was a large split by task type for Opus 4.8 (native harness worse on repository tasks, better on contest-style tasks). Cost comparisons were undermined by 58 runs with no usage telemetry recorded, and the paper also corrects a cost-figure error in its own August 2026 predecessor caused by a telemetry defect.

This is directly relevant to any team choosing between a vendor's own agent scaffolding and third-party alternatives: the common assumption that vendor-native tooling performs better is not supported by this evidence, and cost claims in this space should be treated as unresolved rather than settled. The task-type split (repository vs contest tasks) suggests harness choice may matter more for specific workflows than in aggregate, which is a more useful signal than a single "which harness wins" verdict. Impact is bounded by a null result with wide confidence intervals, which tends to change fewer minds than a clear positive finding.

The task suite is private and not independently reproducible, so the specific numbers cannot be checked by outside researchers, only the released orchestrator and grading code. The repository/contest task-type split is explicitly flagged by the authors as post-hoc and needing a designed replication, so it should not yet be read as a firm finding. There is no vendor pushback or independent replication in the coverage, and no indication this has changed any vendor's product roadmap or pricing.

1. 01Whether the authors or others run a pre-registered follow-up specifically testing the repository-vs-contest task split for Opus 4.8
2. 02Whether Anthropic, OpenAI or Google respond publicly or release their own harness-vs-model benchmarks with full usage telemetry
3. 03Whether other research groups adopt the released orchestrator/grading oracle to attempt independent replication on their own task sets