A new study from Shandong University challenges a common assumption in the AI coding market: that a more elaborate agent harness, meaning the system prompt, tool definitions and context-management rules wrapped around a model, reliably yields better results. According to VentureBeat's coverage, the researchers found that such complexity can multiply the cost of solving coding problems by as much as three times without a measurable accuracy benefit.

Why existing comparisons are shaky

Vendors frequently advertise gains from harness updates, and public leaderboards often mix different harnesses and models. Some prior work reports large spreads when a model is held constant and the harness is swapped, and Zhipu AI promotes a 2.4-point advantage for its ZCode harness over Claude Code with the same model. The Shandong team argues these numbers are hard to interpret for two reasons. Many comparisons run each configuration only once, even though rerunning an identical setup can shift scores noticeably; a cited study saw spreads of 2.2 to 6.0 percentage points across ten reruns on SWE-bench Verified. In addition, harnesses are often not compared on equal terms, as one cited example showed the same harness scoring 19.1% with a minimal adapter and 73.4% with a full one.

Test hygiene also matters. In early web-enabled runs, the researchers found that 312 of 439 DeepSeek and Qwen trials, about 71%, had pulled the original human-written fix from GitHub. Those runs were discarded and network access was blocked in the final evaluation.

Experimental design

The team evaluated Claude Code, OpenCode and mini-SWE-agent, a minimalist open-source harness of roughly 100 lines of Python that exposes one bash tool and does no context compaction. Five models ran through all three harnesses: two locally served open-weight models (Qwen3.6-35B-A3B and Qwen3.8-27B) and three API models (DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview). Claude Opus 5 inside Claude Code served as a top-tier reference. Tasks ran in isolated containers across 492 offline SWE-bench Verified problems: the 45 hardest, which take humans one to four-plus hours, and the remaining 447. All models ran the hard subset, while only the two Qwen models ran the larger pool.

Accuracy: no consistent edge

On the 447-task pool, Claude Code and mini-SWE-agent landed within two points of each other and were statistically equivalent within a five-point margin on both Qwen models. On the 45 hardest tasks, about 13% of problems flipped between pass and fail when the harness changed, and rerunning the same harness produced the same median flip rate. Even DeepSeek's official command-line harness on its own DeepSeek-V4-Flash model scored within the range of the others while posting the highest per-trial cost for that model. The researchers conclude that a vendor's own harness on its own model buys no score advantage.

The only harness effect that exceeded noise was a deficit. OpenCode trailed by up to nine points on the large pool. For Qwen3.8-27B, about half of that gap was tied to runs terminated when the model hit a 32,000-token output cap mid-step, often before writing a patch. Claude Code encountered the same cap but automatically prompted the model to continue.

Cost: the real differentiator

With identical models, tasks and pricing, per-task cost differed by up to 3x. On GLM-5.3-Flash, a Claude Code trial cost nearly double a mini-SWE-agent trial and over triple an OpenCode trial, though OpenCode also stopped after fewer model calls. Tool output volume was similar across harnesses. The gap started with the first call: fixed context from system prompts and tool schemas was 16,581 tokens in Claude Code, 7,025 in OpenCode and 829 in mini-SWE-agent. Because the transcript is resent each turn, that overhead compounds with step count, and the researchers infer Claude Code adds further weight by resending intermediate reasoning blocks. Although 96% to 99% of input tokens hit provider prompt caches, repricing under different cache rates scaled bills without changing which harness was costliest. Rankings also depend on the model: on DeepSeek-V4-Flash, mini-SWE-agent made 141 calls versus 104 for Claude Code and cost slightly more, about 15.2 cents versus 14.6 cents per trial.

Corroboration and implications

The results echo the HarnessTax study from UC Berkeley's Sky Computing Lab and Arena, which examined seven models across Claude Code, OpenAI's Codex CLI and the minimal Pi harness. It found pass rates moved only a few points while costs rose up to fivefold, and Claude Code's initial context averaged over ten times Pi's. The Shandong work extends this with a larger task set and comparisons against ordinary run-to-run variance. It also notes that the weakest model tested, Qwen3.6-35B-A3B, was not the most harness-sensitive.

For enterprise teams, the practical lessons are clear. Treat 2- to 3-point harness gains from single runs as unproven, since a 45-task suite can only detect a 13-point gap about half the time and even 447 tasks resolve roughly 5.2 points. Run baseline configurations at least twice. Trim turn-one preambles and tool schemas, because they are resent every step; removing Claude Code's file-editing tools so Claude Opus 5 edited via bash left success within the control range. Track cost per task alongside success rate. VentureBeat's September VB Pulse survey adds urgency, showing 21% of enterprises using an agent orchestration platform lacked automated controls to halt token spending.

For the market, the research suggests lightweight harnesses such as mini-SWE-agent and Pi offer a cost advantage, while heavyweight commercial harnesses must justify their overhead through features beyond raw benchmark accuracy.