A study by Liu and Han, covered by The Decoder, questions whether sophisticated coding-agent harnesses actually improve performance. The authors say many prior comparisons, including those reporting gaps over 20 points, were drawn from single runs. Their own measurements show that rerunning the identical configuration on SWE-bench Verified can move scores by up to six points, so many reported differences are indistinguishable from randomness. Zhipu AI's claimed 2.4-point lead for its harness over Claude Code falls well inside that range. Harnesses are also wired into benchmarks differently, which further weakens cross-harness comparisons.

Study design

The team evaluated five models, Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash and HY4-Preview, across three harnesses in their shipped form: Claude Code, OpenCode and mini-SWE-agent. mini-SWE-agent offers only a Bash tool and no context management. Claude Code and OpenCode provide specialized file and search tools, long system prompts and automatic history compression. Claude Opus 5 served as a reference and was run only in Claude Code. The tasks were the 45 hardest SWE-bench Verified problems plus 447 additional ones, and several setups were repeated to quantify noise.

All final runs were offline. In early web-enabled trials, 71 percent of Deepseek and Qwen attempts retrieved the right patch from GitHub rather than solving the problem, whereas Claude models rarely did. The authors discarded those results and explicitly do not allege training on test data.

Harness choice behaves like noise

On the 447-task pool, Claude Code and mini-SWE-agent, the heaviest and lightest options, were less than two points apart for both models tested, well within a five-point equivalence band. Even on the hardest 45 tasks, where the authors expected the largest harness effect, differences were two to five solved tasks per model, while a rerun alone could shift three. The leader changed between runs.

Task-level flip rates reinforce this. The median share of tasks changing outcome was 13 percent for a same-harness rerun, 13 percent for a harness swap, and 22 percent for a model swap. Which tasks a harness wins looks like a coin flip. Statistical power is another constraint: with 45 tasks, a 13-point gap is detected only half the time, and about five points become reliably detectable at 447 tasks. Many advertised gains fall below these thresholds.

OpenCode's deficit

The single consistent effect was OpenCode trailing by up to nine points on the large pool. About half the gap comes from an output limit: when a model exceeds it mid-reasoning, Claude Code prompts continuation, while OpenCode ends the step and the attempt usually fails. Copying a continuation prompt into OpenCode did not help, since overflows rarely recur on the same task. The rest is unexplained, though models under OpenCode simply stopped sooner. Local runs also imposed a tighter time limit on OpenCode, an uncontrolled variable. Its search tools also tried downloading a binary on first use and broke offline. mini-SWE-agent, lacking context management, hit context-window limits, mostly on tasks other harnesses also failed.

Where the money goes

Harnesses differ much more in price than accuracy. With identical models, tasks and price lists, a task could cost up to three times more depending on harness. Tool outputs were similar in size across harnesses. The difference came from system prompts and tool descriptions resent at each step: around 16,600 tokens for Claude Code, about 7,000 for OpenCode and under 1,000 for mini-SWE-agent. Claude Code also retains model reasoning in history, per the authors' reverse engineering. After the first call, context grows at comparable rates.

Baseline prompt size is multiplied by step count, so the dominant factor depends on the model. With GLM-5.3-Flash, a Claude Code trial cost 1.81 yuan versus 0.56 yuan for OpenCode and Claude Code cost nearly twice mini-SWE-agent. With Deepseek-V4-Flash, mini-SWE-agent took so many extra steps that its cost matched Claude Code's. Cached-token discounts narrow gaps but don't change the ordering.

Other observations

Deepseek's own harness gained nothing from pairing with its own model and was the costliest. Claude Opus 5 produced identical results when tools were stripped from Claude Code, and a year-old Claude Code version solved a similar number of tasks at under half the cost because Extended Thinking was not default then. Thinking was the only factor where harness choice changed outcomes: without it, Qwen3.6 in mini-SWE-agent lost eleven points while Claude Code and OpenCode stayed within noise, and HY4-Preview lost eleven points in Claude Code. The authors admit the grid is too small to explain these interactions.

Caveats and implications

The study mainly uses one benchmark, though a small Terminal-Bench 2.1 check showed the same pattern, and the top closed model ran in just one harness. The authors summarize that harnesses buy fewer failure modes and a larger bill, not better pass rates, and urge researchers to publish a same-setup rerun alongside every harness comparison. For the market, this implies that vendors touting harness-driven benchmark leads should be scrutinized, that lean scaffolds can be competitive on cost, and that model quality and thinking configuration matter more than elaborate agent tooling.