Your Coding Agent Harness Can Triple the Bill Without Improving the Answer
TL;DR: A new preprint ran five models through three different agent wrappers on the same 492 coding tasks. The finding: the coding agent harness you pick barely changes whether the job gets done — but it changes what you pay by up to 3x. The researchers’ own summary of the vendor question: “A vendor’s own harness on its own model buys no score advantage.”
What the coding agent harness study did
The paper is “What Does a Harness Buy? Tokens, Mostly” by Yangze Liu and Zhongyi Han, posted to arXiv on 3 October and written up by VentureBeat on 23 October, which places the authors at Shandong University.
A coding agent harness is everything wrapped around the model: the system prompt, the tools it is handed, and the rules for managing context. It is the part of an AI coding tool the vendor actually builds — the model usually comes from somewhere else. We wrote in August that the harness matters more than the model. This study is the first serious attempt to price that claim, and the answer is uncomfortable for everyone.
The setup: three coding agent harnesses — Claude Code, OpenCode, and mini-SWE-agent, a deliberately minimal open-source one of about 100 lines of Python with a single tool and no context management. Five models underneath them. 492 tasks from SWE-bench Verified, run in isolated containers.
Where the coding agent harness shows up: the bill
Accuracy barely moved. Across the 447-task pool, the heaviest coding agent harness and the lightest one landed within about five points of each other — inside the margin the test can actually resolve. The one clear difference was OpenCode trailing by up to nine points, and even that was partly an artefact: some runs hit a 32,000-token output cap and were cut short rather than being wrong.
The noise is as big as the effect. On the 45 hardest tasks, swapping the coding agent harness changed the outcome on about 13% of them. Re-running the same harness unchanged flipped about the same share. The tasks one harness wins in a given run are often not the ones it wins in the next.
The cost spread is real and large. Same model, same tasks, one price list: cost per task varied by up to 3x. The cause is unglamorous — a fixed preamble of system prompt and tool definitions resent on every single turn. VentureBeat gives the numbers for the first API call: 16,581 tokens of context in Claude Code, 7,025 in OpenCode, and 829 in mini-SWE-agent. Prompt caching covered 96–99% of input tokens and scaled the bill down, but did not change which harness was most expensive.
A separate study from Berkeley’s Sky Computing Lab and Arena, which VentureBeat cites, reached the same shape across seven models and three harnesses: pass rates moved a few points, costs moved up to fivefold.
Why this matters if you don’t code
You are probably not running SWE-bench. Here is the translation.
When you choose between AI build tools, you are told you are choosing capability. Mostly you are choosing a price, set by the coding agent harness, and the price is set by something you never see: how much text the tool stuffs into every exchange before your request even reaches the model. A heavier wrapper means more tokens per turn, more turns, and a larger bill for the same work.
That reframes the question people ask us most. “Which AI coding tool is best?” has a less exciting answer than anyone wants: among the good ones, they mostly land in the same place, and the difference you will actually feel is in what they cost you to get there. Which is why the practical move is the dull one — track cost per finished task rather than reading benchmark charts, and put a cap on spend before you need one.
There is also a useful piece of scepticism here for anyone reading vendor comparisons. A 45-task benchmark can only reliably detect a gap of about 13 points. Most claimed differences between agents are smaller than that, which means most of them are noise presented as a result.
Four caveats worth keeping
- It is a preprint. Posted to arXiv, not peer-reviewed, and three weeks old before it was widely reported.
- The cost ranking is not fixed. On DeepSeek-V4-Flash the minimal coding agent harness cost slightly more per trial (about 15.2 cents) than Claude Code (about 14.6 cents). “Lighter is cheaper” held generally, not universally.
- These are software-engineering tasks. Whether the same spread applies to building a landing page or an internal tool is untested.
- The benchmark nearly broke. In early runs with web access enabled, 312 of 439 trials — 71% — simply fetched the human-written fix from GitHub. Those were discarded and the network was switched off for the real evaluation. Worth remembering the next time you see a benchmark score quoted without qualification.
Who should care (and who shouldn’t)
How much the coding agent harness question matters to you depends on how you pay.
- On a flat subscription: care least. You are buying an allowance, not tokens; the harness decides how fast you burn it rather than what you are billed.
- Paying per token with your own API key: care most. This is your bill, and it is being set by a preamble you did not write.
- Building a revenue app: worth one afternoon. Run the same task through two tools and compare the finished cost, not the demo.
- An internal tool for your business: pick on fit and support, not on benchmark deltas that the study suggests are mostly noise.
- Learning: ignore all of it. Use whatever you find pleasant; the difference is not about you yet.
Our take
The honest reading is that this is good news badly dressed. If the coding agent harness barely changes the outcome, then the anxiety about picking the right tool is largely misplaced — and that is a relief for anyone who has been paralysed by comparison tables.
What we would not do is treat “lighter is cheaper” as a rule to act on blindly. The minimal coding agent harness in this study has no context management, which is fine for a contained benchmark task and not at all the same as a long, messy real project. The finding to carry away is narrower and more useful: stop reading small benchmark gaps as quality, and start reading your own cost per finished task as the number that matters.
Not sure which tool fits what you are building — and what it will cost you? Take the 60-second Vibe Coding Tool Finder quiz →
Some links on this page are affiliate links. If you buy through them we may earn a commission at no extra cost to you.
FAQ
What is a coding agent harness?
Everything wrapped around the AI model: the system prompt, the tools the agent can call, and the rules for managing context. It is the part an AI coding tool’s vendor builds themselves, while the underlying model is often licensed from someone else.
Does a more expensive AI coding tool give better results?
Not according to this study. Across the harnesses tested, accuracy differences sat inside the test’s own margin of error, while cost per task varied by up to 3x. The researchers found no score advantage for a vendor running its own model on its own harness.
Should I switch tools over the coding agent harness findings?
Probably not on its own. The practical step is to measure what a finished task costs you in the tool you already use, then compare that with one alternative — rather than switching on the strength of a benchmark gap that may be noise.
