Y Combinator chief executive Garry Tan has published an open-source test of AI agent harnesses that keeps the model and the tasks fixed and swaps only the program wrapped around the model. On 52 coding tasks run on OpenAI’s GPT-6.1 Sol, the costliest harness, OpenClaw, spent 2.18 times as much per solved task as the cheapest, Pi, according to the repository’s results.
Pass rates moved far less: the four Sol harnesses solved between 76.0% and 85.6% of the tasks, a spread of 9.6 points, by our arithmetic. Tan shared the repository on X on 14 November with the words “Here’s my open source harness eval repo and what it found”.
Business Pill · Switching Costs Explained
An evergreen Business Engineer explainer on switching costs: why it is hard to leave an AI supplier. It teaches the general idea only and does not discuss g-harness-evals.
The key insight: As we read the results, the harness decides the bill more than the score: on the same model and tasks, cost per solved task varied up to 2.18 times while pass rates on that model moved within 9.6 points, by our arithmetic.
The README defines a harness as “the program wrapped around a model: its system prompt, its tools and its memory” and names Claude Code and Codex as examples. The test keeps “the model (Claude Opus 5.5 or GPT-6.1 Sol) and the tasks the same” and measures how often each harness solves a task, what each solved task costs and how long it takes.
It ran two sets: 52 public Terminal-Bench 2.1 coding tasks, and 20 multi-user tasks in which “the agent is the chat bot of a simulated company, answering several people in turn”. The README says every harness in the rankings is a plain install, without Tan’s own gstack and gbrain projects, which it tests separately.
Cost per solved task against the cheapest harness, GPT-6.1 Sol on 52 public Terminal-Bench 2.1 coding tasks, from the g-harness-evals README after its 9-10 November cost corrections. Pass rates in the tags. Early results on public tasks.
Cost Moved More Than Pass Rate
The repository’s headline finding: “Harnesses differed more on cost than on how often they solved tasks. Compare them on cost per solved task, on your own work.” On Opus 5.5 coding tasks, Pi cost $0.62 per solved task and Claude Code $0.63, while OpenClaw and Hermes Agent cost $0.88 each; Claude Code had the top pass rate at 88.5%.
On Opus, pass rates ran from 76.0% to 88.5%, a 12.5-point spread, against a top cost ratio of 1.46 times, by our arithmetic. On the multi-user tasks, OpenClaw on Opus cost $0.44 per solved task against $0.19 to $0.24 for the other three Opus harnesses, and the README says no pair it tested differed in pass rate there.
The README attributes most of the gap to what is re-sent: “About 81% of its extra cost over Claude Code on Opus is cache writes”, it says of OpenClaw, which “re-sends a much bigger system prompt and bigger tool results”. Our piece on a 99% cut in Claude Code subagent update costs covers another case of harness design moving the bill.
Where Rules and Skills Landed
The test also checked whether agents obey instructions depending on where they sit. A “check gbrain first” rule placed in the system prompt was followed in 30 of 30 turns on several harnesses, the README says, while a skill nobody mentioned was opened 0 times in 30 turns by Claude Code.
Tan’s own skill pack fared poorly in the test of it: told to run gstack’s skills, “Opus read the skill text every time and followed none of its steps (0 of 46 sessions), at about twice the cost per try”. Claude Code and Codex together hold most of the market for these tools, as our piece on Menlo Ventures’ AI coding figures sets out.
The Structural Read
As we read the README, much of the cost gap is design, not intelligence: it says about 81% of OpenClaw’s extra cost over Claude Code on Opus is cache writes from re-sending a bigger system prompt and tool results.
As we read the rule-placement results, where an instruction sits matters: the README reports a rule in the system prompt followed in 30 of 30 turns, while Claude Code opened an unmentioned skill 0 times in 30 turns.
g-harness-evals README, Garry Tan, open source
“Harnesses differed more on cost than on how often they solved tasks. Compare them on cost per solved task, on your own work.”
Three Implications
FOR TEAMS CHOOSING A CODING AGENT The README says to compare harnesses on cost per solved task, on your own work; on Opus 5.5, Claude Code cost $0.63 per solved task and Pi $0.62.
FOR HARNESS MAKERS The README attributes OpenClaw’s extra cost mostly to cache writes from a bigger system prompt and bigger tool results.
FOR ANYONE WRITING AGENT RULES A rule in the system prompt reached 30 of 30 turns on several harnesses; an unmentioned skill was opened 0 times in 30 turns by Claude Code, the README says.
The framework argues: “The surface, not the model, governs autonomy.”
As we read it, Tan’s test puts a price on that surface: the same modelcost up to 2.18 times more per solved task depending on the harness. The framework does not discuss g-harness-evals.
Cost per solved task on the multi-user tasks with Opus 5.5, as shown in the g-harness-evals README; pass rates in the tags. The README notes no statistical test on cost.
What Is Not Established
The README calls these “early (“dev”) results, not a confirmation on hidden tasks”, run on public tasks the authors could see while building, and says the cost differences were measured “with no statistical test”. Costs were corrected on 9 and 10 November, and the 2.18 times figure uses the corrected ratios. Claude Code and Codex are never compared with each other, because they run different models.
The results hold for the harness versions the authors pinned, and shipped settings such as thinking effort differ between harnesses. We read the repository and Tan’s post; neither Tan nor the harness makers were contacted for this piece.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
In Garry Tan’s open-source test, swapping the harness around the same model changed the cost of a solved coding task by up to 2.18 times on GPT-6.1 Sol, while pass rates there moved within 9.6 points, by our arithmetic. The repository’s advice is to compare harnesses on cost per solved task, on your own work.
A note on sourcing. Every result comes from the g-harness-evals README on GitHub, read on 14 November 2026, including its 9 and 10 November cost corrections; the authors call them early results on public tasks. Spreads and multiples not in the README are our arithmetic.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
