Ramp SWE-Bench v2: Native Harnesses Add Up to 9.8 Points

Ramp has published Ramp SWE-Bench v2, a private coding benchmark of 500 tasks built from production work by more than 200 of its engineers (Ramp on X). It runs every model twice: in its lab’s own agent harness and in a bash-only one.

The harness gap is largest outside the top tier. GPT-6.1 Terra and GPT-6 Luna each score 9.8 points higher in OpenAI’s Codex than in the bash-only harness, by our arithmetic from Ramp’s tables.

Business Pill · Vertical Integration

An evergreen Business Engineer explainer on vertical integration. It teaches the general idea only and does not discuss Ramp.

The key insight: On Ramp’s production-grounded benchmark, the agent harness adds up to 9.8 points for models outside the top four but under 1.5 for the leaders, by our arithmetic from Ramp’s tables.

How Ramp Tests

Tasks come from pull requests that Ramp’s in-house coding agent, Inspect, merged into production after engineer review. Each result is a single attempt at high reasoning effort, with web tools disabled.

The “Native” view runs each model in its lab’s harness, such as Claude Code or Codex. The “Bash” view uses mini-swe-agent, “a minimal harness where the only tool is bash”, per Ramp.

Models outside the top four gain up to 9.8 points in their lab's own harness on Ramp SWE-Bench v2, while the l
Models outside the top four gain up to 9.8 points in their lab’s own harness on Ramp SWE-Bench v2, while the leaders gain under 1.5, by our arithmetic from Ramp’s tables.

What Was Posted on X

RampLabs announced Ramp SWE-Bench v2 on X on 18 November: 500 tasks built from work shipped to production by over 200 of its engineers, with frontier models in their native harnesses. The scores in this piece come from the benchmark site it links.

Veeral Patel wrote on X that we can finally answer how much the harness matters without asking someone who is selling one. We quote his reading as his view.

Who Leads

In native harnesses, GPT-6.2 Astra in Codex leads at 76.4%. Google’s Carbon in OpenCode follows at 75.4%, then Claude Fable 5.2 in Claude Code and GPT-6.2 Sol in Codex at 73.4% each.

Where The Harness Matters

For the top models the harness barely moves the score: Fable 5.2 is 0.4 points higher native, Astra 1.4 and Carbon 1.4, by our arithmetic.

Models further down the table gain more. Grok 4.7 is 9.0 points higher in Grok Build, DeepSeek V4.1 Flash 7.4 points higher in its harness, and Claude Sonnet 5.5 5.6 points higher in Claude Code.

Two models do better without their native harness: Claude Opus 5.5 scores 3.4 points higher in the bash-only setup, and Muse Ember 2.6 points higher.

What It Costs

The harness also changes the bill. Astra’s mean cost per task is $3.31 natively against $1.03 bash-only, about 3.2 times, for 1.4 more points, by our arithmetic.

Fable 5.2 runs the other way: $3.79 a task in Claude Code against $6.45 bash-only, with near-identical scores.

Why It Matters

Patel’s reading, on X: “Finally we can answer how much the harness matters without asking someone who’s selling you one” (Veeral Patel). Other agent benchmarks we covered this week include Vals AI’s Lean proof test and OpenAI’s own research-agent data.

Coding agents are also becoming a distribution channel, as with Snap opening Lens building to Claude Code.

The Structural Read

As we read Ramp’s tables, the harness matters most for models outside the top tier, though two of them score higher without it.

As we read the cost columns, the harness is also a pricing decision: the same model can cost three times as much per task.

Ramp, SWE-Bench v2 methodology

“Harnesses shape model behavior, so we run every model two ways.”

Three Implications

FOR ENGINEERING TEAMS On Ramp’s tables, the harness choice moves some models by up to 9.8 points.

FOR MODEL LABS The harness can be worth up to 9.8 points on Ramp’s tasks, by our arithmetic.

FOR BUYERS Astra costs about 3.2 times more per task natively than bash-only on Ramp’s tables.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Agentic Harness War.

The analysis puts it this way: “Foundation models become relatively commoditized as orchestration layers capture differentiation.”

As we read Ramp’s tables, the harness gap is largest outside the top tier. The analysis predates Ramp’s benchmark and does not discuss it.

What Is Not Established

The tasks are private Ramp code, so results may not transfer to other codebases. Grading uses a panel of LLM judges, per Ramp. Ramp’s tables show results for 14 models in native harnesses.

This Week In The AI Supercycle

AI Lost Its Central Banker, the Lab Oligopoly, and the Future of Open Source

This week’s issue: the Nvidia leadership change, read as the AI build-out losing its central banker: who now finances the GPU landlords, backstops the data centres and sponsors open models, and why two labs are becoming the bank.

Read the full issue →

Business Engineer Framework

The Agentic Harness War

A Business Engineer analysis of how agent harnesses and orchestration layers compete as models commoditize.

Read the Map of AI →

The Bottom Line

On Ramp’s 500 production-grounded tasks, the native harness adds up to 9.8 points for models outside the top four but under 1.5 for the leaders, by our arithmetic from Ramp’s tables.

97,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.

A note on sourcing. Scores, costs and methodology are from Ramp’s SWE-Bench v2 site (native and bash views, last updated 17 November), read 19 November. The harness gaps, cost ratio and rankings are our arithmetic from Ramp’s tables. Patel’s comment is from his post on X.

Sources: Ramp SWE-Bench v2 site: native and bash-harness summary tables and methodology, read 19 Nov 2026 · RampLabs on X, 18 Nov 2026 · Veeral Patel on X, 18 Nov 2026

Scroll to Top
FourWeekMBA