Agent Training Across 18 Harnesses Beat One, Study Finds

Training an AI agent through 18 different harness configurations, instead of one fixed harness, raised a small open model’s average score from 29.81 to 39.04 across three agent benchmarks. That is according to a paper by researchers from Renmin University of China’s Gaoling School of Artificial Intelligence, IQuest Research and Beihang University, posted on arXiv on 21 October 2026. The authors call the method Harness Scaling.

A harness is the software wrapped around a model when it works as an agent: the system prompt, the tools it can call, the format of what those tools send back, how earlier steps are kept or summarised, and the workflow it must follow. The paper says existing approaches “typically train through a single fixed harness, which can lead to specialization and limited transfer across harness configurations.” Its models were also tested inside Claude Code, a harness they never saw in training.

Business Pill · AI AGENTS

A short Business Engineer explainer on AI agents that do work rather than answer questions. It teaches the general idea only and says nothing about the paper in this piece.

What the Paper Found

• Unseen harness setups. On 18 held-out harness configurations, Harness Scaling beat single-harness training with both model sizes and both training algorithms tested; for Qwen3-4B-Instruct with GRPO the in-space gain was +9.80 points, the paper reports.
• Little cost at home. For Qwen3-30B-A3B-Instruct with GRPO on ClawGym-Bench, Harness Scaling scored 59.63 on the original training harness against 59.82 for single-harness training, a 0.19-point drop, while adding 10.48 points in-space and 3.60 points inside Claude Code, the paper reports.
• Inside Claude Code. With Qwen3-30B-A3B-Instruct, the method scored an average 41.20 inside Claude Code, against 38.08 for fine-tuning on trajectories from the OpenClaw harness and 36.28 for reinforcement learning through OpenClaw, according to the paper.
• One harness can hurt elsewhere. On JobBench-Easy inside Claude Code, the 4B model trained through one fixed harness with GRPO scored 5.79, below the untrained base model’s 6.78; Harness Scaling scored 14.83, Table 1 shows.
• More is not always better. Training on all 243 configurations performed worse than the structured 18-configuration set under the same training budget, the authors write, and “how harness diversity is organized matters in addition to configuration count.”

Source: Bai et al., ‘Harness Scaling for Generalizable Agentic Reinforcement Learning’, arXiv 2610.29458, 21 October 2026, Tables 1 and 2. All figures are the paper’s. Benchmarks: ClawGym-Bench, JobBench-Easy and PinchBench.

What Was Posted on X

AI researcher Elvis Saravia summarised the paper on X: models trained across many harness configurations did better on harness configurations never seen in training, and his advice is to treat harness diversity as its own scaling axis.

How the Test Worked

The authors split a harness into five parts: system prompt, tool interface, observation, context management and workflow. Each part has three versions, which gives 243 possible harnesses. The variants were generated with Opus-4.7, an Anthropic model, the paper says: for example, tools renamed with opaque identifiers plus distractor tools that do nothing, tool outputs padded with server metadata, or a plan-first workflow with a progress check every three steps.

From the 243, the authors picked 18 configurations that cover every pair of component versions evenly, and trained Qwen3-4B-Instruct and Qwen3-30B-A3B-Instruct on ClawGym-Task, a set of 13.5K workspace tasks, for 300 optimisation steps. Each task was assigned two harnesses that sit close to each other in the design. Testing used three benchmarks the models were not trained on, 18 other held-out configurations, and two outside harnesses, Claude Code and OpenClaw.

In a post on X, AI researcher Elvis Saravia summarised the takeaway this way: “If your model will run inside harnesses you did not train it in, treat harness diversity as its own scaling axis.”

Where the Gains Came From

The authors scored 200 complete runs per harness on six behaviours. They report that the largest gains appear in “state tracking, workflow compliance, and error recovery”, meaning keeping track of task progress, following the required procedure and recovering from failures. Observation reading improved least, the paper says.

Exposing the same task to two different harnesses helped more than repeating it in one, according to the ablations; raising that number to four gave no further improvement. Replacing the paper’s nearest-neighbour assignment of harnesses with random assignment cut scores by 2.32 points in-space on ClawGym-Bench, the paper reports.

Connecting the Coverage

Our Data World piece reported on 20 October that a Meituan-led team used 2,135 training tasks from 1,037 invented worlds to train an open Qwen model, which improved by 20.90 points on the DABstep-Hard benchmark. Our memory-tax piece reported on 23 October that building Mem0’s memory store cost more than full-context answering, in a study by Zedong Peng of Shanghai Jiao Tong University. What’s new: this paper treats the harness itself, not the tasks or the worlds, as the thing to vary in training.

The Structural Read

As we read it, the paper moves a variable that companies control into the training recipe: system prompts, tool names, output formats, memory compaction and workflows are exactly what differ from one company’s agent setup to the next, and the paper varies all five.

As we read it, the outside test is the useful one for buyers: Claude Code was never used in training, and the 30B model trained across varied white-box harnesses scored 41.20 there, against 38.08 and 36.28 for models trained through the OpenClaw harness, the paper reports.

As we read it, the ablations argue against brute force: the paper reports that all 243 configurations did worse than a chosen 18 under the same budget, and that two harnesses per task beat one while four added nothing.

Three Implications

FOR COMPANIES DEPLOYING AGENTS The paper varies five harness parts, from tool names to context compaction; as we read it, these are the settings a company changes when it wires an agent into its own tools.

FOR MODEL BUILDERS The paper reports gains with two model sizes and two training algorithms, GRPO and PPO, on three benchmarks the models were not trained on.

FOR EVALUATORS The paper scores each model on 18 held-out harness configurations and two outside harnesses, not on one fixed setup.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Agentic Harness War.

The framework argues that “The surface, not the model, governs autonomy.” It contrasts two bets: “OpenAI is betting on one harness that absorbs everything” and “Anthropic is betting on many harnesses, embedded where work already lives.”

As we read it, this paper tests the model side of that question: a model trained across many harnesses held up better in one it had never seen. The framework was published in July 2026 and does not discuss this paper.

What Is Not Established

The tests use two open Qwen3 models of 4B and 30B parameters, not the largest frontier models, and scores are Pass@1 averaged over three runs. The harness variants were generated by a model and stay within one designed space of five components; Claude Code and OpenClaw are the only outside harnesses tested. Training ran on 16 Nvidia H20 GPUs for the 4B model and 32 for the 30B model, the paper says; we did not find a dollar cost in the version we read. We read the 21 October version of the paper and did not contact the authors. Nothing here is investment advice.

The Bottom Line

In the paper’s tests, agents trained across 18 varied harnesses scored higher than agents trained through one, both on unseen setups and inside Claude Code. The authors say the gain depends on how the variety is organised, not only on how much of it there is.