AI agents that rewrite their own software scaffolding to score higher also lose tasks they used to solve, according to a paper posted to arXiv on 5 November by researchers from the University of Science and Technology of China, UC Berkeley, Simple Agent Lab, the Beijing Institute for General Artificial Intelligence and Shanghai Jiaotong University. The paper says “All sixteen evolved agents gain previously uncovered tasks while losing previously covered ones”.
By our count from the paper’s Table 1, 11 of the 16 evolved agents became more reliable than the agent they started from and 5 became less reliable. The paper adds that “Eight of eleven reliability-improving Champions cost more per trial than Seed”.
Business Pill · AI Agents Explained
An evergreen Business Engineer explainer on AI agents. It teaches the general idea only and does not describe the paper’s methods.
The key insight: A preprint from researchers including UC Berkeley found that all 16 AI agents that edited their own harness both gained and lost tasks; by our count from its Table 1, 11 became more reliable and 5 less reliable, and the paper says 8 of the 11 gainers cost more per trial.
The paper studies harness evolution, where an agent changes the system around the model rather than the model itself. It says this “keeps model weights fixed while persistently changing the surrounding system that shapes observations and turns model outputs into actions”.
The team ran four published evolution methods, AHE, HyperAgents, A-Evolve and GEPA, “using GPT-5.4”, on two harnesses: MiniSWE, which “permits inner-source changes”, and Codex, which “permits changes to prompts, skills, configuration, and external orchestration”. The benchmarks were tau3-Banking, with 97 general tasks, and Terminal-Bench 2, with 89 coding tasks. Four methods on two harnesses and two benchmarks gives 16 evolved agents, by our arithmetic.
Each evolved agent, which the paper calls a Champion, was compared with its starting version, the Seed, across eight passes. The paper reports two numbers: reliability, the average success rate over the eight passes, and coverage, the share of tasks solved at least once.
Self-edited (evolved) agents compared with their starting versions over eight passes, from Table 1 and Section 4.3 of arXiv:2611.07624 (5 Nov 2026). The 11 and 5 counts are our arithmetic from Table 1; the 8 of 11 cost figure is the paper’s.
What Was Posted on X
New University of California, Berkeley+other top labs paper finds that agents that rewrite their own harness to score higher also lose tasks they once solved, which the average score hides.
Harness evolution lets an agent edit its own prompts, skills, tools, or code while the model stays frozen. Results usually come as 1 higher benchmark score, which can hide lost tasks, higher costs, and gains that don’t carry over.
They ran 4 evolution methods, including GEPA and HyperAgents, with GPT-5.4 on 2 harnesses and 2 benchmarks. Then they re-tested each evolved agent 8 times against its starting version.
All 16 evolved agents gained some tasks and lost others, and 1 raised reliability by 5.2 points while losing 10 tasks it used to cover. Of 11 agents that became more reliable, 8 also cost more per run.
When you evolve an agent harness, track tasks lost, cost per run, and results on unseen tasks, not just the average score.
Rohan Paul highlighted the paper on 9 November, summarising that all 16 evolved agents gained some tasks and lost others and that 8 of 11 agents that became more reliable also cost more per run. The advice to track tasks lost, cost per run and results on unseen tasks is the post’s own.
What the Average Score Hides
The abstract states: “In one general-task case, reliability rises by 5.2 points despite losing ten previously covered tasks.” Section 4.2 adds that “even when both aggregate metrics improve, coverage does not expand monotonically” and concludes that “Gains and losses should therefore be reported separately”.
Table 1 shows the spread. The largest reliability gain was HyperAgents on the Codex harness on tau3-Banking, from 30.3% to 36.9%, or 6.6 points. Five of the 16 Champions lost coverage against their Seed, by our count, including two that fell 3.4 points on Terminal-Bench 2 with Codex.
On cost, the paper says the three reliability gains that did not cost more per trial “use the source-editable harness on coding tasks”. It also notes that “higher per-trial cost can coexist with fewer attempts and lower cost to reach comparable coverage”.
Who Flagged It
Rohan Paul highlighted the paper in a post on X on 9 November, writing that when you evolve an agent harness you should “track tasks lost, cost per run, and results on unseen tasks, not just the average score”.
As we read the paper, its central point is about measurement: a single higher score can combine newly solved tasks with tasks the agent no longer solves, which is why it asks for gains and losses to be reported separately.
As we read Table 1, results vary by setting: the same method can raise reliability in one setup and lower it in another, for example GEPA rose 0.8 points on tau3-Banking with MiniSWE but fell 1.8 points on Terminal-Bench 2 with Codex.
Wang et al., arXiv:2611.07624, 5 November 2026
“In one general-task case, reliability rises by 5.2 points despite losing ten previously covered tasks.”
Three Implications
FOR TEAMS EVOLVING AGENTS The paper asks for gains and losses to be reported separately, and Rohan Paul’s post adds tracking tasks lost and cost per run.
FOR BUYERS OF AGENT TOOLS A higher average score from a self-improving agent may include tasks it no longer solves, per the paper’s Section 4.2.
FOR COST PLANNING The paper says 8 of 11 reliability-improving agents cost more per trial than their starting version.
The framework says: “The agentic surface — the harness — is becoming where value pools in the entire AI stack.”
As we read it, this paper shows the cost of letting that surface change itself: the harness is where agents improve, and also where they quietly lose work they could already do.
What Is Not Established
The results come from one paper using one model, GPT-5.4, two harnesses and two benchmarks; the paper itself calls for “explicit applicability conditions”. It is a preprint on arXiv. The counts of 11 more reliable, 5 less reliable and 5 with lower coverage are our arithmetic from the paper’s Table 1.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
A preprint from researchers including UC Berkeley found that all 16 self-edited AI agents it tested both gained and lost tasks, that 11 of the 16 became more reliable by our count, and that 8 of those 11 cost more per run, according to the paper.
A note on sourcing. Findings, figures and quotations are from the arXiv preprint 2611.07624 (5 November 2026), which has not been peer reviewed. Counts of agents that became more or less reliable, or lost coverage, are our arithmetic from the paper’s Table 1. The advice to track tasks lost and cost per run is Rohan Paul’s, from a post on X.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
