A new benchmark led by Carnegie Mellon researchers, Singularity-Bench, asks AI agents to build the systems used to make AI: a pre-training run, a mid-training mix, an RL post-training run, an inference engine, a coding harness and a cluster orchestrator, each scored against a production tool on the same GPUs. In the paper, posted to arXiv on 12 November, Anthropic’s Opus 5.5 running in Claude Code is the only one of 11 agents that matches or beats the human-built baseline on all six tasks.
Its inference engine earned 11.52 USD per GPU-hour against 8.18 for SGLang, the paper reports: about 41% more, by our arithmetic. Its Singularity-Index, where 1 equals the human baselines, is 1.252, against an average of 0.58 for the other 10 agents, or about 2.2x.
Business Pill · Switching Costs Explained
An evergreen Business Engineer explainer on switching costs: why it is hard to leave an AI supplier. It teaches the general idea only and does not discuss Singularity-Bench.
The key insight: As we read the paper, the result is about the whole stack rather than one task: Opus 5.5 in Claude Code is the only one of 11 agents that the authors score at or above the human-built baseline on all six layers, from pre-training to cluster orchestration, with a Singularity-Index of 1.252 against 0.58 for the rest.
The paper says each task gives the agent a minimal starting point and “evaluates them against mature human-engineered baselines on identical hardware”. The baselines include the OLMo 2 recipe in Megatron-LM for pre-training, SGLang for serving, Claude Code itself for the harness task and NVIDIA’s Dynamo for cluster orchestration, run on Nvidia GB300 GPUs.
The authors also red-teamed the evaluation framework against reward hacking, covering 9 of the 10 categories in the OWASP Top 10 for Agentic Applications, and built an extension called Breakwater that, they say, “blocks all attacks tested in our study”.
Singularity-Index by agent from Table 5 of the Singularity-Bench preprint (arXiv:2611.16591, 12 November 2026), where 1.0 equals the human-built baselines; the other 10 agents average 0.58.
Where Opus 5.5 Topped the Production Tools
The paper reports that Opus 5.5 scored above the baseline on five tasks and matched it on the sixth, post-training, at 0.533 against 0.539. On inference, it says Opus 5.5 also cut the 90th-percentile time to first token from 12.0 to 0.48 seconds, 25x faster by our arithmetic, and that six of the 11 engines scored zero.
On cluster orchestration, the paper puts Opus 5.5 at 0.510 against Dynamo’s 0.460, about 11% higher by our arithmetic. On the harness task, it says only Opus 5.5 (0.468) exceeded Claude Code (0.438). In pre-training, it reports a validation perplexity of 12.77 against 14.22 for the OLMo 2 baseline, where lower is better.
The Cost of Coming First
The paper lists Opus 5.5’s API spend at $425.41, against $84.52 for GPT-6.1 Sol in Codex (Singularity-Index 0.969) and $4.36 for DeepSeek V4.1 Flash in Codex (0.698), the best open-weight agent. By our arithmetic, Opus 5.5 cost about 98x DeepSeek and about 5x GPT-6.1 Sol. The paper says proprietary agents averaged 1.01 and open-weight agents 0.50.
As we read the costs, the ranking is not the same as the value ranking: the paper lists Opus 5.5 at $425.41 in API spend against $4.36 for DeepSeek V4.1 Flash, which it places on the cost-index Pareto frontier with GPT-6.1 Sol and Opus 5.5.
As we read the harness finding, the paper says the same model can score very differently in Claude Code and in Codex, so the agent is the pairing of model and harness, not the model alone.
Singularity-Bench preprint, arXiv:2611.16591, 12 November 2026
“Still, matching the human baselines is far from saturating Singularity-Bench”
Three Implications
FOR AI LABS The paper says Opus 5.5 earned 11.52 USD per GPU-hour on its inference engine against 8.18 for SGLang, and cut 90th-percentile time to first token from 12.0 to 0.48 seconds.
FOR TOOL MAKERS The paper scores Opus 5.5 at 0.510 against Dynamo’s 0.460 on cluster orchestration and 0.468 against Claude Code’s 0.438 on the harness task.
FOR EVALUATORS The authors say they found reward-hacking vulnerabilities in the Harbor framework and built Breakwater, which they say blocks all attacks tested in their study.
The framework argues: “The surface, not the model, governs autonomy.”
As we read it, the paper’s harness finding points the same way: it reports different scores for the same model in Claude Code and in Codex. The framework does not discuss Singularity-Bench.
What Is Not Established
The authors say “matching the human baselines is far from saturating Singularity-Bench” and that Opus 5.5’s raw scores leave substantial headroom on all six tasks. They also write that “the scale of the tasks is smaller than that required for true RSI due to limited resources”, calling the benchmark a proxy for the skills such an agent needs.
The paper’s 19 authors list affiliations including Carnegie Mellon, NVIDIA, Meta, Amazon AGI and Together AI; it thanks NVIDIA for the compute and lists funding from Google, Amazon, Intel and others. It does not list Anthropic as an author affiliation or funder. The results are a preprint and were not independently verified; the authors were not contacted for this piece.
This Week In The AI Supercycle
AI Lost Its Central Banker, the Lab Oligopoly, and the Future of Open Source
This week’s issue: the Nvidia leadership change, read as the AI build-out losing its central banker: who now finances the GPU landlords, backstops the data centres and sponsors open models, and why two labs are becoming the bank.
In a Carnegie Mellon-led preprint, Opus 5.5 in Claude Code is the only one of 11 agents to match or beat production AI tools on all six tasks, earning 11.52 USD per GPU-hour on inference against SGLang’s 8.18, at an API cost of $425.41 against $4.36 for the cheapest agent tested.
A note on sourcing. The tasks, scores, costs, affiliations and limitations come from the preprint “Singularity-Bench: Can Agents Build Agents?” (arXiv:2611.16591), submitted on 12 November 2026 and read in full on 14 November; Rohan Paul’s post on X flagged it. Ratios are our arithmetic. The results have not been peer reviewed, and the authors were not contacted for this piece.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
