Apple ML Research has released RSI-Inference Bench, a benchmark that asks whether AI coding agents can improve the software that serves AI models. On its leaderboard, published with the code on 16 October 2026, Anthropic’s Sonnet-5.5 ranks first of 12 agents. It made model serving 1.68x faster on average (a geometric mean) across 27 optimization and bug-fix tasks, and scored 26.7% pass@1 on adding support for a new model.
The 26.7% is the best Day-0 score on the board, which project lead Ruixiang Zhang calls “Unsaturated” in his announcement on X. He writes that the team “spent $100K API tokens, 50K GPU hours to find out.”
Business Pill · LATENCY VS THROUGHPUT
A Business Engineer explainer on latency and throughput, the two speeds an AI serving system trades off. It explains the concept in general and says nothing about Apple’s benchmark.
The key insight: As we read it, the benchmark splits the job in two, and the scores differ sharply between the halves: the best agent made serving 1.68x faster on average, but scored 26.7% pass@1 on getting a new model to run at all.
Update (26 Oct): Google’s Gemini-4-Argon joins the leaderboard at sixth. The leaderboard update (snapshot of 25 October, 22:08 UTC) now ranks 15 agents, up from 12 at launch. Gemini-4-Argon enters sixth with a rating of 1675.64, just behind GPT-6-Astra at 1676.18; Sonnet-5.5 still leads, now at 2043.92, ahead of Opus-5.5 at 1903.12.
On Day-0 Model Support, Gemini-4-Argon scored 6.7% pass@1 and fully passed one of the 10 tasks, the only agent outside Anthropic and OpenAI with a pass; on optimization its 1.39x speed-up ranks fourth, ahead of GPT-6.1-Sol (1.37x) and GPT-6-Astra (1.35x). Apple researcher Ruixiang Zhang, the project lead, wrote on X that it is “the first model outside @AnthropicAI & @OpenAI to pass a Day-0 Model Support task.” Sonnet-5.5’s rating was 2022 at launch.
The repository describes 37 tasks drawn from two production inference engines, vLLM (20 tasks) and SGLang (17). An agent gets the engine at a pinned commit, a model and 1 to 8 GPUs, and works offline for up to 8 hours; a separate verifier builds the edited engine, serves the model and measures it.
The tasks follow a model’s life in production. Day-0 Model Support (10 tasks) asks the agent to serve a newly released model and is scored pass or fail. Inference Optimization (20 tasks) asks it to make serving faster. Bug Fix (7 tasks) asks it to find and fix a slowdown. The last two are scored by measured speed-up, and a result that fails its correctness check counts as 1.00x.
Hardware is split between Nvidia H100 (17 tasks) and B200 (20 tasks). The README counts 30 mixture-of-experts models among the 37 tasks, and 13 tasks that use FP4 precision.
Per the RSI-Inference Bench leaderboard (Apple ML Research, data snapshot 12 October 2026): each agent’s speed-up over the task baselines, as a geometric mean over 20 throughput-optimization and 7 bug-fix tasks on vLLM and SGLang. A result that fails its correctness check counts as 1.00x.
We spent $100K API tokens, 50K GPU hours to find out. We built environments and benchmark from real-world #vLLM and #SGLang tasks, covering the full lifecycle of model inference engineering:
๐ Day-0 Model Support โก Inference Optimization ๐ง Bug Fix
โ Unsaturated: best agent at 26.7% pass@1 โ Comprehensive coverage of the inference stack โ Verifiable rewards: binary and continuous
Ruixiang Zhang, the project lead named in Apple ML Research’s repository, announced RSI-Inference Bench on X on 16 October 2026. The post gives the team’s spend on the work and the best agent’s 26.7% pass@1; the leaderboard scores in this piece come from the project’s own leaderboard page.
The Results
The leaderboard snapshot of 12 October 2026 ranks 12 agents by a joint rating over both tracks. Sonnet-5.5 leads with a rating of 2022, ahead of Opus-5.5 (1889), Fable-5.1 (1696), GPT-6.1-Sol (1680) and GPT-6-Astra (1667).
On Day-0 Model Support, Sonnet-5.5 scored 26.7% pass@1 and fully passed 5 of the 10 tasks at least once. Opus-5.5 scored 13.3%. Fable-5.1, GPT-6.1-Sol and GPT-6-Astra scored 10.0%, and the other seven agents scored 0%. By our count of the task table, 5 of the 10 new models, including gpt-oss-120b on SGLang and MiniMax-M3 on vLLM, were not fully served by any agent.
On speed, Sonnet-5.5 averaged 1.53x on the throughput tasks and 2.19x on the bug-fix tasks, for 1.68x overall. Opus-5.5 reached 1.48x, Fable-5.1 1.41x and GPT-6.1-Sol 1.37x. The two Gemini Flash agents finished last at 1.07x and 1.04x, with 3 and 4 results that failed the correctness gate.
What a Run Costs
The leaderboard prices every attempt from published API prices as of 9 October 2026: 658 attempts and 319,023 model requests. The median optimization attempt cost $39.14 for Sonnet-5.5, $55.59 for Opus-5.5, $11.68 for GPT-6.1-Sol and $0.89 for DeepSeek-V4.1-Flash, which ranked seventh on speed at 1.24x.
Sonnet-5.5 also used most of its time: a median of 7.1 agent hours per optimization attempt, against an 8.3-hour limit.
Agents Working on AI Infrastructure
The benchmark’s name refers to self-improving inference systems, and its abstract says a coding agent edits a pinned inference engine and is scored on measured throughput against the untouched baseline, gated on correctness.
As we read it, speed-up is the easier half to show: every agent on the board scored above 1.00x on the optimization track, while seven of the 12 scored 0% on Day-0 Model Support.
As we read it, the bug-fix tasks are where agents gained most: Sonnet-5.5 averaged 2.19x there against 1.53x on the throughput tasks, and the other leaders show the same gap.
As we read it, cost and rank do not line up: the leaderboard puts DeepSeek-V4.1-Flash’s median optimization attempt at $0.89 against $39.14 for Sonnet-5.5, for 1.24x against 1.68x.
Ruixiang Zhang, Apple ML Research, on X, 16 October 2026
“Can AI agents improve the systems that run AI?”
Three Implications
INFERENCE ENGINE TEAMS The tasks come from vLLM and SGLang work, and the verifier builds the edited engine, serves the model and measures it.
MODEL LABS The leaderboard names each lab’s agents and gives a speed-up and pass@1 for each on the same 37 tasks.
AI INFRASTRUCTURE BUYERS The leaderboard prices every attempt, from $0.89 to over $100 at the median, so the score can be read against what it cost to get.
The framework puts it this way: “Inference is a revenue engine: continuous, distributed across millions of applications and agents, incurring marginal cost with every query.”
As we read it, Apple’s benchmark measures whether agents can lower that marginal cost by making serving faster. The framework was written in March 2026 and does not discuss this benchmark.
What Is Not Established
The accompanying paper is listed as “coming soon” in the repository, so the method is known only from the README, the leaderboard page and Zhang’s post. The $100K and 50K GPU-hour figures come from that post; the leaderboard itself gives costs per attempt.
The scores come from Apple ML Research’s own runs on its own tasks. We read the repository and the leaderboard data on 17 October 2026, did not run the benchmark and did not contact Apple or the model makers.
Business Engineer Framework
The State of The Inference Economy
A Business Engineer framework on why inference, not training, is where the AI economy now makes and spends its money.
Apple ML Research’s RSI-Inference Bench tests whether AI agents can support, speed up and repair the vLLM and SGLang engines that serve AI models, across 37 tasks on H100 and B200 GPUs.
Its leaderboard ranks Sonnet-5.5 first, with a 1.68x average speed-up and a 26.7% pass@1 on Day-0 model support, the best score on the board.
A note on sourcing. On 17 October 2026 we read Apple ML Research’s RSI-Inference Bench repository (README and citation file), the data behind its leaderboard page (snapshot of 12 October 2026) and project lead Ruixiang Zhang’s post on X of 16 October 2026. Every score and cost comes from the leaderboard; the $100K and 50K GPU-hour figures come from Zhang’s post. The count of new models no agent fully served is ours, from the leaderboard’s task table.
The paper is listed as coming soon; we did not run the benchmark or contact Apple. The references to our earlier reporting link to our own pieces.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
