The capacity of the world’s AI chips to run AI agents grew 4.6 times in a year, while their raw computing power grew 2.6 times, Epoch AI estimated on 18 November, as we reported. Yet renting an Nvidia B200 chip cost $6.94 an hour on 19 November, up 23.5% since the start of September on Silicon Data’s index, as we covered.
This explainer covers what Epoch’s Inference Performance Index measures, why memory matters more than raw speed for running AI, and what the sources say about why rents went up anyway. Memory is explained in our HBM explainer, and faster ways to generate text in our diffusion model explainer.
We track how AI models compare on our AI model benchmarks page, and chip rents and compute contracts on our AI compute commitments tracker.
Business Pill: GPUs Explained
An evergreen Business Engineer explainer on why AI runs on GPUs. It teaches the general idea only and does not discuss Epoch’s index or chip rents.
The key insight: By Epoch AI’s Inference Performance Index, AI chips’ capacity to run agents grew 4.6 times in a year against 2.6 times for raw compute, and Epoch says memory and chip-to-chip links can matter more than arithmetic for inference; Silicon Data’s B200 rent still rose 23.5% to $6.94 an hour.
What Is Inference, and Why Measure It Separately?
Training builds an AI model. Inference is running it: every answer, every agent step and every tool call uses chip time. On OpenRouter, a platform that routes developer requests across AI model providers, agentic API keys consumed roughly seven times as many tokens as human API keys in late September 2026, Epoch says.
Chips are usually compared by FLOP/s, the number of arithmetic operations they can do per second. Epoch AI argues that this misses most of what matters for inference: “memory and inter-chip communication can matter more than arithmetic throughput for inference performance.”
Agents are also taking on longer jobs. METR said the length of task frontier agents finish half the time is doubling about every 3.6 months and now reaches up to 27 hours on tasks that can be checked automatically, as we reported.

What Does the Inference Performance Index Measure?
Epoch’s index “compares how many requests each chip can serve at once across a range of required output speeds.” Version 1 focuses on agentic workloads with long, cached contexts and combines results across four representative models.
Scores are in GB300 equivalents, with Nvidia’s GB300 scoring one. A score of two means Epoch estimates a chip can serve twice as many users or agents at once as a GB300.
Why Does Memory Matter More Than Speed?
When a model answers, it first reads the input and stores it in a cache, then writes the reply one token at a time, reading that cache on every step. Model weights sit in high-bandwidth memory, or HBM, and are pulled onto the chip as needed, Epoch explains.
A coding agent can carry hundreds of thousands of tokens of history from one request to the next. Epoch’s example uses about 200,000 cached tokens. Holding and reading that cache uses memory and bandwidth, not just arithmetic.
That is why Epoch estimates the GB300 provides about 25 times the H100’s inference capacity per chip on its agentic workloads, with only 2.5 times the 8-bit FLOP per second. “Comparing FLOP/s alone would therefore underestimate GB300’s inference capacity relative to H100 by a factor of ten,” Epoch says.
How Much Did Capacity Grow?
Applying the index to its chip-sales estimates, Epoch puts the agentic inference capacity of AI chips sold to date at about 1.45 million GB300 equivalents in the second quarter of 2025 and about 6.67 million a year later, 4.58 times. Peak 8-bit compute grew 2.59 times over the same period.
Nvidia’s Blackwell and Blackwell Ultra chips “account for 74% of the added capacity”, Epoch says, with GB300 systems delivering “roughly 8× the inference capacity of H200s per GPU.” Inference capacity per unit of peak compute rose by about 77% over the year.
Epoch expects the pace to slow. Chips sold in the second quarter of 2026 added 3.3 times the capacity of those sold a year earlier, and its modelled capacity per GPU rises 8 times from H200 to GB300 but only 2.2 times from GB300 to Vera Rubin VR200.
What Happened to Chip Rents?
Silicon Data’s B200 index stood at $5.62 an hour entering September and $6.94 on 19 November. For one chip running all year, that is about $60,800 of rent against about $49,200, by our arithmetic.
On Bloomberg’s Odd Lots podcast, Stanley Lee of Liquid Compute said some of the demand behind the rise is real, but “a lot of it is the cost of capital”, according to an episode summary cited in our rents piece.
The Structural Read
As we read it, more capacity per chip has not, so far, made AI computing cheaper to rent on Silicon Data’s B200 index.
As we read it, the gap between capacity and compute is why chip comparisons that quote only FLOP/s can mislead buyers of inference.
Epoch AI, Inference Performance Index
“Comparing FLOP/s alone would therefore underestimate GB300’s inference capacity relative to H100 by a factor of ten.”
Three Implications
FOR CHIP BUYERS Epoch says FLOP/s alone can understate inference capacity by a factor of ten.
FOR AI LABS Epoch expects capacity growth to slow as generational gains shrink.
FOR RENTERS B200 rent rose 23.5% to $6.94 an hour on Silicon Data’s index.
What Is Not Established
Epoch’s figures are modelled estimates. Its uncertainty intervals cover shipment and hardware-mix uncertainty, 4.2 to 5.1 times for inference growth, but not uncertainty in modelled inference performance, and differences in software efficiency are not modelled. The B200 index is one price series, and the reasons for its rise are the views of the people quoted.
The Bottom Line
By Epoch AI’s Inference Performance Index, the capacity of AI chips sold to date to run agents grew 4.6 times in a year against 2.6 times for raw compute; Epoch says memory and chip-to-chip links can matter more than arithmetic for inference. B200 rents still rose 23.5% to $6.94 an hour, which a Liquid Compute executive said was partly real demand but “a lot of it is the cost of capital”, per an episode summary.
97,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.
Frequently Asked Questions
What is Epoch AI's Inference Performance Index?
It is Epoch AI's measure of how many requests an AI chip can serve at once across a range of required output speeds, on agentic workloads with long, cached contexts. Scores are in GB300 equivalents, with Nvidia's GB300 scoring one.
Why is FLOP/s a poor guide to inference?
Epoch says memory and inter-chip communication can matter more than arithmetic throughput. It estimates the GB300 has about 25 times the H100's inference capacity per chip with only 2.5 times the 8-bit FLOP per second, so FLOP/s alone would underestimate it by a factor of ten.
How fast did inference capacity grow?
Epoch estimates the agentic inference capacity of AI chips sold to date grew 4.6 times from the second quarter of 2025 to the second quarter of 2026, against 2.6 times for peak 8-bit compute, with Nvidia's Blackwell chips accounting for 74% of the added capacity.
Why did AI chip rents rise anyway?
Silicon Data's B200 index rose 23.5% to $6.94 an hour between early September and 19 November. On Bloomberg's Odd Lots podcast, Stanley Lee of Liquid Compute said some of the demand is real but a lot of it is the cost of capital, per an episode summary,
A note on sourcing. The capacity figures, the 74% Blackwell share, the 8x and 2.2x per-GPU figures, the 77% rise in capacity per unit of compute, the 3.3x additions and the uncertainty intervals are from Epoch AI’s 18 November data insight. The description of the index, the cache example and the 25x, 2.5x and factor-of-ten comparison are from Epoch’s 9 November Inference Performance Index report. The B200 rental prices are Silicon Data’s index as reported in our 21 November piece; the Odd Lots quote is from an episode summary cited there. The METR figures are from our 21 November piece on Jack Clark. The per-chip-year rents are our arithmetic.
Sources: Epoch AI (Jason Li), ‘Agentic inference capacity grew 4.6x in a year, versus 2.6x for compute capacity’, data insight, 18 Nov 2026 · Epoch AI, ‘Inference Performance Index: changing the way we measure inference capacity’, report, 9 Nov 2026 · Silicon Data, B200 rental index, via our 21 Nov piece









