Nvidia’s GB300 Serves 25x the AI Agents of H100, Epoch Says

Epoch AI estimates that Nvidia’s GB300 provides about 25 times the H100’s inference capacity per chip on the agentic workloads it models, with only 2.5 times the 8-bit compute. In a report published on 9 November, the research group says that comparing FLOP/s alone “would therefore underestimate GB300’s inference capacity relative to H100 by a factor of ten.”

Epoch’s new Inference Performance Index (IPI) adds memory and inter-chip communication to raw compute. Epoch’s dataset includes two chips with the same compute: the H100 and H200 both list 1,979 trillion FP8 operations per second, yet the H200 scores about 3.1 times as high, by our arithmetic from Epoch’s scores.

Business Pill · THE MEMORY WALL

A short Business Engineer explainer on the memory wall, why memory capacity and bandwidth limit computing. It teaches the general idea only and says nothing about Epoch’s index.

What Epoch Measured

The index “compares how many requests each chip can serve at once across a range of required output speeds”. Scores are expressed in GB300 equivalents, with the GB300 NVL72 scoring one by definition. A score of two, Epoch says, means a chip can serve twice as many users or agents at once as a GB300.

Version 1 focuses on agentic workloads. Epoch says agentic inference already accounts for the bulk of token use on OpenRouter, where in late September agentic API keys consumed roughly seven times as many tokens as human API keys. The modelled workload pairs a long cached context of 262,144 tokens with 1,024 new tokens per prefill.

Epoch estimates performance with Nvidia’s AISimulate tool from five hardware inputs: arithmetic throughput, memory capacity, memory bandwidth, interconnect bandwidth, and interconnect grouping and latency. It averages each chip’s capacity, relative to GB300, across four model architectures: GLM-5.2, Kimi K3, Qwen3-Coder-480B and a proxy for MAI-Base-2. Output speeds from 50 to 400 tokens per second per request are compared.

What Was Posted on X

Epoch AI posted on 9 November that it estimates FLOP/s alone understates GB300’s inference advantage over H100 by 10x, and introduced the Inference Performance Index, which accounts for memory bandwidth and inter-chip communication alongside compute. The figures in this piece come from Epoch’s report and hardware dataset, linked above.

The Scores

In Epoch’s dataset, Nvidia’s Vera Rubin (VR200) scores 2.18 and AMD’s MI455X 2.89. The GB300 NVL72 scores 1, the B300 HGX 0.54, the B200 0.39, the H200 0.12 and the H100 0.04. AMD’s MI355X scores 0.55.

System configuration also matters, Epoch says. It estimates the GB300 NVL72 has about 1.9 times the modelled capacity per GPU of the B300 HGX, although both use Blackwell Ultra GPUs. The GB300 system connects groups of 72 GPUs through high-bandwidth links, against groups of eight for the B300 HGX.

Epoch says memory and inter-chip communication can matter more than raw compute for inference. Its dataset lists the same FP8 compute for the H100 and H200, but 141 GB of memory at 4.8 TB/s for the H200 against 80 GB at 3.35 TB/s for the H100. The scores of 0.123 and 0.04 differ by about 3.1 times, by our arithmetic.

Where Google’s TPUs Sit

Epoch’s dataset also scores custom chips. Google’s TPU v8i scores 0.91 and TPU v7 Ironwood 0.74, Microsoft’s Maia 200 0.68, and Amazon’s Trainium3 0.35. Google is said to be adding two eighth-generation TPU variants, bringing its lineup to four, as we reported in Google said to add secure, dual-chip TPUs.

The FLOP/s ranking and the index ranking diverge here too. The TPU v8i lists about 2.02 times GB300’s FP8 compute but scores 0.91; Ironwood lists 0.92 times GB300’s FP8 compute and scores 0.74, by our arithmetic from Epoch’s dataset. For market context, we tracked how AMD, Broadcom, Alphabet and TSMC added about $376 billion in market value while Nvidia lost about $184 billion by mid-morning on Monday, by our arithmetic.

A Common Unit for Capacity

Epoch says GB300 equivalents give a common unit to estimate a data center’s total inference capacity, “or even the world’s”. The report does not give a world total. Epoch used the same unit in its October estimate of how many AI agents chips shipped through 2027 could run, and its earlier work on falling hardware costs is covered in the cost of AI performance falls 13x every year.

The ordering lines up with outside benchmarks, Epoch says: SemiAnalysis InferenceX’s Kimi K3 results show Rubin and GB300 ahead of B300 and MI355X, with H200 far behind. In its validation, GB300’s measured Kimi K3 throughput closely tracked predictions assuming 50% of modelled performance.

The Structural Read

As we read the index, it changes the unit of comparison from peak arithmetic to how many agents a chip can serve at once at a required speed. Epoch says memory and inter-chip communication can matter more than arithmetic throughput, and its scores are modelled hardware potential rather than measured results.

Three Implications

FOR BUYERS OF COMPUTE Epoch says GB300 equivalents give a common unit to estimate a data center’s total inference capacity, which can be compared across vendors and systems.

FOR SYSTEM DESIGN Epoch estimates the GB300 NVL72 at about 1.9 times the B300 HGX per GPU on the same Blackwell Ultra chip, with groups of 72 linked GPUs against eight.

FOR CUSTOM SILICON In Epoch’s dataset, Google’s TPU v8i scores 0.91 and Ironwood 0.74 against GB300’s 1, with Microsoft’s Maia 200 at 0.68 and Amazon’s Trainium3 at 0.35.

The Business Engineer Lens

This story maps onto the Business Engineer framework The AI Memory Chokepoint.

The framework says “HBM growth is outpacing accelerator demand, making memory the rate-limiting factor in the AI scaling equation.”

As we read it, Epoch’s index is a way to put a number on that constraint chip by chip: it scores inference capacity using memory capacity, memory bandwidth and interconnect alongside compute. The framework predates the index, and Epoch does not cite it.

What Is Not Established

The scores are modelled, not measured. Epoch says they “describe modeled hardware potential” and do not account for differences in software maturity. It notes that a geometric mean instead of its arithmetic mean would substantially lower Hopper’s scores, and that changing the model basket can change a chip’s score. For Kimi K3 alone, it predicts about a 46-fold GB300 advantage over H200, against eightfold across the basket. Epoch has not established that its precision choices preserve equivalent model quality.

The Bottom Line

Epoch’s index rates chips by how many agents they can serve at once, not by peak arithmetic. On that measure the GB300 comes out about 25 times an H100 on 2.5 times the FP8 compute, and two Nvidia chips with identical compute differ about 3.1 times, by our arithmetic from Epoch’s dataset.