Analysis draws on Cerebras CEO Andrew Feldman’s remarks on the Mad Podcast and Gennaro Cuofano’s “Beyond NVIDIA’s Moat” (The Business Engineer). Market facts (HBM supply, chip roadmaps) corroborated by public reporting linked below.
The race for more GPUs and more FLOPS has quietly hit a physical ceiling — and the constraint that now governs AI economics is measured in gigabytes per second, not teraflops.
The Memory Wall — Key Numbers
~140 GB
Weights moved per output token from a 70B-parameter model (roughly 100 HD movies)
On a recent episode of the Mad Podcast, Cerebras CEO Andrew Feldman introduced a framing that cuts through the noise around AI infrastructure: a single output token has a three-stage lifecycle — Training, Prefill, and Decode — and each stage operates under completely different physics. Feldman’s insight, synthesized and extended by Gennaro Cuofano in Beyond NVIDIA’s Moat (The Business Engineer), reframes the entire AI hardware debate. The question was never just “how much compute?” It was always “compute for which stage?”
Training is a one-time cost — enormous, but amortized across millions of queries. Prefill, which reads your prompt and encodes context, is parallelizable: throw more GPUs at it and it goes faster. Decode is the stage that writes your answer, one token at a time, and it cannot be parallelized. Token 500 cannot be computed before token 499. That sequential dependency is a wall, and it is the wall where AI economics are now being decided.
The practical consequence shows up in supply chains. High-bandwidth memory (HBM) — the specialized DRAM that sits closest to the GPU die — is effectively sold out through 2026 at SK Hynix, Samsung, and Micron, with SK Hynix’s late-July 2026 earnings serving as the tightest forward read on capacity. TSMC’s CoWoS advanced-packaging line, which bonds HBM to logic dies, is equally congested. Leading-edge process nodes face the same queue. These are not separate bottlenecks — they are three consecutive stations on the single road that every decode operation must travel.
How the Bottleneck Shifted
2022–2023
Compute scarcity dominates. GPU allocation determines who can train frontier models. FLOPS is the currency.
2024–2025
Reasoning models (OpenAI o-series, DeepSeek-R1) multiply decode passes per query. Output tokens price ~5× input tokens — the first visible signal of memory-bound economics at scale.
2026 (Now)
HBM sold out through year-end across all three major suppliers. CoWoS and advanced-node queues congested. SK Hynix late-July earnings are the market’s forward indicator. Agentic workloads add a second decode multiplier.
Structural Response
Prefill/decode disaggregation accelerates: AWS Trainium for prefill, Cerebras WSE for decode. Value migrates toward memory, packaging, and power infrastructure.
The key insight: In decode, the expensive operation is not arithmetic — it is movement. To emit a single token from a 70B-parameter model, the system must transfer the model’s entire weight matrix from memory to compute, perform one small calculation, and emit one token. Then repeat. A thousand-word answer moves the equivalent of 100 HD movies a thousand times. Buying more FLOPS does not help. The bottleneck is bandwidth.
The Structural Read
The product directions that define the current frontier — reasoning models, AI agents, verification loops, multi-turn reinforcement — all share one property: they multiply decode. A standard completion is a single decode pass. A reasoning model drafts, decomposes, verifies, and revises — each step is another pass through the memory-bound stage. An agent multiplies again by the number of turns and tool calls. The industry has a name for the aggregate cost effect: the reasoning tax.
The directional figures from current API pricing illustrate the order of magnitude: a query handled as a simple completion costs roughly $0.01; routed through a reasoning-plus-agentic stack, the same query can cost $0.50–$1.00 — a 50–200× range that reflects output tokens already priced at approximately 5× input tokens across major providers. These are directional signals, not accounting-grade figures, but the direction is unambiguous. The frontier is not growing out of the memory wall; it is building products that walk deeper into it.
There is also a demand vector that does not show up in GPU procurement discussions: agents run their coordination and action logic on CPUs, and those CPUs draw on the same constrained memory pool. As Gennaro Cuofano frames in The Agentic CPU Turn, this creates a buried second-order pressure on the very memory infrastructure that the GPU narrative has already saturated. The supply crunch compounds from two directions simultaneously.
Map of AI — Where Value Is Moving
“The memory wall does not threaten NVIDIA’s position — NVIDIA’s moat remains intact for the coming decade. What it does is reveal that the physical floor beneath the model layer — memory, foundry, packaging, power — is collecting a toll on every token produced, regardless of which model produces it. That is where value is migrating at the margin, and it is migration driven by physics, not strategy.”
The second structural consequence is disaggregation. Because prefill and decode obey different physics, the industry is increasingly splitting them onto purpose-built silicon. The AWS Trainium-for-prefill, Cerebras WSE-for-decode pattern is the clearest public example of this logic in production. This is not a niche architectural preference — it is the hardware industry adjusting to a workload that has two fundamentally different bottlenecks residing in the same inference pipeline.
Three Implications
IMPLICATION 1 — Value Accrues to the Physical Floor
HBM, CoWoS advanced packaging, and leading-edge process nodes are the three consecutive toll booths on every inference operation. With all three congested through at least end-2026, pricing power sits with SK Hynix, Samsung, Micron, and TSMC — regardless of which model wins the benchmark wars. The layer that is scarce is the layer that captures margin. SK Hynix’s late-July earnings are the single most important forward-looking read on whether that constraint eases or hardens into 2027.
IMPLICATION 2 — The Reasoning Tax Is a Structural Cost, Not a Transitional One
Every product direction the frontier labs are pursuing — longer context, multi-step reasoning, agentic loops — adds decode passes. The reasoning tax is not a temporary artifact of early-stage models that will engineer itself away; it is an expression of the sequential dependency in autoregressive generation. Companies building on top of frontier APIs need to model this cost as a durable input cost, not a line item that improves with scale. The unit economics of AI products are memory-bound, and that constraint informs every pricing, margin, and product-velocity decision downstream.
IMPLICATION 3 — Disaggregated Inference Is the New Architecture Default
The prefill/decode split is not an optimization — it is a recognition that the two stages require different hardware. As agentic workloads add CPU-side memory pressure alongside GPU-side decode pressure, the monolithic “one chip for inference” model becomes harder to defend economically. Purpose-built silicon for each stage — and the orchestration layer that routes between them — is where the next generation of infrastructure differentiation will be built. The backstop economics of this transition are explored in our AI vendor-guaranteed financing piece.
Business Engineer Framework
The Map of AI — Where This Sits in the Stack
The memory-wall shift maps precisely onto the Map of AI’s physical infrastructure layers — the layer beneath model weights, inference compute, and application logic. Understanding which layer is scarce, which is commoditizing, and which is collecting monopolistic tolls is the analytical lens that separates durable competitive positions from temporary ones. The full 9-layer map, updated for the decode-first era, is the place to orient any infrastructure or productstrategy in AI right now.
NVIDIA’s position is secure — that is not the story. The story is that the AI stack has developed a second center of gravity, and it sits in the physics of moving weights rather than calculating them. Every reasoning model, every agent, every verification loop is another deposit into the decode stage’s bill — and that bill is paid in memory bandwidth that is, right now, structurally sold out. The companies that understand this are designing hardware to split the workload, pricing products to absor
Gennaro is the creator of FourWeekMBA, which reached about four million business people, comprising C-level executives, investors, analysts, product managers, and aspiring digital entrepreneurs in 2022 alone | He is also Director of Sales for a high-tech scaleup in the AI Industry | In 2012, Gennaro earned an International MBA with emphasis on Corporate Finance and Business Strategy.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
