Diagram and data compiled by TrendForce (July 2026).
A July 2026 TrendForce diagram adds high-bandwidth flash and a pooled SSD tier to the inference stack — a signal that the next cost battles in AI infrastructure will be fought in memory, not only in compute.
THE MEMORY HIERARCHY — TRAINING VS. INFERENCE (TRENDFORCE, JULY 2026)
5 Tiers
Training stack SRAM → HBM → DRAM → Local SSD → Shared Storage
7 Tiers
Inference stack Adds HBF + SSD POD (intended positioning, not shipping)
HBF
High-Bandwidth Flash slots between HBM and DRAM
SSD POD
Pooled SSD tier slots between Local SSD and Shared Storage
Green tiers = newly added for inference. TrendForce: intended positioning, not current market availability.
A diagram published by market-research firm TrendForce in July 2026 maps the shift in AI memory architecture as the industry moves from model training to large-scale inference. The training hierarchy it depicts is the one the industry has lived with for several years: on-chip SRAM at the top, then HBM (high-bandwidth memory — the stacked, high-cost DRAM that sits closest to the GPU die), then conventional DRAM, local SSDs, and shared storage at the base. Familiar, well-understood, already expensive.
The inference hierarchy in TrendForce’s diagram adds two tiers shown in green: HBF — high-bandwidth flash — inserted between HBM and DRAM, and an “SSD POD” tier inserted between local SSDs and shared storage. The green coding signals that these are new additions specific to the inference use case. TrendForce is explicit that the diagram reflects intended positioning of each memory type, not current market availability. HBF and SSD POD are best read as emerging categories — a research house’s framework for where the industry appears to be heading — rather than standardized, shipping product tiers.
That caveat matters and should travel with every reading of the diagram. Memory-industry transitions take years; the specific tier names are not yet standardized across vendors; and whether HBF or SSD POD become discrete, widely-adopted product categories on any particular timeline is unproven. What is notable is that a serious industry research firm is redrawing the stack at all — and redrawing it specifically for inference.
The key insight: The attention on GPUs has made inference look like a compute problem. TrendForce’s diagram frames it differently: serving AI at scale is fundamentally a memory-capacity problem, and the hierarchy is being extended specifically to address it — with the critical caveat that the new tiers are directional, not yet delivered.
The Stack, Layer by Layer
On-Chip SRAM
TRAINING + INFERENCE
Fastest, smallest, most expensive. Lives on the die. Present in both stacks.
HBM — High-Bandwidth Memory
TRAINING + INFERENCE
Stacked DRAM bonded close to the accelerator. The bandwidth workhorse of both training and inference — but capacity-limited and expensive per gigabyte.
HBF — High-Bandwidth Flash
INFERENCE — NEW
TrendForce’s intended new tier between HBM and DRAM. Flash with enough bandwidth to serve inference workloads at larger capacity and lower cost than HBM. Emerging category — not a shipping standard.
DRAM
TRAINING + INFERENCE
Conventional system memory. Lower bandwidth than HBM, higher capacity, lower cost. Present in both stacks.
Local SSD
TRAINING + INFERENCE
Node-local flash storage. High capacity, acceptable latency for checkpoint and data loading. Both stacks.
SSD POD
INFERENCE — NEW
TrendForce’s intended tier between Local SSD and Shared Storage. Pooled, disaggregated SSD capacity accessible across nodes. Emerging category — intended positioning, not current availability.
Shared Storage
TRAINING + INFERENCE
Network-attached, distributed object or file storage. Largest capacity, lowest cost, highest latency. The base of the hierarchy in both stacks.
The Structural Read
Training and inference are different businesses shaped by different bottlenecks. Training is a compute-and-bandwidth problem executed in concentrated, time-bounded bursts — you need enormous throughput for weeks, then you stop. Inference at scale is a memory-capacity problem run continuously: many concurrent users, growing context windows, and the expanding KV cache (the memory buffer that stores intermediate attention states across multi-step, agentic sessions) all demand large amounts of fast-enough memory held for a long time, around the clock.
HBM (high-bandwidth memory) is excellent at the fast end of that requirement — but it is physically constrained in how much capacity you can stack close to a die, and it is among the most expensive memory per gigabyte available. Standard DRAM and local SSDs offer more capacity at lower cost, but the latency and bandwidth gap between them and HBM is significant. The inference bottleneck lives precisely in that gap. TrendForce’s two new tiers — HBF and SSD POD — are attempts to build out the middle of the hierarchy: memory fast enough and large enough for inference, at a cost per gigabyte that makes serving tokens economically rational at scale.
Structural Lens
The Missing Middle of the Stack
Every technology stack has a “missing middle” — the band between the premium-fast-expensive tier and the commodity-slow-cheap tier where the real volume eventually concentrates. In memory, HBM owns the premium band; shared storage owns the commodity base. The inference era is creating commercial gravity in the middle. Whoever fills it cost-effectively captures a real and growing slice of AI infrastructure spend. The honest bracket: this is a research firm’s directional map, not a confirmed product roadmap — the timelines and form factors are unproven.
“The price of an AI token is the sum of all the layers beneath it — compute, memory, networking, energy, and foundry. The GPU gets the narrative; the memory hierarchy absorbs much of the cost. When the hierarchy is restructured, so is the unit economics of inference.”
Three Implications
IMPLICATION 1 — INFERENCE IS A MEMORY PROBLEM, NOT ONLY A COMPUTE ONE
The GPU narrative dominates AI infrastructure coverage, but serving models at scale is increasingly bounded by how much fast memory you can afford to keep online. Long context windows and accumulating KV caches in a
Gennaro is the creator of FourWeekMBA, which reached about four million business people, comprising C-level executives, investors, analysts, product managers, and aspiring digital entrepreneurs in 2022 alone | He is also Director of Sales for a high-tech scaleup in the AI Industry | In 2012, Gennaro earned an International MBA with emphasis on Corporate Finance and Business Strategy.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
