An AI chip has two jobs. It calculates, and it fetches. The numbers that make up the model sit in memory, and they have to be carried to the processor before any calculation can happen.
A model is large. Each time it produces one word, it has to read almost all of itself, which means billions of numbers moved from memory to the processor. So the speed of an answer depends on two things: how fast the processor calculates, and how fast memory can deliver the model to it.
What Memory Bandwidth Is
The delivery speed is called memory bandwidth. Think of it as the width of the road between memory and the processor. A wide road moves a lot of the model per second. A narrow road moves little, however powerful the processor at the far end.
For years, processors got faster much more quickly than memory did. The road became the narrow part. The processor is ready, and it is waiting for data.
A Worked Example
The video uses round numbers. A model takes up 100 GB, and memory can deliver 1,000 GB per second. The model can be read 10 times a second, and since each word needs one read, that is 10 words per second at most. This holds however fast the processor is.
Now double the speed of the processor. Nothing changes: still 10 words per second. Double the memory bandwidth instead and you get 20. The point is which change helps, not the specific numbers.
Doubling the processor changes nothing in this example. Doubling memory bandwidth doubles the ceiling. Round numbers for illustration.
Why It Matters
Engineers call this the memory wall: the point where the limit is no longer how fast you can compute, but how fast you can feed the computer. The video gives three reasons it matters.
First, fast memory becomes as scarce and as valuable as the processors themselves. Second, smaller models run faster simply because there is less to read. Third, serving many users together helps, because one read of the model can answer many requests at once.
The Business Engineer framework maps where each part of the AI stack sits and who holds the leverage. These pills are short lessons from the same library.
Gennaro is the creator of FourWeekMBA, which reached about four million business people, comprising C-level executives, investors, analysts, product managers, and aspiring digital entrepreneurs in 2022 alone | He is also Director of Sales for a high-tech scaleup in the AI Industry | In 2012, Gennaro earned an International MBA with emphasis on Corporate Finance and Business Strategy.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
