There are two ways for an AI system to be fast. The first is latency: how long one person waits for an answer. The second is throughput: how many answers the system produces in total every second.
Think of a sports car and a bus. The car leaves the moment you sit down, but it carries one person. The bus carries many, but it waits until it is full.
Batching
AI chips work the same way. They can answer one request at a time, immediately. Or they can collect many requests and process them together. That is called batching.
With batching, the chip does far more work in total, but each request waits a little longer.
A Worked Example
The video uses round numbers. One at a time, a chip gives one answer every second. In a batch of eight, it takes two seconds but delivers eight answers. That is four answers per second, four times the output from the same chip.
So each answer costs a quarter as much, but everyone waited twice as long. This is the trade-off: you can make AI faster for each person, or cheaper for everyone. Pushing one costs you the other.
Four times the output from the same chip, at the cost of each request waiting twice as long. Round numbers for illustration.
Why the Same Model Has Different Prices
The trade-off explains why the same model is sold at different prices. A live conversation needs its answer now, so it pays the premium. A report that runs overnight can wait in the queue, so it gets the discount.
And as more work is handed to agents running in the background, more of it can wait. More of it takes the bus.
Why AI Canโt Be Both Instant and Cheap builds on Inference. Batching lets a chip serve many requests together at the cost of each waiting longer, a choice in how inference is served.
Why AI Canโt Be Both Instant and Cheap builds on Percentiles. Latency is how long one person waits, and percentiles show the slow end of those waits that an average hides.
Rate Limits(lesson unlocks soon) contrasts with Why AI Canโt Be Both Instant and Cheap. Both are about busy capacity: rate limits cap requests per minute, batching trades each user’s wait for more answers overall.
The Business Engineer framework maps where each part of the AI stack sits and who holds the leverage. These pills are short lessons from the same library.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
