A model that reads one sentence one word at a time has half forgotten the start by the time it reaches the end.
The fix was a new design: read every word at once, and for each word ask which other words matter most.
A Worked Example
Take the sentence โShe sat on the bank of the river.โ Is bank a place for money, or the side of a river? Read one word at a time, the clue is half forgotten by the end.
Read all at once, bank looks hard at river, and the meaning is settled. That looking is called attention. The design built on it is the transformer.
Why the Design Scaled
Two things followed. Because every word is read at once, the work can be split across thousands of chips.
That is what made today’s huge models possible.
The Bill
But every word looks at every other word. So if you double the text, the work roughly quadruples.
Attention weighs every word against every other. That is its power, and its bill.
Doubling the text roughly quadruples the work, because every word looks at every other word. Rule of thumb from the video.
Transformers and Attention builds on GPUs. Transformers read every word at once so the work splits across thousands of chips, the side-by-side sums GPUs are built for.
Transformers and Attention constrains Context Window. Attention weighs every word against every other, so doubling the text roughly quadruples the work a longer window must pay for.
Scaling Laws builds on Transformers and Attention. Reading every word at once let work spread across thousands of chips, which made today’s huge models and their scaling possible.
Embeddings builds on Transformers and Attention. Both lessons use the word bank to show meaning depends on context; embeddings turn that meaning into coordinates on a map.
The Business Engineer framework maps where each part of the AI stack sits and who holds the leverage. These pills are short lessons from the same library.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
