Quantization Explained: How a Large AI Model Is Shrunk to Fit

Business Pill 42 · How a model is shrunk to fit

A model too large for a machine can be made smaller by storing its numbers with less precision. That is quantization.

A short explainer video. The numbers in it are round numbers for illustration.

The Short Answer

An engineer wants to run an AI model on her laptop. But the model needs 16 GB of memory, and the laptop has eight. It does not fit.

What the Model Is Made Of

Inside the model are billions of numbers, each one stored with great precision, such as 0.73281946.

So she rounds them. 0.73 is close enough. Each number now takes a quarter of the space.

Quantization, With Numbers

That rounding is called quantization: storing a model’s numbers with less precision, so it becomes smaller and faster.

In the video’s example, the model shrinks from 16 GB to 4. Now it fits on the laptop, and it answers faster.

The arithmetic uses the video’s round numbers. The 16 GB model is twice the laptop’s 8 GB, so it does not fit. A quarter of 16 GB is 4 GB, which is half of the laptop’s 8 GB, so it does.

Bar chart: a model needing 16 GB shrinks to 4 GB, against a laptop with 8 GB of memory
Sixteen GB does not fit in 8 GB. After rounding, 4 GB does. Round numbers for illustration.

The Price

The price is a little accuracy. Usually the loss is small, but on hard tasks it can show.

Why It Matters

So the question to ask is whether a smaller, cheaper model is good enough for this task.

The video’s advice in one line: round the numbers, shrink the model, then test what you lost.

The Question to Ask

  1. Is a smaller, cheaper model good enough for this task?

How This Concept Connects

Layer 2 of 8 · Models & Training, on the Business Engineer Concept Map

ParametersDistillationOn-Device AIQuantization
  • Quantization builds on Parameters. Quantization stores the model’s numbers, its parameters, with less precision so it becomes smaller and faster.
  • Quantization contrasts with Distillation. Both cut running cost: quantization rounds the same model’s numbers, distillation trains a smaller student on a teacher’s answers.
  • On-Device AI (lesson unlocks soon) builds on Quantization. On-device AI needs a model small enough to live inside the phone, the shrinking that quantization performs.
Business Engineer framework
The State of the Inference Economy
Quantization lowers inference cost and memory needs.
Business Engineer framework
The Apple Silicon Disruption
Shrunken models enable on-device AI.
See all 141 concepts on one map
Get the full Concept Map as a free poster.
Get the map

More Business Pills

Scroll to Top
FourWeekMBA