Multimodal Explained: AI Models That Work With Images, Sound and Video

Business Pill 50 · Models that see and hear

A multimodal model works with more than text: images, sound and video too. That matters because most work is not typed.

A short explainer video, under a minute. It uses a story about a customer with a photo of a broken part.

At a glance

The Short Answer

A customer walks up to a helpdesk holding a photo of a broken part. The old robot only reads text. “Please describe it in words,” it says.

She tries: “It is the bendy thing near the door.” A newer robot simply looks at the photo. “That is the hinge. Part 47.”

What Multimodal Means

That ability is called multimodal. It describes a model that works with more than text: images, sound and video too.

Why It Matters

It matters because most work is not typed. It is photos from the field, screenshots, phone calls and scanned forms.

A text-only model cannot take any of that in until a person puts it into words first.

Two Cautions

A picture costs more to process than a sentence.

And the robot can misread a photo with full confidence. A confident answer from a photo still needs checking.

The Question for Your Own Work

Most work is never written down. A multimodal model can see it and hear it.

The video’s question is which of your work is stuck because it is not text.

Related Frameworks

More Business Pills

• The Memory Wall: Why a Faster AI Chip Is Not Faster AI
• Tokens per Watt: What an AI Data Centre Actually Produces
• Why AI Can’t Be Both Instant and Cheap: Latency vs Throughput
• The Model and the Harness: Why Same-Model Products Differ
• Context, Not Capability: Why a Smart AI Model Gives Poor Answers
• Discardable Software: When Code Is Cheap Enough to Throw Away
• Human in the Loop vs Human on the Loop: Supervising AI Agents
• The AI Audit Problem: When Making Work Is Cheaper Than Checking It
• AI Evaluation as Acceptance Test: How to Know It’s Good Enough
• Extensibility Is Control: Who Holds the Power in an AI Product
• RLHF Explained: How an AI Model Learns What People Prefer
• Pretraining Explained: How an AI Model Learns Before Anyone Teaches It
• Fine-Tuning Explained: How to Adapt a General AI Model to One Job
• Tokens Explained: The Unit AI Reads, Writes and Bills In
• Embeddings Explained: How a Machine Compares Meaning
• RAG Explained: How a Model Answers From Your Documents
• AI Hallucination Explained: Why Models State False Things
• Distillation Explained: How a Small Model Learns From a Large One
• Reasoning Models Explained: What Changes When AI Thinks First
• Tool Use Explained: How an AI Model Goes From Text to Action
• Capex and Depreciation Explained: Why Chip Lifetime Drives AI Profits
• Run-Rate Revenue Explained: What an AI Company’s Number Means
• Backlog Explained: Revenue That Is Signed but Not Yet Earned
• Switching Costs Explained: Why It Is Hard to Leave an AI Supplier
• Temperature Explained: The Dial That Sets How Predictable an AI Model Is
• Guardrails Explained: Rules Enforced Around an AI Model, Not Inside It
• Benchmarks Explained: Why a Test Score Is Not Your Own Result
• Agent Memory Explained: How an AI Assistant Remembers You Between Chats
• MCP Explained: The Shared Plug That Connects AI Models to Tools
• Quantization Explained: How a Large AI Model Is Shrunk to Fit
• Parameters Explained: Where an AI Model’s Knowledge Lives
• Scaling Laws Explained: Why AI Labs Keep Building Bigger
• Synthetic Data Explained: When a Model Writes Its Own Training Examples
• Transformers and Attention Explained: How AI Reads Every Word at Once
• Context Window Explained: How Much Text an AI Model Can Keep in View
• Prompt Injection Explained: When Text a Model Reads Gives the Orders
• Inference Explained: The Cost Paid on Every AI Answer
• AI Agents Explained: From Answering to Doing