Multimodal Explained: AI Models That Work With Images, Sound and Video

Business Pill · Models that see and hear

A multimodal model works with more than text: images, sound and video too. That matters because most work is not typed.

A short explainer video, under a minute. It uses a story about a customer with a photo of a broken part.

At a glance

What Multimodal Means

That ability is called multimodal. It describes a model that works with more than text: images, sound and video too.

Why It Matters

It matters because most work is not typed. It is photos from the field, screenshots, phone calls and scanned forms.

Two Cautions

A picture costs more to process than a sentence.

Table of Contents

The Short Answer

A customer walks up to a helpdesk holding a photo of a broken part. The old robot only reads text. “Please describe it in words,” it says.

She tries: “It is the bendy thing near the door.” A newer robot simply looks at the photo. “That is the hinge. Part 47.”

What Multimodal Means

That ability is called multimodal. It describes a model that works with more than text: images, sound and video too.

Why It Matters

It matters because most work is not typed. It is photos from the field, screenshots, phone calls and scanned forms.

A text-only model cannot take any of that in until a person puts it into words first.

The Idea on One Board

What Multimodal Means

The old robot

The old robot only reads text.

The newer robot

A newer robot simply looks at the photo.

Two Cautions

A picture costs more to process than a sentence.

And the robot can misread a photo with full confidence. A confident answer from a photo still needs checking.

The key insight: It matters because most work is not typed. It is photos from the field, screenshots, phone calls and scanned forms.

The honest limit

A picture costs more to process than a sentence.

And the robot can misread a photo with full confidence. A confident answer from a photo still needs checking.

Where This Shows Up in AI Business Today

From our news coverage

Norway Proposes Temporary AI Glasses Ban in Public Places

Our report on Norway’s proposed temporary ban on AI glasses says that, as NTB reports it, the minister is worried that people risk being photographed, filmed or recorded without knowing it, and that the government wants a temporary ban on AI glasses in selected public places. The report rests on NTB and NRK, not the press release. Images, sound and video are the inputs a multimodal model works with, as this lesson describes.

The Business Engineer Lens

Where this idea sits in the Business Engineer library:

TrendsThe Product Overhang Doctrine“In any product cycle, there are two curves to follow. The first is what your underlying technology can do. The second is what your users can actually do with it.”Read on Business Engineer →StrategyThe AI Value Chain“Today, the AI value chain is built on a complex ecosystem that spans hardware, cloud, software, AI models, and vertical and consumer applications.”Read on Business Engineer →

Business Engineer Framework

The Map of AI

The Business Engineer framework maps where each part of the AI stack sits and who holds the leverage. These pills are short lessons from the same library.

Business Engineer

the Map of AI — 200+ companies across 9 layers

Read →

Business Engineer

the AI value chain — a multi-layered analysis

Read →

Explore the Map of AI →

The Question for Your Own Work

Most work is never written down. A multimodal model can see it and hear it.

The video’s question is which of your work is stuck because it is not text.

The One Question to Ask

Which of our work is stuck because it is not text?

How This Concept Connects

Layer 2 of 8 · Models & Training, on the Business Engineer Concept Map

Multimodal drives the cost of Inference. A picture costs more to process than a sentence, so taking in images and sound raises the cost of each answer.
Computer Use builds on Multimodal. Computer use loops on looking at the screen, clicking and checking what changed, which needs a model that reads images.
Physical AI (lesson unlocks soon) builds on Multimodal. Physical AI models see, decide and move, and seeing the world rather than reading text is the multimodal capability.