What Is Perplexity? The AI Number OpenAI Wants to Cap

This is perplexity the measurement, not Perplexity the AI search company. Perplexity scores how well a language model predicts the next word; lower means better predictions.

On Sunday OpenAI published its internal perplexity measurements: its log-perplexity fell from about 0.99 for GPT-3.5 to about 0.76, and its chief scientist proposed common limits on the measure, as we covered in OpenAI’s perplexity disclosure.

We track model scores on our AI model benchmarks page. Below: what the number means, why it keeps falling, and what a limit on it would do.

Business Pill · The Model and the Harness

An evergreen Business Engineer explainer. It teaches the general idea only and does not discuss this case.

The key insight: Perplexity scores how surprised a language model is by the next word, defined as the exponentiated average negative log-likelihood; OpenAI says its log-perplexity fell from about 0.99 to about 0.76, about 2.69 to 2.14 as perplexity by our conversion, and its chief scientist proposes common limits on the measure.

What Perplexity Measures

Perplexity is defined as “the exponentiated average negative log-likelihood of a sequence”, per the Hugging Face Transformers documentation. In plain words: show the model text it has not seen, ask how likely it thought each next word was, and average how surprised it was.

Flow diagram: how perplexity is scored, from unseen text and next-word predictions to average surprise (loss) and its exponential (perplexity)
How perplexity is scored: the model sees text it was not trained on, gives odds for each next word, and its average surprise on a log scale is the cross-entropy loss; turning that log back gives perplexity, per Hugging Face’s definition. OpenAI’s approximate log values: 0.99, 0.91, 0.76; as perplexity 2.69, 2.48, 2.14 by our arithmetic.

The same documentation says perplexity is “equivalent to the exponentiation of the cross-entropy between the data and model predictions.” The cross-entropy, or loss, is the logarithm; perplexity is that number turned back.

OpenAI’s post puts it this way: “Perplexity measures how well a model predicts the next token on average; lower perplexity means better predictions.”

OpenAI's approximate log-perplexity on its internal held-out dataset: about 0.99 for GPT-3.5, 0.91 for GPT-4,
OpenAI’s approximate log-perplexity on its internal held-out dataset: about 0.99 for GPT-3.5, 0.91 for GPT-4, 0.76 for the model that began producing new maths results, and a next window of about 0.68 to 0.7, per its 22 Nov 2026 post. Converted to perplexity, assuming the natural log: 2.69, 2.48, 2.14 and 1.97 to 2.01, our arithmetic.

Why Lower Means Smarter, Per OpenAI

OpenAI’s chief scientist Jakub Pachocki describes pretraining as “training a compressor (or, equivalently, predictor) on a very large corpus of data.” A model that predicts text better has, in his framing, learned more about it.

He writes that GPT-3.5, at a log-perplexity of about 0.99, was “capable of fluent conversation”, and GPT-4, at about 0.91, “was proficient at utilizing chain-of-thought reasoning productively”.

Below 0.8, he writes, OpenAI has “seen the emergence of transformative capabilities in fields like cybersecurity and mathematics”. A model at about 0.76 “started producing groundbreaking results” on open maths problems, per the post.

Why the Number Keeps Falling

Researchers have measured how loss falls as models grow. A 2020 OpenAI paper by Jared Kaplan and colleagues found “The loss scales as a power-law with model size, dataset size, and the amount of compute used for training”, per its arXiv abstract.

A 2022 DeepMind paper, “Training Compute-Optimal Large Language Models”, found that for compute-optimal training “the model size and the number of training tokens should be scaled equally”, per its arXiv abstract.

So more chips, more data and better methods push the number down in a fairly predictable way. Pachocki adds that AI helping with pretraining research could speed this up, “as soon as 2027”.

OpenAI’s Numbers, Turned Into Perplexity

OpenAI reports log values. Converting them back, assuming the natural logarithm used in the Hugging Face definition, perplexity fell from about 2.69 for GPT-3.5 to about 2.48 for GPT-4 and about 2.14 for the 0.76 model, by our arithmetic.

That is about 20% lower than GPT-3.5. Pachocki expects “further transformative capabilities” in a window of about 0.68 to 0.7, which converts to about 1.97 to 2.01, by our arithmetic.

What a Limit on Perplexity Would Mean

Pachocki proposes “common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone”.

His example of a milestone is “the development of a successor to CoT monitors”, the tools that watch a model’s written-out reasoning. The post names no number for the first limit and does not say who would set it.

He writes that publishing the measurements “gives competitors a clearer picture of where we stand, but that concern is outweighed by the urgency of establishing shared safety measures across the industry.”

The Structural Read

As we read it, perplexity is attractive as a shared yardstick because it is one number every lab already tracks, which is the case Pachocki makes.

As we read the post, the weak point is comparability: each lab measures on its own data, which Pachocki names as a problem himself.

Jakub Pachocki, OpenAI Chief Scientist, 22 Nov 2026

“Perplexity measures how well a model predicts the next token on average; lower perplexity means better predictions.”

Three Implications

FOR READERS Perplexity is the exponentiated average negative log-likelihood of text, per Hugging Face; lower means the model predicts the next word better.

FOR LABS OpenAI’s chief scientist proposes common limits on perplexity and other measures that developers would not cross until AI monitoring reaches a concrete milestone.

FOR COMPARISONS OpenAI’s values are approximate and on an internal dataset; its post says labs use different evaluation data.

What Is Not Established

The values are OpenAI’s own, approximate, and measured on an internal dataset no one else can check. Pachocki’s post notes that labs use different evaluation data, so scores are hard to compare across labs. We had not seen on-record responses from other developers by publication.

This Week In The AI Supercycle

A World Without an Order, and an AI Central Bank Without a Successor

This week’s issue: the week AI memory stayed largely sold out while the people holding the shares paid: why the AI Supercycle is not a dot-com replay, read through geopolitics, macro and technology, with the gap between building and financing the widest we’ve recorded.

Read the full issue →

Business Engineer Framework

The AI Supercycle

The Business Engineer map of who builds, finances and captures value in AI.

Read the Map of AI →

The Bottom Line

Perplexity is how surprised a model is by the next word, and OpenAI says its models went from about 2.69 to about 2.14 on that scale, by our conversion. Its chief scientist wants labs to agree a line on it that none would cross until monitoring catches up; whether rivals publish their own scores is the next thing to watch.

97,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.

Frequently Asked Questions

What is perplexity in AI?

Perplexity is defined as the exponentiated average negative log-likelihood of a sequence, per the Hugging Face Transformers documentation. It scores how well a language model predicts the next token; lower perplexity means better predictions.

Is this the same as Perplexity the company?

No. Perplexity the measurement is a standard score for language models. Perplexity is also the name of an AI search company, which is unrelated to this metric.

What did OpenAI disclose about perplexity?

In 'Perplexity and Intelligence' on 22 Nov 2026, OpenAI chief scientist Jakub Pachocki gave approximate log-perplexity values on an internal dataset: about 0.99 for GPT-3.5, 0.91 for GPT-4 and 0.76 for the model that began producing new maths results.

What limit is OpenAI proposing?

Pachocki proposes common limits, based on perplexity and other measurements, that developers would commit not to cross until the ability to understand and monitor AI behavior reaches a concrete milestone, such as a successor to chain-of-thought monitors. The post names no number.

A note on sourcing. The definition of perplexity is from the Hugging Face Transformers documentation. The scaling findings are from the arXiv abstracts of Kaplan et al. (2020) and Hoffmann et al. (2022). OpenAI’s log-perplexity values, capabilities and proposal are from Jakub Pachocki’s post ‘Perplexity and Intelligence’, 22 November 2026. The conversion from log values to perplexity, assuming the natural logarithm, and the percentages are our arithmetic.

Sources: OpenAI, ‘Perplexity and Intelligence’, by Jakub Pachocki, Chief Scientist, 22 Nov 2026 · Hugging Face Transformers documentation, ‘Perplexity of fixed-length models’ · Kaplan et al., ‘Scaling Laws for Neural Language Models’, arXiv:2001.08361 (2020) · Hoffmann et al., ‘Training Compute-Optimal Large Language Models’, arXiv:2203.15556 (2022) · Our piece: OpenAI’s Loss Fell 0.99 to 0.76; It Proposes Common Limits (22 Nov 2026)

Scroll to Top
FourWeekMBA