OpenAI’s Loss Fell 0.99 to 0.76; It Proposes Common Limits

OpenAI published its internal pretraining scores on Sunday for the first time, in a post called “Perplexity and Intelligence” by chief scientist Jakub Pachocki. On its main held-out dataset, the log-perplexity of its models fell from about 0.99 for GPT-3.5 to about 0.76 for the model that started producing new maths results.

Perplexity measures how well a model predicts the next token on average; lower is better. By our arithmetic, the drop from 0.99 to 0.76 is about 23%, and the step after GPT-4 (0.15 points) is almost twice the step from GPT-3.5 to GPT-4 (0.08).

Pachocki proposes “common limits” on this measure that labs would commit not to cross until AI monitoring reaches a concrete milestone. Sam Altman called it “the most important thing OpenAI has published in awhile.” We track model scores on our AI model benchmarks page.

Update, 22 November 2026, 17:40 UTC: In a follow-up post on X, Sam Altman wrote: “Labs agreed under the White House Accords to establish standards and best practices to improve the safety of their systems, and I believe a discussion about the use of perplexity can further these goals.” In the OpenAI post, Chief Scientist Jakub Pachocki wrote: “We have recently witnessed unprecedented acceleration in another domain. For more than a year, we have been posing open problems in mathematics to our models as one of the ways of evaluating them. Once we upgraded to a model of log-perplexity ~0.76, it started producing groundbreaking results.” He added that “as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements,” and that he thinks progress on alignment and monitoring “is unlikely to be fast enough to allow the full potential acceleration in pretraining.” No other lab had responded on record by the time of this update.

Business Pill · The Model and the Harness

An evergreen Business Engineer explainer. It teaches the general idea only and does not discuss this case.

The key insight: OpenAI disclosed that its models’ approximate log-perplexity fell from about 0.99 (GPT-3.5) to about 0.76, about 23% lower by our arithmetic, and its chief scientist proposed industry-wide common limits on that measure until AI monitoring reaches a concrete milestone.

What OpenAI Disclosed

The post gives approximate log-perplexity values on the held-out dataset OpenAI has tracked since GPT-3.5, which it says is sourced from its internal codebase. GPT-3.5 sat at about 0.99 and GPT-4 at about 0.91.

Current models are below 0.8, Pachocki wrote, and the model that began producing what he calls groundbreaking results on open maths problems was at about 0.76.

He wrote that publishing the measurements gives competitors a clearer picture of where OpenAI stands, but that this concern “is outweighed by the urgency of establishing shared safety measures across the industry.”

OpenAI’s stated log-perplexity fell from about 0.99 for GPT-3.5 to about 0.76; by our arithmetic that is
OpenAI’s stated log-perplexity fell from about 0.99 for GPT-3.5 to about 0.76; by our arithmetic that is about 23% lower.

Perplexity, in One Line

Perplexity is how surprised a model is by the next word or token in text it has not seen. OpenAI tracks its logarithm, also called cross-entropy loss.

By our arithmetic, converting back from the log, perplexity fell from about 2.69 for GPT-3.5 to about 2.14 for the 0.76 model, about 20% lower.

Bar chart of drops in OpenAI's approximate log-perplexity: 0.08 from GPT-3.5 to GPT-4, 0.15 from GPT-4 to the 0.76 model, and 0.06 to 0.08 to the next window OpenAI flags. Data: OpenAI; arithmetic FourWeekMBA
By our arithmetic, the drop from GPT-4 to the 0.76 model (0.15 points) is almost twice the drop from GPT-3.5 to GPT-4 (0.08).

The Proposed Limit

Pachocki proposes “common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone.” His example of a milestone is a successor to chain-of-thought monitors.

He wants “an industry standard that specifies a first limit on development, measured by some key metric we can track across labs,” and a framework for setting successive limits. The post does not name a number for that first limit or say who would set it.

He also writes that frontier developers should jointly set shared norms for evaluating such models and red-teaming their safety cases, including shared transparency standards.

Why Now

Pachocki expects further “transformative capabilities” as models enter a window of about 0.68 to 0.7. By our arithmetic, that is 0.06 to 0.08 points, or about 8% to 11%, below the 0.76 model.

He writes that AI taking part in pretraining research itself could cause a step change, and that “as soon as 2027 we could conceivably see a similar avalanche of fundamental pretraining advancements.”

OpenAI’s chart shows a steep acceleration in 2026, which the post attributes mostly to long-term research and compute investments and only in smaller part to automation. We covered OpenAI rating GPT-6.2 Astra ‘High’ for AI self-improvement here.

The Limits of the Metric

The post lists four problems. Labs use different evaluation datasets and training data, so numbers are hard to compare across labs.

Perplexity does not map to a task people care about. It does not count gains from chain-of-thought reasoning. And the floor is unknown: as models approach it, small perplexity gains may mean larger leaps in capability, Pachocki wrote.

The Structural Read

As we read it, publishing the numbers puts a price on transparency: rivals now know roughly where OpenAI stands, which Pachocki says he judged worth it.

As we read it, the proposal only works if other labs publish comparable scores, and the post itself names cross-lab comparison as the first problem.

Jakub Pachocki, OpenAI Chief Scientist, 22 Nov 2026

“I further propose common limits, based on perplexity and other measurements, that developers are committed to not crossing until our ability to understand and monitor AI behavior achieves a certain concrete milestone.”

Three Implications

FOR RIVAL LABS Pachocki asks frontier developers to jointly set shared evaluation norms and transparency standards, per OpenAI’s post.

FOR REGULATORS The post proposes a first limit measured by a metric that can be tracked across labs, with a concrete monitoring milestone before going beyond it.

FOR OPENAI Pachocki wrote that publishing the numbers gives competitors a clearer picture of where OpenAI stands.

What Is Not Established

The numbers are OpenAI’s own, on an internal dataset no one else can check, and the post gives approximate values rather than a full table. Other labs have not published comparable figures.

By publication, we had not seen on-record responses from Anthropic, Google or other developers to the proposal. The UK AI Security Institute’s recent testing of GPT-6.2 Astra is here.

This Week In The AI Supercycle

A World Without an Order, and an AI Central Bank Without a Successor

This week’s issue: the week AI memory stayed largely sold out while the people holding the shares paid: why the AI Supercycle is not a dot-com replay, read through geopolitics, macro and technology, with the gap between building and financing the widest we’ve recorded.

Read the full issue →

Business Engineer Framework

The AI Supercycle

The Business Engineer map of who builds, finances and captures value in AI.

Read the Map of AI →

The Bottom Line

OpenAI put a number on its own progress, 0.99 to 0.76, and asked rivals to agree a line none would cross until monitoring catches up. What to watch is whether any other lab publishes its own pretraining scores.

97,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.

A note on sourcing. All log-perplexity values, quotes and the proposal are from OpenAI’s post “Perplexity and Intelligence” by Jakub Pachocki, 22 November 2026, which reports approximate values on an internal held-out dataset. Sam Altman’s quote is from his post on X the same day. Differences, percentages and the conversion from log-perplexity to perplexity are our arithmetic.

Sources: OpenAI (Jakub Pachocki, Chief Scientist), ‘Perplexity and Intelligence’, 22 Nov 2026 · Sam Altman on X, 22 Nov 2026, quoting Jakub Pachocki’s post on X

Scroll to Top
FourWeekMBA