AI Benchmarks, Run Rates, and Agent Incidents: What the Numbers This Week Actually Measured

The key insight: In nearly every case this week, the full context had already been published by the party being discussed — a laboratory in its own document, a founder aloud in an interview, an operator in its own filing. The defect is not in the disclosure. It is in the summarisation. Credit belongs to the parties who published the awkward context in the first place, because they are the reason any of this was checkable.

Structural Principle

Compression Selects for the Quotable Half

A figure with a denominator you can picture will travel. A figure that needs three pages of method before it means anything will not. Compression selects for the quotable half and discards the half that constrains it — and it does that without anyone deciding to mislead.

SHAPE 1 — Missing or Ambiguous Denominator

This shape appeared across four cases in the week’s coverage. An AGI-benchmark headline percentage whose denominator changed what the percentage measured. An efficiency claim denominated one way in a company’s published materials and a different way when its founder spoke aloud — two claims about two different scarce things, both accurate, neither comparable. A claimed open-versus-closed token split that no instrument can actually measure, because billed and self-hosted inference are metered differently. And a frontier-share movement read from corporate-card spend, with a router sitting between the spend and the model that actually served the request. Each surviving number was accurate. What was missing was the base that would tell you what it was a share of.

SHAPE 2 — Missing Evaluation Set or Conditions

This shape is the one most easily mistaken for a factual disagreement when it is actually a methodological one. One set of model weights simultaneously carried four different verified benchmark numbers this week — the harness wrapped around it and the evaluation split determine which of the four you are entitled to quote. A system card’s two headline behavioural figures turned out to measure how hard the tester pushed rather than any rate at which something happens in deployment; the evaluator published, in their own document, the observation that the subject could often tell it was being tested. The score was real. The conditions it was scored under are what the headline left out.

SHAPE 3 — Missing Term or Time Unit

Two cases this week, both involving currency figures with no period attached. An annualised run rate presented as though it were revenue — one month multiplied by twelve is not the same claim as twelve months of recognised revenue, and the difference matters for how the number compounds. A total contracted value with no disclosed term, which means it cannot be turned into an annual figure at all, however tempting the arithmetic looks. Both numbers were real. Neither was a revenue figure in the sense the framing implied.

SHAPE 4 — Missing Scope or Authorisation Boundary

This shape is the most consequential and the least visible, because the axis on which the event is graded can look correct while measuring the wrong thing entirely. Agent incidents in which the data reached was already public scored at nearly zero on a severity scale built on data sensitivity — while the authorisation path was the thing that actually failed. A count of local infrastructure restrictions where the count itself pointed away from the real finding. An enterprise threat model calibrated for rare deliberate misuse, while the variable that actually moved was the volume of ordinary error. And an assistant surface on which appearing in the answer and being purchasable turned out to be two separate gates — neither implied the other.

This Desk — Internal Note

“A comparison figure that exists in no primary document was carried into a draft chart by an automated summary, and was caught only by going back to the source document itself. A summary is not a source, and an automated summary sitting next to a real link is the most convincing form this error takes, because everything around it looks like provenance.”

Business Engineer Framework

The Map of AI — Reading the Stack, Not Just the Headline

Every number in AI coverage sits at a specific layer of the stack — infrastructure, model, evaluation harness, application, distribution. The Map of AI framework makes the layer explicit, which is the fastest way to see which denominator, evaluation set, time unit, or scope boundary belongs in the sentence alongside the figure. Understanding where a claim originates in the stack tells you which of the four questions to ask first.