Dwarkesh Study: Better Data Cut Pretraining Compute 12x

Dwarkesh Patel and Jerry Han trained year-by-year combinations of open model recipes and open data corpora from 2019 to 2025 and found that better data alone reached the 2019 stack’s benchmark score with 12.00 times less compute, against 3.72 times for the newer model recipe alone, in a study published on Patel’s site on September 8. Patel resurfaced it on X on November 11, writing that it “got totally buried cause it came out the same day as Navier Stokes”.

Both upgrades together measured 15.06 times, not the roughly 44.6 times that multiplying the two would give, the post’s own figure notes. By our arithmetic, that means the six years of model-recipe work added only about 1.25 times on top of the 2025 data, while the 2025 data added about 4 times on top of the 2025 recipe.

Business Pill · Data Flywheel Explained

An evergreen Business Engineer explainer on the data flywheel. It teaches the general idea only and does not discuss the Patel and Han study.

What the Study Measured

The authors write that for each year from 2019 to 2025, “a new open model recipe was published which codified that year’s publicly known algorithmic tweaks”, and “there was also a new public data corpus”. The 2019 pair was GPT-2 on OpenWebText; the 2025 pair was OLMo-2 on UltraFineWeb.

They trained the combinations from scratch at compute budgets of 1e17 to 1e19 FLOPs and scored them on the OLMES eval, which the post says “aggregates 10 different relatively easy benchmarks, mostly multiple choice QA”. A compute multiplier is the ratio of the compute the reference needs to reach a score to the compute the candidate needs to reach the same score.

What Was Posted on X

Dwarkesh Patel posted on 11 November at 03:33 UTC that the study ‘got totally buried’ because it came out the same day as Navier Stokes, calling it super interesting evidence about the true major source of algorithmic progress. The piece reads the underlying write-up.

Patel’s original post of 8 September, saying he and co-author Jerry Han pretrained year-representative recipes and corpora and that data improvements contributed 12.0x against 3.7x for model improvements. The piece checks those figures against the write-up and its figure caption, which puts the measured joint gain at 15.06x.

Data 12x, Models 3.7x

The headline finding, in the post’s words: “3.24x more compute efficiency gains have come from data improvements rather than model improvements (12.0x for data and 3.7x for models), at the 1e19 FLOPs compute budget”. The figure caption gives the error bands: 12.00 times [9.30, 14.85] for data and 3.72 times [2.81, 4.60] for the recipe, as 68% bootstrap intervals.

Year on year, the authors report compute efficiency gains of “1.24x [1.19, 1.29] on the model side and 1.51x [1.45, 1.57] on the data side”, and 1.57 times measured jointly. They note that is “much lower than Anson Ho et al.’s mean estimate of 3x YoY”, and list reasons, including that many gains “might be scale dependent”.

Why the Gains Do Not Simply Multiply

Patel’s September post on X said the gains “stack independently – a better dataset helps every architecture about equally, and vice versa”. The write-up states it more narrowly: a linear model explains 88% of the variance in benchmark scores as additive model and data effects, at 3.16e18 FLOPs.

In compute terms, the figure caption is explicit: “If the two upgrades were independent, both together would give” about 44.6 times, and “the measured x15.06 means they overlap”. By our arithmetic, the multiplied figure is about three times the measured one.

What It Means for Data Spend

By our arithmetic, a 12 times compute multiplier means reaching the same score with about 8% of the compute, about 92% less, in these small runs. That is the economic stake the authors name at the outset: “The answer has big implications for the economics of frontier labs and the pace of future progress.”

One of their proposed next steps is to “figure out the implied value of data through lab spending on data brokers, environment producers, etc., relative to their spending on compute and researchers”. Data suppliers are already raising at scale: we reported on Nvidia joining Mercor’s $750 million round at a $20 billion valuation and on the training-data companies at NeurIPS.

The authors also caution that the model work mattered in a different way: its main contribution “was making larger amounts of compute usable in the first place”, through innovations such as “MoEs, sparse attention variants, stability innovations” and FlashAttention.

The Structural Read

As we read it, the study shifts where a lab’s edge would come from at this scale: from architecture tweaks that every open recipe publishes to the data pipeline that selects what goes into training. The authors stress that their runs are small and that model work mattered mostly by making larger compute usable.

Three Implications

FOR DATA SUPPLIERS The authors propose measuring the implied value of data through lab spending on data brokers and environment producers relative to compute and researchers.

FOR COMPUTE BUDGETS By our arithmetic, a 12 times multiplier means reaching the same score with about 8% of the compute in these runs.

FOR THE FRONTIER The authors write that the data improvements they tested might matter less for larger models, and the runs stop at 1e19 FLOPs.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Four Intelligence Moats.

The framework reads: “Each moat is built by a different data pipeline.” It also says: “The transformer is a commodity now”.

As we read it, a study in which the 2025 data did about three times the work of the 2025 model recipe fits that thesis at small scale. The framework predates the study and does not discuss it.

What Is Not Established

The runs top out at 1e19 FLOPs, and the authors call the experiment “done at an extremely small scale”. They write that “The data improvements we investigated here might matter less for larger models”, and that the corpora studied are mostly curations of the same Common Crawl, “consumption of a finite stock”.

The study did not test synthetic data, which the authors say is “obviously being widely used at the labs”, and it covers pretraining only; they note that “so many of the gains over the last two years have come from RL”. The results depend on one benchmark suite and on the recipe and corpus picked for each year.

The Bottom Line

In Dwarkesh Patel and Jerry Han’s small-scale runs, 2025 data alone cut the compute needed to match the 2019 stack’s score 12.00 times, the 2025 model recipe alone 3.72 times, and both together 15.06 times. By our arithmetic, the newer recipe added about 1.25 times on top of the newer data.