Paper: SFT Hits 43.7 vs RL’s 43.8 With 9.5x More SFT Data

Supervised fine-tuning (SFT) matched reinforcement learning (RL) on general reasoning once it was trained on as many rollouts as RL uses, according to a paper from researchers at the National University of Singapore, Yale University and Cisco Foundation AI, posted on arXiv on 11 November. A Qwen2.5-7B model fine-tuned this way scored 43.7 against 43.8 for the same model trained with RL.

Our number: the SFT dataset that closed the gap held about 138,000 rollouts, 9.5x the 14,512 in the usual one-rollout-per-prompt setup, by our arithmetic. With one rollout per prompt, SFT scored 38.7, so the gap to RL fell from 5.1 points to 0.1.

Business Pill · Distillation Explained

An evergreen Business Engineer explainer on how a small model learns from a large one. It teaches the general idea only and does not discuss this paper.

The key insight: In this paper, the gap between fine-tuning and RL on general reasoning came down to the training set: SFT on one rollout per prompt trailed RL by 5.1 points, and SFT on RL’s full rollout budget trailed by 0.1, by our arithmetic.

What the Paper Tested

The authors trained Qwen2.5-7B on math problems from the DAPO dataset and then scored it on twelve general reasoning tasks outside math, including GPQA-Diamond and MMLU-Pro. The paper states that the RL run generated 8 rollouts per prompt, while SFT usually sees one correct rollout per prompt, so “RL trains on 8× more rollouts than SFT”.

They then grew the SFT set in three steps. Matching RL’s solvable prompts gave about 10,000 pairs. Matching its correct rollouts gave about 39,000. Matching its total rollouts, wrong answers included, gave about 138,000, a set the paper says “also matches the gradient computation cost of SFT to that of RL”.

General-reasoning average across twelve tasks for Qwen2.5-7B trained on math problems: base model 33.8; SFT wi
General-reasoning average across twelve tasks for Qwen2.5-7B trained on math problems: base model 33.8; SFT with one rollout per prompt (14,512 rollouts from the RL-trained model) 38.7; SFT on rollouts from the larger Qwen3-14B model at the full budget 36.7; SFT on RL-model rollouts at the full budget (about 138,000) 43.7; RL (GRPO) 43.8; SFT on a 90/10 mix at the full budget 44.4. Source: Hou et al., arXiv 2611.14912, Tables 5 and 6, 11 November 2026. Standard errors about 0.4 points, which the paper says do not capture variation across training seeds. The 9.5x is our arithmetic.

What It Found

On the full rollout budget, SFT on data generated by the RL-trained model scored 43.7, against 43.8 for RL. A mix of 90% of that data and 10% from the larger Qwen3-14B model scored 44.4. The paper reports standard errors of about 0.4 points on these averages.

About 98,000 of the 138,000 rollouts had wrong final answers, 71% by our arithmetic. The paper says such rollouts “still contain correct intermediate reasoning steps that can benefit SFT”.

The data source also mattered. SFT on rollouts from the larger Qwen3-14B model alone scored 36.7, 7.1 points below RL, by our arithmetic. The paper says “data from a model closer to the one being fine-tuned can be more effective than data from a stronger model”.

The authors link the gain to rare solution methods. Using Qwen3-32B to label the methods in each rollout, they identified 138, and say “larger datasets cover more tail methods”. In a separate test with compute held fixed, fewer unique rollouts trained for more epochs did worse, and the paper concludes that “the number of unique rollouts, matters more than compute”.

What the Posts Said

Pedro Domingos, whose X profile describes him as a professor of computer science at UW, shared the paper on X on Tuesday with the question “The last nail in the coffin of RL for LLMs?” He was quoting another X user’s post, which called it a “Great paper on why SFT can generalize just as well as RL if you give it enough diverse training data”.

The Business Lever

The lever is post-training data. The paper moves the comparison from the training algorithm to the size and mix of the training set. Its best SFT data still came from rejection sampling with the RL-trained model, so RL remained part of how that data was made.

The paper also says matching the full rollout budget aligns SFT’s gradient compute with RL’s. It does not report a cost saving. On Tuesday, OpenAI’s Mark Chen said OpenAI measures its AI research interns by the compute Codex runs end to end.

The Structural Read

As we read the paper, its result is about data volume and mix, not a cheaper method: the matched set also matched RL’s gradient compute, and its best data came from the RL-trained model.

As we read the figures, the 0.6-point edge of the best mix over RL sits within about one and a half standard errors, so the paper’s own numbers support a tie more firmly than a win.

Hou et al., arXiv 2611.14912, 11 November 2026

“both the SFT training data source and data scale are essential for unlocking the full potential of SFT”

Three Implications

FOR MODEL LABS The paper says matching RL’s full rollout budget, wrong answers included, let SFT generalize like RL on Qwen models of 7B and 8B parameters; it did not test larger models.

FOR POST-TRAINING DATA The paper found rollouts from a model closer to the one being trained beat rollouts from the larger Qwen3-14B, 43.7 to 36.7 at the full budget.

FOR RL PIPELINES The best SFT data in the paper came from rejection sampling with the RL-trained model, so RL was still part of how it was made.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Four Intelligence Moats.

The framework says: “Each moat is built by a different data pipeline.”

As we read it, the paper puts the same weight on the pipeline: which model generated the training rollouts, and how many, changed the result more than the choice between SFT and RL. The framework predates this paper and does not discuss it.

What Is Not Established

This is one paper, not a consensus. The results come from 7B- and 8B-parameter Qwen models trained only on math, and the authors say their standard errors “do not capture variation across training seeds due to practical computation constraints”. The 0.6-point edge of the best mix over RL is about one and a half standard errors.

Domingos’s “last nail” is his question, not the paper’s claim. The paper does not test frontier-scale models or say whether labs would change their post-training. We did not contact the authors.

This Week In The AI Supercycle

AI Lost Its Central Banker, the Lab Oligopoly, and the Future of Open Source

This week’s issue: the Nvidia leadership change, read as the AI build-out losing its central banker: who now finances the GPU landlords, backstops the data centres and sponsors open models, and why two labs are becoming the bank.

Read the full issue →

Business Engineer Framework

The Four Intelligence Moats

A Business Engineer analysis of how each AI data pipeline becomes a moat, and how each moat decays.

Read the Map of AI →

The Bottom Line

A NUS, Yale and Cisco paper found that fine-tuning matched RL on general reasoning, 43.7 to 43.8, once it trained on RL’s full rollout budget, 9.5x its usual data by our arithmetic. Its best data still came from an RL-trained model, and its error bars do not capture variation across training seeds.

95,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.

A note on sourcing. Figures and quotations are from Hou et al., “Rethinking ‘RL Generalizes, SFT Memorizes’: The Role of SFT Data”, arXiv 2611.14912 (11 November 2026), read in full from the PDF, and from two X posts. The 9.5x, 5.1-to-0.1-point, 71%, 7.1-point and one-and-a-half-standard-error figures are our arithmetic. The results come from 7B- and 8B-parameter models, and the paper says its standard errors do not capture variation across training seeds. We did not contact the authors.

Sources: Hou et al., Rethinking “RL Generalizes, SFT Memorizes”: The Role of SFT Data, arXiv 2611.14912, 11 Nov 2026 · @pmddomingos on X, 19:05 UTC, 17 Nov 2026 · @gurtej__gill_ on X, 16 Nov 2026

Scroll to Top
FourWeekMBA