A Salesforce AI Research paper found that training one reinforcement-learning expert per domain and then distilling them into a single model matched a jointly trained model on math and code with 42.0% fewer total training FLOPs (arXiv 2611.08453, posted November 5).
The same paper found that distilling from a large external teacher model generally lost to plain reinforcement learning on accuracy per training FLOP. On code, the researchers report that RL reached 46.4% at 0.6 EFLOP, while distillation from a 32B teacher peaked at 28.7% after 68.7 EFLOP.
Business Pill · Distillation Explained
An evergreen Business Engineer explainer on how a small model learns from a large one. It teaches the general idea only and does not discuss this study.
The key insight: A Salesforce AI Research paper posted on November 5 found that distilling per-domain RL experts into one model matched joint RL training on math and code with 42.0% fewer total FLOPs, while distillation from large external teachers generally lost to plain RL per training FLOP.
The paper, Unpacking the Compute Efficiency of Strong-to-Weak OPD against GRPO, is by Zeyu Leo Liu, Yang Li, Bo Pang, Emre Can Acikgoz, Xuan-Phi Nguyen, Anurag Koul, Srijan Bansal, Erik Nijkamp, Shafiq Joty and Semih Yavuz of Salesforce AI Research. Acikgoz is also listed with UIUC.
It compares two ways of post-training a model. On-policy distillation (OPD) trains a student on its own outputs, with a frozen teacher scoring every token. Group Relative Policy Optimization (GRPO) is a reinforcement-learning method that rewards correct final answers. The paper notes that OPD “adds one frozen-teacher forward pass over every token it trains on; GRPO adds nothing”.
The researchers tracked held-out accuracy against cumulative training FLOPs. Students were Qwen3 models of 1.7B, 4B and 8B parameters, and teachers included Qwen3 thinking models of 4B, 8B, 14B and 32B. Math was scored on AIME24, AIME25 and AMC23; code on LiveCodeBench v6.
Training compute at the LiveCodeBench v6 accuracy the paper reports for each method, Qwen3-4B-Base student. Data: Liu et al., Salesforce AI Research, arXiv 2611.08453. The 114x ratio (68.7 / 0.6 EFLOP) is our arithmetic.
What Was Posted on X
New Salesforce paper finds that distilling from a big external teacher usually costs more compute than plain RL, while merging RL-trained domain experts saves 42%.
On-policy distillation lets a student learn from a teacher’s per-token feedback on its own outputs, and many expect this dense signal to beat RL. But every training token also needs a teacher forward pass.
They compared distillation and GRPO by held-out accuracy per training FLOP, varying task difficulty, model sizes, domains, and training settings.
With an external post-trained teacher, GRPO generally won, and distillation often inflated response length. Distilling per-domain GRPO experts into 1 student matched joint GRPO on math plus code with 42.0% fewer total FLOPs.
To save compute, train per-domain experts with RL and distill them into 1 model, rather than distilling from a big external teacher.
A post by Rohan Paul at 00:17 UTC on 8 November summarising the Salesforce paper: it says distilling from a big external teacher usually cost more compute than plain RL, and that distilling per-domain RL experts into one student matched joint RL on math plus code with 42.0% fewer total FLOPs. Its closing advice, to train per-domain experts with RL and distill them rather than distil from a big external teacher, is the poster’s summary.
Big External Teachers Cost More
With an external post-trained teacher, the authors write, “OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation”.
On LiveCodeBench, the paper reports GRPO reaching 46.4% at 0.6 EFLOP, distillation from an external 8B teacher peaking at 36.7% at 45.7 EFLOP, and distillation from an external 32B teacher peaking at 28.7% at 68.7 EFLOP. By our arithmetic, the 32B-teacher run used about 114 times the compute of the RL run and scored 17.7 points lower.
Where Distillation Paid Off
The result reversed when the teacher was the student’s own RL checkpoint. On math, the paper says OPD recovered that checkpoint’s accuracy “with 50× fewer distillation FLOPs than GRPO needs to reach the same accuracy”, or about 6 times fewer at the default setting of eight rollouts per prompt.
The authors add that on a single domain the RL checkpoint “is already the desired model, so distilling it adds to total compute”. The saving comes from combining domains. On a math and code mix, two RL experts cost 10.7 EFLOP and distilling them into one student added 0.55 EFLOP, about 5.2%. On four domains, four experts cost 14.4 EFLOP and distillation added 1.15 EFLOP, about 8.0%.
Counting both stages, the paper reports that the merged student matched joint GRPO training with 42.0% fewer FLOPs on two domains and 31.6% fewer on four. Distilling from a single external 8B teacher on four domains collapsed on most of them, the paper says, scoring 0% on math and code.
The Business Lever
The paper’s introduction notes that dense teacher feedback suggests distillation “may learn from fewer rollouts than outcome-reward RL”, but that every training token needs a teacher forward pass. Measured in FLOPs, the cheaper route in its tests was RL per domain followed by distillation into one model, rather than distillation from a bigger outside model. The authors write that recent open-source models use this per-domain-expert recipe.
As we read the paper, the teacher’s provenance mattered more than its size: the student’s own RL checkpoint made distillation efficient, while larger external teachers did not.
As we read the multi-domain results, distillation’s role in this paper is consolidation: the saving comes from merging separately trained experts, and on a single domain the paper says distilling adds to total compute.
As we read it, the comparison is in training FLOPs on Qwen3 students up to 8B parameters plus one Olmo-3 model, so it shows relative compute efficiency in these settings rather than dollar costs at frontier scale.
Liu et al., arXiv 2611.08453
“With an external post-trained teacher, OPD generally loses to GRPO in accuracy per training FLOP at matched settings, even when the teacher is further trained with GRPO, and often suffers response-length inflation.”
Three Implications
FOR TEAMS POST-TRAINING MODELS The paper reports that on code, GRPO reached 46.4% at 0.6 EFLOP while distillation from an external 32B teacher peaked at 28.7% at 68.7 EFLOP.
FOR MULTI-DOMAIN TRAINING The paper reports that distilling two RL experts added 0.55 EFLOP to their 10.7 EFLOP of training, about 5.2%, and four experts added 1.15 EFLOP to 14.4 EFLOP, about 8.0%.
FOR RESEARCHERS The paper says recent efficiency reports use GPU-hours, gradient-step ratios or partial FLOPs comparisons, and that it compares the two methods at matched training FLOPs.
The essay lists four scaling paradigms, “pretraining, post-training, inference-time, agentic-loop”, and says the framework “tells you which scaling regime a model company is betting on and where the next capability gain comes from.”
As we read it, this paper is about the cost side of the post-training regime: how much compute it takes to turn a base model into a capable one, and which recipe used less in these tests. The essay does not discuss this study.
What Is Not Established
The experiments use Qwen3 students up to 8B parameters, plus one Olmo-3 model, and specific math and code datasets. The paper is a preprint. It does not report dollar costs, and FLOP counts do not capture every cost of a production training run. This piece makes no forecast.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
A Business Engineer essay on the shape of the AI build-out, including its four scaling paradigms: pretraining, post-training, inference-time and agentic loops.
A Salesforce AI Research paper found that distilling RL-trained domain experts into one model matched joint RL training on math and code with 42.0% fewer total FLOPs, while distilling from a large external teacher generally lost to plain RL per FLOP, with a 32B teacher peaking at 28.7% on code after 68.7 EFLOP against RL’s 46.4% at 0.6 EFLOP.
A note on sourcing. Figures and quotations are from the paper by Liu et al. of Salesforce AI Research, posted to arXiv on 5 November 2026 (2611.08453). We did not rerun the experiments. The compute ratio and point gap are our arithmetic from the paper’s reported figures.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
