Giving a looped transformer small loop-specific adapters raised its parameter count by only 1.1% to 4.3%, against 51.4% for doubling the model’s depth, according to a paper posted to arXiv on 29 October by researchers at FAIR at Meta and MIT. The comparison is for 300M-parameter models, the paper says.
That is roughly 12 to 47 times fewer added parameters than doubling depth, by our arithmetic (51.4 divided by 4.3 and by 1.1). The paper adds that the doubled-depth model still reached lower bits per byte at comparable training FLOPs.
Business Pill · The Memory Wall
A short explainer on memory as a constraint in AI systems. It teaches the general idea only and says nothing about Meta or this paper.
The key insight: At 300M parameters, loop-specific adapters added 1.1% to 4.3% to a looped model’s parameters against 51.4% for doubling its depth, roughly 12 to 47 times fewer added parameters by our arithmetic, but the paper says the doubled-depth model still reached lower bits per byte at comparable training FLOPs.
The paper, titled How to Loop Your Transformer: Loop Scaling, Specialization, and Parallel Inference, studies looping, which the authors define as reusing weights “by repeatedly applying the same transformer, or parts of it, making the model deeper without increasing its parameter count.”
The eight authors list affiliations at FAIR at Meta and MIT. The experiments span models with up to 1B parameters, trained on up to 155B tokens, using up to 1.07 × 1021 training FLOPs and up to 32 loops, according to the abstract.
The headline finding is a trade-off. “We find that looping the full transformer generally gives the strongest quality at a fixed parameter budget, whereas looping a middle block of layers gives a better return when training FLOPs are constrained,” the authors write.
More loops do not always help. “Adding loops is not uniformly beneficial: gains eventually saturate, reverse, or become unstable,” the abstract says. The authors respond by adding small loop-specific low-rank adapters, which they say improve quality, with larger gains as more loops are added.
What Was Posted on X
New Meta and MIT paper finds reusing a transformer’s layers in loops can beat a plain model.
Looping runs the same layers several times, giving a model more depth without more parameters. But each loop costs compute and normally runs in sequence for every token.
They trained dense models up to 1B parameters on up to 155B tokens, testing where to loop and how many times. They also added tiny per-loop adapters and repaired a parallel looping method so loops can run at once during decoding.
Looping the whole stack gave the best quality at a fixed size, while looping middle layers paid off best when training compute was tight. Per-loop adapters added only 1.1 to 4.3% more parameters, versus 51.4% for doubling depth, and helped more as loops increased.
Rohan Paul summarised the Meta and MIT paper on X on 2 November: looping runs the same layers several times for more depth without more parameters, but each loop costs compute. His figures (models up to 1B parameters on up to 155B tokens; per-loop adapters adding 1.1 to 4.3% more parameters versus 51.4% for doubling depth) match the paper, which this piece cites directly.
Increase in parameters for a 300M looped model: loop-specific adapters add 1.1% to 4.3%, doubling the model’s depth adds 51.4%, plain looping adds none. Figures from section 3 of How to Loop Your Transformer (arXiv 2610.40325, FAIR at Meta and MIT). The paper says the doubled-depth model still reached lower bits per byte at comparable training FLOPs.
Where Looping Still Costs
The adapters save parameters, not compute. The paper calls loop-specific specialization “a parameter-efficient, rather than compute-efficient, way to improve quality.”
Decoding is the second cost. The abstract says looped transformers suffer “a decoding slowdown proportional to the number of loops.” Parallel Loop Transformers pass hidden states between loops across adjacent token positions so loop stages can run concurrently, but the authors write that in their experiments the original formulation “does not consistently retain the quality gains of ordinary looping.”
Their fix keeps the current-token embedding and normalises it and the previous-token hidden state separately before combining them. The paper says this “recovers strong performance across model scales and training budgets.” It also says the authors did not measure wall-clock decoding latency or throughput.
Why It Matters for AI Costs
The paper’s own framing is that looping “is a multidimensional design problem that must jointly account for stored parameters, training FLOPs, data efficiency, and inference efficiency.”
As we read the results, the paper points to different designs for different limits: looping the full transformer for a fixed parameter budget, looping a middle block when training FLOPs are tight, and loop-specific adapters as a way to add quality with few new stored weights rather than with less compute. We tracked the cost side of AI usage in Musk said Grok Bot cost too much as AI plans move to meters.
The Structural Read
As we read the paper, looping trades stored weights for compute: the model reuses weights it already has, and the paper says every extra loop still has to be run, which slows decoding in proportion to the number of loops.
As we read the results together, the choice depends on the constraint, as the authors say: looping the full transformer did best at a fixed parameter budget, and looping a middle block did best when training FLOPs were constrained.
Ekbote, Pagnoni, Sachan, Bhosale, Lewis, Fazel-Zarandi, Rajput and Gonugondla, in How to Loop Your Transformer (arXiv 2610.40325)
“Together, these results show that effective looping requires matching the design to the resource constraint and inference setting.”
Three Implications
FOR TEAMS SHORT OF MEMORY The paper reports that loop-specific adapters raised parameters by 1.1% to 4.3% at 300M, against 51.4% for doubling the model’s depth.
FOR TEAMS SHORT OF COMPUTE The paper calls loop-specific specialization “a parameter-efficient, rather than compute-efficient, way to improve quality.”
FOR ANYONE READING THE HEADLINE NUMBER The models tested go up to 1B parameters and 155B training tokens, and the paper says the authors did not measure wall-clock decoding latency or throughput.
The framework says: “While attention remains fixed on GPU shortages and frontier model capabilities, the binding constraint on AI scaling has quietly shifted to High Bandwidth Memory (HBM).”
As we read it, looping is a model-design response on the parameter side of that constraint: the paper reports depth gains with few new stored weights, while each loop still costs compute. The framework is the owner’s argument and does not discuss this paper.
The models tested go up to 1B parameters, far below frontier scale, and the paper does not say whether the results hold for larger models. It reports no wall-clock latency figures. We found no statement from Meta on whether it uses looping in its production models. We did not run the experiments.
This Week In The AI Supercycle
Anthropic’s Reality Check, The Capital Squeeze, & The New World Order
This week’s issue: for two years the AI build-out was priced as a growthstory. This week it started being priced as a credit story.
The Meta and MIT paper reports that looping a transformer can add depth for a small share of the parameters, 1.1% to 4.3% with adapters against 51.4% for doubling depth at 300M, while the doubled-depth model still did better at comparable training compute.
A note on sourcing. This piece relies on the paper How to Loop Your Transformer (arXiv 2610.40325, posted 29 October 2026), read in full on 2 November, and an X post by Rohan Paul. All figures are the authors’ own measurements. The 12-to-47-times comparison is our arithmetic from the paper’s parameter figures. We did not run the experiments.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
