A research paper posted to arXiv on 16 October reports that a looped Mixture-of-Experts model with 8.0B resident parameters “uses 75% fewer resident parameters than a 112-layer feedforward MoE baseline” under matched training and inference compute, and still scores higher on the GSM8K maths benchmark.
In the paper’s Table 5, the looped model scores 83.62 on GSM8K against 80.82 for the 32.0B-resident feedforward model. That is 24.0B fewer stored parameters, by our arithmetic. It is a research result at 8B scale, not a deployed product.
Business Pill · THE MEMORY WALL
A short explainer on the memory wall, why memory capacity and bandwidth limit computing. It teaches the general idea only and says nothing about this paper.
The key insight: The paper reports that a looped MoE with 8.0B resident parameters scored 83.62 on GSM8K against 80.82 for a 32.0B-resident feedforward MoE, with 75% fewer resident parameters; the feedforward model still scores higher on seven of ten benchmarks.
The paper is “Towards Looped Models Done Right, Part I: Topology, Input Injection, Recurrent-State Design”, by Benhao Huang, Chufan Shi, Junlin Chen, Shicheng Wen, Zhengzhong Liu, Eric Xing and Xuezhe Ma. The paper lists the Institute of Foundation Models, USC and CMU as affiliations.
Looped models reuse the same Transformer blocks several times instead of stacking new ones. The authors compare two designs: an Ouro-style loop that repeats the full stack, and a Huginn-style loop that repeats only middle layers between separate input and output layers.
The abstract says that “at the 730M-parameter and 336B-token scale, the complete Huginn architecture outperforms Ouro across all ten benchmarks”.
Benchmark scores from Table 5 of arXiv 2610.23036: a looped Huginn-style MoE with 8.0B resident parameters against a 112-layer feedforward MoE with 32.0B resident parameters, both trained on 500B tokens. Labels show looped minus feedforward, in points. Research models trained by the authors.
The MoE Test
For the larger test, both looped models have 8.0B resident parameters and 793.9M active parameters per pass, according to the paper, trained on 500B tokens. The feedforward reference has 112 layers and 32.0B resident parameters, matched on logical depth and active compute.
In Table 5, the Huginn-style model scores higher than Ouro on eight of ten benchmarks, with its largest gains on GSM8K and MATH500. Against the feedforward model, the abstract says it “outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks”.
The paper also says the feedforward MoE leads overall. In Table 5 it scores higher on seven of the ten benchmarks: ARC-C, HellaSwag, MMLU, BBH-CoT, TriviaQA and both coding tests.
The Business Lever: Stored Parameters Are Memory
Resident parameters are the weights a model has to keep in memory. If each one were stored in 2 bytes, a common 16-bit format, 8.0B parameters would take about 16 GB and 32.0B about 64 GB, by our arithmetic. The paper does not state a storage precision.
As we read the abstract, the authors credit the sandwich layout for gains in context extraction and reasoning, and input injection for gains in knowledge retrieval and context use.
As we read Table 5, the looped model’s edge over the larger feedforward model is narrow and concentrated in GSM8K and DROP.
As we read the abstract, the comparison holds training and inference compute fixed, so the saving the paper reports is in stored parameters, not in compute.
Huang et al., Towards Looped Models Done Right, Part I (arXiv 2610.23036)
“Despite its smaller physical parameters, it outperforms the feedforward baseline on DROP and GSM8K, matches it on MATH500, and remains competitive on the other benchmarks.”
Three Implications
FOR MODEL BUILDERS The paper reports that the Huginn-style design scored higher than Ouro on eight of ten benchmarks at the MoE scale it tested.
FOR MEMORY DEMAND The paper reports 8.0B resident parameters for the looped model against 32.0B for the feedforward reference, at matched compute.
FOR THE EVIDENCE The paper reports results from the authors’ own runs on 500B tokens and publishes code and checkpoints on GitHub.
The framework says “HBM growth is outpacing accelerator demand, making memory the rate-limiting factor in the AI scaling equation.”
As we read it, the framework treats memory as a constraint on AI scaling; the paper measures stored parameters, not memory hardware, and does not discuss HBM or memory prices. The framework predates the paper.
What Is Not Established
The results come from the authors’ own training runs, with looped models at 8.0B resident parameters against a 32.0B-resident feedforward reference; we did not reproduce them. The paper does not report serving cost, latency or memory use in production, and it does not say whether any lab plans to use the design. The authors publish code and checkpoints on GitHub.
This Week In The AI Supercycle
Execution Fault Lines, Concentrated Demand, and Geopolitical Firewalls
This week’s issue: the AI build is not short of demand for its parts. It is short of cheap money, and its end revenue rests on a narrow base of large, often venture-funded buyers.
A looped 8.0B model scored 83.62 on GSM8K against 80.82 for a 32.0B feedforward model, the paper reports, with 75% fewer resident parameters. The feedforward model still leads on most other benchmarks, and the work is a research result at 8B scale.
A note on sourcing. Figures, names and quotes come from the arXiv paper 2610.23036 (its first page, abstract, Section 4 and Table 5). Memory sizes are our arithmetic under a stated 2-byte assumption. The results are the authors’ own; we did not reproduce them or contact the authors.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
