1,000 Runs Find Fine-Grained MoE Is LLMs’ Biggest Win

Researcher Nikita Pavlichenko says he ran “more than 1,000 experiments” to test the architecture changes labs have made since GPT-3, and his write-up, posted on X on 30 October, finds that “the best architecture change since GPT-3 is fine-grained MoE”.

In his runs, DeepSeek-V3-style fine-grained MoE improved loss by 4.9%, 5.6% and 6.2% at the three trained widths. His fit leaves a 2026 hybrid with about 2% of predicted loss that architecture can still reduce, against about 11% for a 2021 GPT-3 reference: about 5.5x less room, by our arithmetic.

Short explainer · 1,000 Runs: MoE Gave 6.2%, Most Tricks Under 2%

A vertical news Short narrated from this piece’s headline and opening line.

Business Pill · THE MODEL AND THE HARNESS

A short evergreen explainer on the model and the harness around it. It is general background, not about this story.

What He Tested

Pavlichenko writes that he trained the models with bux, “the framework I built for architecture experiments”, on code files and Python repositories from Stack-Edu plus web text from FineWeb-Edu. The widths ran from d_model 512 to 1280, which he puts at 25M to 394M non-embedding parameters for the GPT-3 baseline, on B300 GPUs.

He measures quality in bits per byte on held-out code, repositories and web text, where lower is better. He also measures serving cost in vLLM, and uses a scaling-law fit to predict each change at equal serving cost after 1T tokens. He lists the limits himself: the scaling ladder is small, the data and benchmarks are toy, and he calls it “a fun evening project”.

Which Changes Held Up

Fine-grained MoE was the clearest gain, and he writes that, unlike many small changes, it got better with width. He also writes that “With MoE, an active parameter does about as much as 4-5 parameters in a dense model”.

Linear attention came next. Replacing three of four attention layers with KDA improved loss by 3.7%, 3.4% and 3.4%, and he writes that it let him serve 3.6x the parameters for the same cost. Gated attention helped by 2.1%, 1.4% and 1.5%; hyper-connections by 2.4%, 2.2% and 1.3%; and QK-norm by about 1% at every width.

Several familiar changes did less. GQA made loss about 1.1% worse at every width, though he writes that cutting the KV cache let him serve a 4-9x bigger model for the same money. Multi-token prediction “doesn’t measurably improve BPB”, and RoPE’s early gain over learned positions shrank as the models grew.

The Business Lever: What Training Budgets Buy

He predicts his final architecture is 9.4% better than GPT-3 at equal serving cost and 1T tokens, and writes that “Most of the improvement was already there in DeepSeek-V3, with fine-grained MoE and MLA”. On his fit, the 2025-26 changes look much bigger on small models than at serving size.

That bears on where lab budgets go. If architecture has about 2% of predicted loss left to reduce, as his fit says for the 2026 hybrid, more of the gain has to come from data, compute and post-training. It is one researcher’s small-scale study, and a separate paper this week reported a looped 8B model that scored higher than a 32B model on two of ten tests with far fewer stored parameters.

The Structural Read

As we read the article, most of the measured gains come from two families, sparse experts and linear attention, while norms, MLPs and positions move results by 1-2% at most by serving size.

As we read the article, several changes look better on his small models than his fit predicts at serving size; he writes that gains at the trained sizes are 1.5-3x larger than the predictions.

As we read the article, serving cost changes the ranking: GQA is worse per parameter in his runs but lets him serve a 4-9x bigger model for the same money.

Three Implications

FOR MODEL BUILDERS The author reports fine-grained MoE and linear attention as the largest single gains in his runs.

FOR SERVING The author reports that GQA, MQA, MLA and sliding windows mostly help by cutting the KV cache, letting him serve a 4-9x bigger model.

FOR THE EVIDENCE The author reports more than 1,000 runs at up to 394M non-embedding parameters for the GPT-3 baseline, and lists the limits of his setup himself.

The Business Engineer Lens

This story maps onto the Business Engineer framework The State of the Inference Economy.

The framework says “Training was a cost center: episodic, concentrated among roughly twenty frontier model labs, amortizable across trillions of tokens.”

As we read it, the framework treats serving as the growing cost line; the author’s study ranks every change at equal serving cost, which is the same lens applied to architecture. The framework predates the article and does not discuss it.

What Is Not Established

The results are the author’s own runs at up to 394M non-embedding parameters for the baseline; we did not reproduce them. The serving-size predictions extrapolate far beyond the trained sizes, and he writes that the code was “largely written by Claude Opus 5.5”. He does not say which lab, if any, uses his recipe.

The Bottom Line

Across more than 1,000 runs, Pavlichenko found fine-grained MoE the biggest architecture gain since GPT-3, with linear attention next, and most other changes worth 1-2% at best by serving size. His fit leaves about 2% of predicted loss for architecture to win on a 2026 model, against about 11% in 2021.