When a US export control directive cut off Claude Fable 5 for 19 days in June, its users’ GitHub pull requests were merged about as often on the lower-scoring Opus 4.8, a new preprint from Harvard’s Yining Hua finds. Access to Fable 5 raised the 7-day merge rate by 1.4 percentage points, a change the paper says was not statistically significant.
Fable 5 led Opus 4.8 by 10.8 points on SWE-Bench Pro, per the two models’ system cards cited in the paper. The study asks whether that kind of gap in AI model benchmarks shows up in work that maintainers actually accept.
The paper is a preprint posted on arXiv on 17 November and has not been peer reviewed. It measures merges, not code quality or time saved.
Business Pill · The Cost of Checking
An evergreen Business Engineer explainer. It teaches the general idea only and does not discuss this study.
The key insight: Having access to Claude Fable 5, which led Opus 4.8 by 10.8 points on SWE-Bench Pro, raised its users’ 7-day pull-request merge rate by only 1.4 points (not statistically significant), per a Harvard preprint that used a 19-day suspension as a natural experiment; five 2026 model upgrades showed no significant rise either.
How the Study Worked
The directive ran from 12 June to 1 July 2026, the paper says. During that window no customer could use Fable 5, and most of its users switched to Opus 4.8. We covered the restoration on 1 July.
Hua tracked the pull requests of 297 Fable 5 users before, during and after the suspension. The control group was 8,908 developers who used Opus 4.8 throughout.
Models were identified from a co-author line naming the Claude model, which Claude Code can add to commits. Those model-named pull requests are about 1% of merged Claude Code pull requests, the paper says.

What the Numbers Show
Access to Fable 5 changed the 7-day merge rate by +1.4 points, with a 95% interval from −1.4 to +4.3, per the paper. Per 100 pull requests, that is about one or two more merged within a week, and the interval includes zero.
Hua repeated the test across five model upgrades in 2026, covering 1,208 developers and 15,442 pull requests. The pooled change in the merge rate was −0.9 points, with an interval from −1.9 to +0.2.
Merged pull requests per week changed by +2.4%, with an interval from −4.9% to +10.2%. Failed automated checks did not change after the upgrades either, at +0.3 points.

Why the Ceiling Matters
Almost everything was already merged. In the week before each upgrade, developers who switched got 94.5% of their pull requests merged within seven days, and 77.1% within an hour, the paper reports.
On that base, a 1.4-point rise is about 1.5% in relative terms, by our arithmetic. The paper notes the merge rate had little room to rise, but says merging within an hour and the rate among developers with unmerged pull requests did not change either.
What the Author Flags
The suspension lasted only 19 days, so that analysis alone has wide intervals; the five upgrades give narrower ones, Hua writes. The upgrade analysis is observational, because developers chose when to switch.
The paper also notes that losing Fable 5 meant switching models, so speed and usage limits may also have changed; it estimates the overall effect of losing and regaining access.
It cannot tell whether a person or a script clicked merge, and it cannot see fixes made after a merge that do not go through a revert. The dataset and code are public, with developer names hashed.
The Structural Read
As we read it, the study tests acceptance, not effort saved: with 94.5% of pull requests already merged, merge rates leave little room for any model to show a gain.
As we read it, the result questions how much benchmark leads tell buyers about everyday coding work, rather than whether newer models are better.
Yining Hua, arXiv:2611.23216
“This also shows that benchmark gains of 5 to 18 points did not show up in the work developers actually kept.”
Three Implications
FOR ENGINEERING LEADERS The preprint found no significant change in merge rates with access to a model 10.8 points higher on SWE-Bench Pro.
FOR AI LABS Across five 2026 upgrades, the pooled merge-rate change was -0.9 points, per the paper.
FOR BUYERS OF CODING TOOLS The paper’s measure is acceptance of pull requests; it does not measure time saved or code quality.
What Is Not Established
The study does not measure how long developers spent on each pull request, the size or difficulty of the changes, or code quality beyond reverts and failed checks. It covers only pull requests that name their Claude model.
It has not been peer reviewed, and it covers Claude Code users only, so it does not show how other coding agents compare.
What This Means
The paper does not say Fable 5 is no better at hard tasks. It says a 10.8-point benchmark lead did not show up in how often these developers’ pull requests were merged.
The Bottom Line
In one natural experiment and five upgrades, higher-scoring Claude models did not raise how often developers’ pull requests were merged, per this preprint. What to watch is whether peer review or replications on other agents confirm it.
97,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.
A note on sourcing. All study figures, intervals, dates and sample sizes are from Hua’s arXiv preprint 2611.23216v1 (17 Nov 2026), which has not been peer reviewed; SWE-Bench Pro scores are as cited in the paper from the models’ system cards. The per-100 restatement and the 1.5% relative change (1.4 / 94.5) are our arithmetic. We first saw the paper via a post by @rohanpaul_ai.
Sources: Hua Y. Higher-Scoring Coding Models Did Not Get More Work Done. arXiv:2611.23216v1 [cs.SE], 17 Nov 2026 (Harvard University) · Replication dataset and code (hashed identifiers), linked from the paper · FourWeekMBA, US lifts export controls on Claude Fable 5 and Mythos 5, 1 Jul 2026









