Fable 5’s 10.8-Point Benchmark Lead Moved Merges 1.4 Points

When a US export control directive cut off Claude Fable 5 for 19 days in June, its users’ GitHub pull requests were merged about as often on the lower-scoring Opus 4.8, a new preprint from Harvard’s Yining Hua finds. Access to Fable 5 raised the 7-day merge rate by 1.4 percentage points, a change the paper says was not statistically significant.

Fable 5 led Opus 4.8 by 10.8 points on SWE-Bench Pro, per the two models’ system cards cited in the paper. The study asks whether that kind of gap in AI model benchmarks shows up in work that maintainers actually accept.

The paper is a preprint posted on arXiv on 17 November and has not been peer reviewed. It measures merges, not code quality or time saved.

Business Pill · The Cost of Checking

An evergreen Business Engineer explainer. It teaches the general idea only and does not discuss this study.

The key insight: Having access to Claude Fable 5, which led Opus 4.8 by 10.8 points on SWE-Bench Pro, raised its users’ 7-day pull-request merge rate by only 1.4 points (not statistically significant), per a Harvard preprint that used a 19-day suspension as a natural experiment; five 2026 model upgrades showed no significant rise either.

How the Study Worked

The directive ran from 12 June to 1 July 2026, the paper says. During that window no customer could use Fable 5, and most of its users switched to Opus 4.8. We covered the restoration on 1 July.

Hua tracked the pull requests of 297 Fable 5 users before, during and after the suspension. The control group was 8,908 developers who used Opus 4.8 throughout.

Models were identified from a co-author line naming the Claude model, which Claude Code can add to commits. Those model-named pull requests are about 1% of merged Claude Code pull requests, the paper says.

The paper’s period estimates for Fable 5 users’ 7-day merge rate stay close to zero before, during
The paper’s period estimates for Fable 5 users’ 7-day merge rate stay close to zero before, during and after the 19-day suspension; most 95% intervals include zero.

What the Numbers Show

Access to Fable 5 changed the 7-day merge rate by +1.4 points, with a 95% interval from −1.4 to +4.3, per the paper. Per 100 pull requests, that is about one or two more merged within a week, and the interval includes zero.

Hua repeated the test across five model upgrades in 2026, covering 1,208 developers and 15,442 pull requests. The pooled change in the merge rate was −0.9 points, with an interval from −1.9 to +0.2.

Merged pull requests per week changed by +2.4%, with an interval from −4.9% to +10.2%. Failed automated checks did not change after the upgrades either, at +0.3 points.

Bar chart of estimated changes: Fable 5 access +1.4 points on 7-day merge rate; five upgrades -0.9 points; weekly merged PRs +2.4%; failed checks +0.3 points; all intervals include zero. Data: Hua, arXiv 2026
None of the paper’s main estimates is statistically different from zero, per Hua’s Table 2.

Why the Ceiling Matters

Almost everything was already merged. In the week before each upgrade, developers who switched got 94.5% of their pull requests merged within seven days, and 77.1% within an hour, the paper reports.

On that base, a 1.4-point rise is about 1.5% in relative terms, by our arithmetic. The paper notes the merge rate had little room to rise, but says merging within an hour and the rate among developers with unmerged pull requests did not change either.

What the Author Flags

The suspension lasted only 19 days, so that analysis alone has wide intervals; the five upgrades give narrower ones, Hua writes. The upgrade analysis is observational, because developers chose when to switch.

The paper also notes that losing Fable 5 meant switching models, so speed and usage limits may also have changed; it estimates the overall effect of losing and regaining access.

It cannot tell whether a person or a script clicked merge, and it cannot see fixes made after a merge that do not go through a revert. The dataset and code are public, with developer names hashed.

The Structural Read

As we read it, the study tests acceptance, not effort saved: with 94.5% of pull requests already merged, merge rates leave little room for any model to show a gain.

As we read it, the result questions how much benchmark leads tell buyers about everyday coding work, rather than whether newer models are better.

Yining Hua, arXiv:2611.23216

“This also shows that benchmark gains of 5 to 18 points did not show up in the work developers actually kept.”

Three Implications

FOR ENGINEERING LEADERS The preprint found no significant change in merge rates with access to a model 10.8 points higher on SWE-Bench Pro.

FOR AI LABS Across five 2026 upgrades, the pooled merge-rate change was -0.9 points, per the paper.

FOR BUYERS OF CODING TOOLS The paper’s measure is acceptance of pull requests; it does not measure time saved or code quality.

What Is Not Established

The study does not measure how long developers spent on each pull request, the size or difficulty of the changes, or code quality beyond reverts and failed checks. It covers only pull requests that name their Claude model.

It has not been peer reviewed, and it covers Claude Code users only, so it does not show how other coding agents compare.

What This Means

The paper does not say Fable 5 is no better at hard tasks. It says a 10.8-point benchmark lead did not show up in how often these developers’ pull requests were merged.

This Week In The AI Supercycle

A World Without an Order, and an AI Central Bank Without a Successor

This week’s issue: the week AI memory stayed largely sold out while the people holding the shares paid: why the AI Supercycle is not a dot-com replay, read through geopolitics, macro and technology, with the gap between building and financing the widest we’ve recorded.

Read the full issue →

Business Engineer Framework

The AI Supercycle

The Business Engineer map of who builds, finances and captures value in AI.

Read the Map of AI →

The Bottom Line

In one natural experiment and five upgrades, higher-scoring Claude models did not raise how often developers’ pull requests were merged, per this preprint. What to watch is whether peer review or replications on other agents confirm it.

97,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.

A note on sourcing. All study figures, intervals, dates and sample sizes are from Hua’s arXiv preprint 2611.23216v1 (17 Nov 2026), which has not been peer reviewed; SWE-Bench Pro scores are as cited in the paper from the models’ system cards. The per-100 restatement and the 1.5% relative change (1.4 / 94.5) are our arithmetic. We first saw the paper via a post by @rohanpaul_ai.

Sources: Hua Y. Higher-Scoring Coding Models Did Not Get More Work Done. arXiv:2611.23216v1 [cs.SE], 17 Nov 2026 (Harvard University) · Replication dataset and code (hashed identifiers), linked from the paper · FourWeekMBA, US lifts export controls on Claude Fable 5 and Mythos 5, 1 Jul 2026

Scroll to Top
FourWeekMBA