Claude Fable 5.2 accepted a broken premise as valid in 8 of 100 nonsense questions at its max reasoning setting, according to BullshitBench, the benchmark built by Peter Gostev, which added the model on 11 November. Claude Opus 4.8, ranked first, did so once, so Fable 5.2’s rate is 8 times higher, by our arithmetic.
Fable 5.2 also refused only 2 of 100 questions, against 33 for Claude Fable 5 at its xhigh setting, a drop of 31 points, by our arithmetic. Gostev summed it up on X: “Good news is that refusals (i.e. null response) are WAY down”, but “results are quite a bit worse – ranked 40 & 53rd”.
Business Pill · Benchmarks Explained
An evergreen Business Engineer explainer on why a test score is not your own result. It teaches the general idea only and does not discuss BullshitBench.
The key insight: As we read the leaderboard, Fable 5.2 did not turn its old refusals into clear pushback: of the 31 fewer refusals than Fable 5, 10 became clear answers, 16 partial challenges and 5 accepted nonsense, by our arithmetic, so it answers more often but calls out a broken premise less cleanly.
The repository says BullshitBench “measures whether models detect nonsense, call it out clearly, and avoid confidently continuing with invalid assumptions”. Version 2 has 100 questions covering 13 nonsense techniques across software, finance, legal, medical and physics questions, and 250 model and reasoning variants.
A three-judge panel of Claude Sonnet 4.6, GPT-5.2 and Gemini 3.1 Pro Preview grades each answer as clear pushback, a partial challenge or accepted nonsense. The clear score divides clear answers by attempts minus refusals, so a refusal is left out of that score rather than counted as a miss.
The README adds that the test “does not measure how often models incorrectly reject valid questions”. The canonical leaderboard file orders models by average grade and places the two Fable 5.2 settings 38th and 39th; Gostev’s chart, sorted by clear pushback excluding refusals, shows them 40th and 53rd.
Outcomes of 100 nonsense questions on BullshitBench V2 for Claude Fable 5 (xhigh), Fable 5.1 (max) and Fable 5.2 (max): clear pushback, partial challenge, accepted nonsense and refusal, per the benchmark’s published leaderboard of 11 November 2026.
What Was Posted on X
Fable 5.2 on BullshitBench – bad news and good news. Good news is that refusals (i.e. null response) are WAY down. Fable 5 denied 1/3 of responses, now refusals are nearly gone.
But results are quite a bit worse – ranked 40 & 53rd. Fable/Opus were otherwise dominating Top 10.
Peter Gostev, who built BullshitBench, posted this at 23:12 UTC on 11 November with a chart of the leaderboard’s ranks 39 to 56. He calls the drop in Fable 5.2’s refusals the good news and its ranks of 40th and 53rd the bad news; the piece checks both against the benchmark’s own published leaderboard file.
Where the Refusals Went
At max reasoning, Fable 5.2 gave 66 clear pushbacks, 24 partial challenges and 8 accepted answers, with 2 refusals. Fable 5 at xhigh gave 56 clear, 8 partial and 3 accepted, with 33 refusals. Fable 5.1 at max gave 64 clear, 16 partial and 6 accepted, with 14 refusals.
Of the 31 fewer refusals between Fable 5 and Fable 5.2, 10 became clear pushbacks, 16 became partial challenges and 5 became accepted nonsense, by our arithmetic. That is why the newer model gives more clear answers in absolute terms, 66 against 56, yet a lower clear score: 67.3% against 83.6% once refusals are excluded.
At the low setting the pattern holds. Fable 5.2 low refused 2 times against 36 for Fable 5 low, and its clear score excluding refusals is 63.3% against 70.3%, with 30 partial challenges and 6 accepted answers.
Against the Top of the Board
Claude Opus 4.8 with no reasoning leads at 95 clear pushbacks, 1 accepted answer and no refusals. Nine of the top 10 variants on the canonical board are Anthropic models, with Qwen3.8 Max third, by our count; the board also lists Claude Fable 5.1 low eighth and Fable 5 xhigh 11th.
Fable 5.2 at max sits 27.7 points behind the leader on clear score excluding refusals, by our arithmetic. Claude Opus 5.5, the newest Opus on the board, scores 68.9% at max and 69.2% at low, with 9 to 10 refusals per 100, close to Fable 5.2’s 67.3%.
The release that added Fable 5.2 also added Gemini 4 Carbon, Grok 4.7, Kimi K3.1 Pro, GPT-6.1 Terra, Muse Ember and Tencent Hy4, each at two reasoning settings, bringing version 2 to 25,000 outcomes, according to the changelog. We covered the model’s launch in Anthropic’s Fable 5.2 release and its lead on Agent Arena.
The Structural Read
As we read the scoring rule, a refusal drops out of the clear score instead of counting as a miss, which is why Fable 5 could post 83.6% on 67 answered questions while Fable 5.2 posts 67.3% on 98, even though Fable 5.2 gave 10 more clear answers in absolute terms, by our arithmetic.
As we read the top of the board, the Anthropic models that rank highest are older ones with no refusals: Opus 4.8, Sonnet 4.6, Opus 4.5 and Opus 4.6 fill most of the top seven places, while Opus 5.5 and Fable 5.2 sit between 67.3% and 69.2% on clear score.
Peter Gostev, BullshitBench author, on X, 11 November 2026
“Fable 5 denied 1/3 of responses, now refusals are nearly gone.”
Three Implications
FOR ANTHROPIC Its newest Fable refuses 2 of 100 nonsense questions on this test, against 33 for Fable 5, while accepting 8, against 3 for Fable 5.
FOR BUYERS The README says the test does not measure how often models wrongly reject valid questions, so it shows one side of the refusal trade-off only.
FOR RIVAL LABS The same release added Gemini 4 Carbon, Grok 4.7, Kimi K3.1 Pro, GPT-6.1 Terra, Muse Ember and Tencent Hy4 at two settings each.
Answers that treated a broken premise as valid, of 100 attempts, for six Claude variants on BullshitBench V2, per the benchmark’s leaderboard of 11 November 2026. The 8x ratio is our arithmetic.
What Is Not Established
The benchmark does not say why Fable 5.2 refuses less, and we did not ask Anthropic for comment. A null response counts as a refusal here, and the test does not check whether a model wrongly rejects valid questions, so it cannot show whether fewer refusals also changed behaviour on ordinary prompts. The judges are themselves models, including one from Anthropic.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
On BullshitBench, Claude Fable 5.2 refused 2 of 100 nonsense questions against Fable 5’s 33, and accepted 8 against Opus 4.8’s 1. Most of the freed answers became partial challenges, by our arithmetic, which lowered its clear score to 67.3% and its places to 40th and 53rd on Gostev’s sort.
A note on sourcing. All scores come from BullshitBench’s public GitHub repository, read in full on 12 November: its README, its changelog entry of 11 November and its version 2 leaderboard file. The ranks 40th and 53rd and the quoted comments are from Peter Gostev’s post on X. Differences, ratios and counts are our arithmetic. We did not contact Anthropic or the benchmark’s author for this piece.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
