The same Anthropicmodel scored 60% on the API and 94% on claude.ai in one of the company’s multi-turn suicide and self-harm safety evaluations, a gap of 34 percentage points, Anthropic disclosed on 13 November in a research post on building safety evaluations. The model was Claude Fable 5.1; the claude.ai figure came after Anthropic added targeted instructions to its system prompt.
The post sets out recommendations for designing, running and reporting safety evals, “based on our own experiences”. It also says Anthropic’s single-turn safety evals are saturated, with harmless response rates between 94% and 98% for every recent model, and that confidence intervals on its multi-turn API evals can exceed ±10 percentage points.
Business Pill · Benchmarks Explained
An evergreen Business Engineer explainer on why a test score is not your own result, the same point Anthropic makes about eval configuration. It teaches the concept only and does not discuss Anthropic’s post.
The key insight: As we read the post, the most useful figure for a buyer is the configuration gap: the same model, Claude Fable 5.1, scored 60% on the API and 94% on claude.ai in one self-harm eval, so a safety number means little without the setup it was measured in.
Anthropic writes that safety evals inform “whether it’s ready to deploy, which safeguards ship alongside it, and what our next round of safety training and research should focus on”. It adds: “a safety claim is only as sound as the eval behind it, and good ones are hard to build”.
The post is organised in four stages: safety first, design, run and report. It closes with three priorities: “write down the construct you want to measure before collecting data, validate your grader against human judgment before trusting any number it produces, and report uncertainty so readers can tell signal from noise”.
Anthropic draws a parallel with psychology, where large-scale replication projects found that many published findings did not hold up, and the field responded with reforms such as preregistration. The post recommends preregistering metrics, comparisons and thresholds before seeing results.
Appropriate response rates on multi-turn safety evaluations that Anthropic published on 13 November 2026: Claude Fable 5.1 scored 60% on the API and 94% on claude.ai with a new system prompt in a suicide and self-harm eval; on a romance-scam eval on claude.ai, Claude Opus 5 scored 98% and Opus 5.5 94%, a difference Anthropic says was not statistically significant.
The Numbers in the Post
On configuration, the post says that in testing Claude Fable 5.1 “the difference in the appropriate response rate between these two configurations on our multi-turn safety evals was as high as 34 percentage points”, comparing the raw model on the API with claude.ai, where the system prompt is applied.
The self-harm example explains the gap. Reviewing transcripts before Fable 5.1’s release, Anthropic says, it saw that Claude “sometimes implicitly validated self-harm as a coping strategy”. It added a system-prompt instruction and other targeted mitigations; the post reports “60% on the API vs 94% on claude.ai with the new system prompt”.
On noise, Anthropic measured a multi-turn romance-scam eval on claude.ai where Claude Opus 5 scored 98% and Opus 5.5 scored 94%. The post says “the difference between the two models wasn’t statistically significant, so we couldn’t conclude that one performed better than the other”.
On scale, the post says Anthropic’s single-turn safety suites run across 16 policy areas in seven languages and across multiple product surfaces. It treats 100 items as the floor for a complex multi-turn eval and 1,000 for a simpler single-turn one.
Spreads Anthropic reported in Building strong safety evaluations, 13 November 2026, in percentage points. The 4-point differences (98 minus 94) are our arithmetic.
How Anthropic Builds and Grades the Tests
Multi-turn scenarios start from a domain expert’s specification and a fixed first user turn; a separate Claude model plays the user, and each scenario runs several times. “One bad turn” fails the whole conversation, the post says. A grader model then scores each transcript against a rubric for the risk area.
The post’s advice on graders includes making criteria atomic and binary, failing loudly when a grader breaks, using a different model as grader from the one tested, and validating the grader against human labels. It warns that graders “favor longer responses” and can recognise and favour their own outputs.
For safety, the post says evals should run in a sandbox with no internet access and be monitored in real time. It points to the UK AI Security Institute’s open-source Inspect framework and Anthropic’s Petri toolkit, built on Inspect, as existing tooling.
Why Buyers Read Eval Reports
Anthropic says its system cards report safety results for both the API and claude.ai, and attach confidence intervals to scores. The post’s own figures show why the configuration line matters: a developer calling the raw model and a consumer using claude.ai can see different safety behaviour from the same model.
As we read it, the saturation figure matters as much: single-turn harmless response rates of 94% to 98% for every recent model leave little room to tell models apart, which is why Anthropic says those suites serve mainly as regression checks.
As we read the romance-scam example, a four-point difference between two Opus models was not statistically significant; a reader comparing system cards would need the confidence intervals that Anthropic says its system cards attach.
Anthropic, Building strong safety evaluations, 13 November 2026
“But a safety claim is only as sound as the eval behind it, and good ones are hard to build.”
Three Implications
FOR ENTERPRISE BUYERS Developers on the API use the raw model, the post says, while claude.ai applies a system prompt; the 34-point gap it reports is a reason to ask which configuration a safety score describes.
FOR EVAL BUILDERS The post sets floors of 100 items for a complex multi-turn eval and 1,000 for a simpler single-turn one, and asks for preregistered metrics and thresholds.
FOR READERS OF SYSTEM CARDS Anthropic says confidence intervals on its multi-turn API evals can exceed plus or minus 10 percentage points, so small gaps between models may be sampling noise.
The framework sets out “why the safety design is the most important business decision in the release”.
As we read it, the eval guidance is the measurement side of that decision: the numbers a buyer sees depend on how the test was built and run. The framework does not discuss this post.
What Is Not Established
The post gives the 34-point gap for one model and one family of evals; it does not say how large the API and claude.ai gap is for other models or risk areas. It does not give the confidence interval for the 60% and 94% figures, and it does not name the other targeted mitigations beyond the one quoted instruction.
The recommendations describe Anthropic’s own practice as the company reports it; we did not see the underlying eval data, and no outside party has checked these figures in what we read. We did not contact Anthropic for this piece.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
Anthropic’s guidance says a safety claim is only as sound as the eval behind it, and its own figures show how much the setup moves a score: 60% on the API against 94% on claude.ai for Claude Fable 5.1, and a 98% to 94% gap between two Opus models that was not statistically significant. The post asks eval builders to report the configuration and the uncertainty with every number.
A note on sourcing. We read Anthropic’s research post Building strong safety evaluations, published on 13 November 2026, in full on anthropic.com. All figures are the company’s own, as published; we did not see the underlying eval data. Unite.AI reported the post earlier; we worked from the primary. We did not contact Anthropic for this piece.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
