Coinbase: Newer AI Models Missed More Fraud on Its Benchmark

Coinbase engineers report that on the company’s own fraud benchmark of 16,140 transactions, the newer version in each of three frontier model families caught less fraud than the version it replaced: Opus 5 against Opus 4.5, Sonnet 5 against Sonnet 4.6, and GPT-5.6 (sol) against GPT-5.4. The results are in Part 2 of a three-part series on Coinbase’s engineering blog, posted on 7 October 2026; CryptoSlate reported on it on 10 October.

In all three pairs the newer version had lower recall, F1 and dollar-weighted recall. The post’s summary reads: “Newer LLMs are not necessarily better at domain-specific tasks.” In Part 3, Coinbase says a post-trained open-weight model, Qwen3.5-9B, beat Opus 4.5 on all four of its detection metrics.

Where the Fraud Agent Sits

The series is about Coinbase Onramp, which lets users buy crypto inside partner apps, including through guest checkout without a Coinbase account. Part 1 says Coinbase placed an LLM risk agent downstream of its existing machine-learning model and rules engine, for selected transactions.

Coinbase says the agent “can block a transaction that would otherwise be allowed, but cannot override an upstream block”, and that it returns only a risk level, such as RISK_LEVEL: 5, without an explanation. It used Claude Opus for the initial deployment; with an input prompt of roughly 7,000 tokens, Coinbase reports a median latency of about 1.5 seconds per call.

In an online experiment where the treatment group received 50% of traffic, Part 1 reports 30% fewer fraudulent transactions, a 33% lower fraud rate by transaction count and 22% less fraud value than the existing stack alone, with total transactions up 5%, users up 3% and revenue up 2%. Coinbase says projected savings from fraud prevention were roughly three times the estimated inference cost.

How Coinbase Built the Benchmark

Part 2 describes a fixed benchmark of 16,140 transactions across 7,293 users, drawn from nine weeks of production data before the agent was rolled out. The cohort contains 813 confirmed fraudulent transactions. Coinbase says it chose the pre-rollout period so that the agent’s own blocking decisions could not bias which fraud outcomes became observable.

Each model was tested as the decision agent alone, on the same transactions, with the same contextual-review procedure and a fixed risk-to-decision policy. The post says this asks “a practical upgrade question: how does a different model behave in the existing decision setup?”

Coinbase measures four things: precision, recall, F1, and dollar-weighted recall, the share of total fraudulent value the agent identified. It writes that a model “can miss relatively few transactions yet fail to detect a disproportionate share of the financial exposure.”

The Results, Family by Family

Opus 5 scored lower than Opus 4.5 on all four metrics, by 8.7 percentage points on precision, 0.8 on recall, 5.5 on F1 and 1.9 on dollar-weighted recall, according to Coinbase’s Figure 1.

Sonnet 5 scored lower than Sonnet 4.6 on all four: 11.0 points on precision, 22.2 on recall, 16.8 on F1 and 22.9 on dollar-weighted recall. Coinbase says this “was not simply a trade-off in which better precision compensated for lower recall.”

GPT-5.6 (sol) gained 11.5 points of precision over GPT-5.4, but recall fell 20.7 points, dollar-weighted recall 21.8 and F1 6.1. Coinbase says its GPT results are “consistent with more conservative classification.”

Why, According to Coinbase

Coinbase does not claim to know the cause. The post says: “Our benchmark shows a regression, not its cause.” It lists plausible explanations from published research: shifting decision boundaries and prompt behavior across model snapshots, trade-offs introduced by post-training, and changes in training-data mixtures.

Its advice is to review precision, recall and dollar-weighted recall together, to test a candidate under the existing decision setup before adapting prompts or thresholds, and to check latency, reliability and cost.

The Open-Weight Step

Part 3 says Coinbase post-trained Qwen3.5-9B on its proprietary Onramp fraud dataset using reinforcement learning with verifiable rewards, scoring the model’s risk classifications against known historical fraud outcomes. It used SkyRL on Anyscale with LoRA, and says the model was trained on one NVIDIA RTX PRO 6000 96GB GPU “for a total cost of under $100.”

Against the Opus 4.5 decision agent, Coinbase reports gains of 7.5 points on precision, 12.0 on recall, 9.6 on F1 and 35.4 on dollar-weighted recall. Its chart labels these “percentage points, not relative percent”. Against the unadapted Qwen3.5-9B, the gains were 24.2, 19.6, 23.0 and 45.1 points.

In online production serving, Coinbase reports a median end-to-end latency of 0.683 seconds against 1.515 seconds for Opus 4.5, and 1.036 against 3.312 seconds at the 99th percentile. The post notes that detection quality and latency were measured in separate evaluations. Part 3 lists Anyscale team members among its authors.

The Structural Read

Coinbase reports two kinds of change. Opus 5 and Sonnet 5 lost ground on every metric, while GPT-5.6 (sol) gained 11.5 points of precision and lost 20.7 points of recall, which Coinbase says is consistent with more conservative classification.

For Sonnet and GPT, dollar-weighted recall moved most. Coinbase says the newer Sonnet and GPT versions identified 22.9 and 21.8 points less of the fraudulent value, the measure it calls particularly important for payment risk.

As we read it, the same question runs through other recent tests of AI systems on narrow tasks, such as Sakana AI’s benchmark for LLM peer reviewers: the general score of a model says little about how it performs on one firm’s specific job.

Three Implications

TEAMS RUNNING AI IN PRODUCTION Coinbase says a candidate model should first be tested under the existing decision setup, with any new prompts or thresholds evaluated separately.

PAYMENT AND RISK TEAMS Coinbase measures dollar-weighted recall alongside precision, recall and F1, and reports the largest drops on that metric for Sonnet and GPT.

OPEN-WEIGHT MODEL USERS Coinbase reports that a post-trained Qwen3.5-9B, trained for under $100 on one GPU, beat Opus 4.5 on all four detection metrics on its benchmark.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Open vs Closed Meta-Framework.

The framework states the pattern as: “Close the SCARCE layer (keep proprietary) + Open the ABUNDANT layer (commoditize)”, and adds that “identifying the right layers is only half the battle.”

As we read it, Coinbase’s series is a worked case: in Part 3 the scarce inputs it describes as its own are the proprietary fraud dataset, the benchmark and the outcome-based rewards, while the model underneath is an open-weight Qwen3.5-9B.

What Is Not Established

The three posts report changes in percentage points; the underlying precision, recall and F1 scores for each model do not appear in the text or figures we read. The benchmark and dataset are Coinbase’s own and are described as proprietary, so the results cannot be independently reproduced from the posts.

Part 2 measures each model under one fixed prompt and decision policy, and Coinbase says a candidate that needs new prompts or thresholds should be evaluated separately. The comparison is a replay of historical transactions, which the post says is “not to measure the live business impact of deploying each one.” The posts do not include responses from Anthropic or OpenAI.

The Bottom Line

Coinbase reports that on its fraud benchmark the newer Opus, Sonnet and GPT versions all had lower recall, F1 and dollar-weighted recall than the versions before them, that it does not know why, and that a nine-billion-parameter open-weight model post-trained on its own data beat Opus 4.5 on all four metrics at lower latency. The company’s conclusion is to choose models “using high-quality, domain-specific benchmarks, with reproducible evaluation and ideally deterministic scoring against verified outcomes.”