Visa put 22 AI models from 12 labs through 15 shopping challenges, and they failed 42.8% of test runs, Visa said on 5 November as it launched the Visa Agentic Commerce Benchmark. In about half of the failed runs, 54%, the AI still made a purchase.
By our arithmetic, that is about 23% of all runs ending in a purchase after a failed test, roughly 229 of 990 runs.
Business Pill · Agentic Commerce
An evergreen Business Engineer explainer on agentic commerce. It teaches the general idea only and does not discuss Visa or its benchmark.
The key insight: Visa’s first benchmark round put 22 AI models from 12 labs through 15 shopping challenges: they failed 42.8% of runs, and 54% of the failed runs still ended in a purchase, about 23% of all runs by our arithmetic.
The benchmark asks a model to work out what a shopper needs through conversation, then buy or decline. Visa’s release says: “If nothing fits the customer’s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.”
The challenges included buying a TV, a laptop or a personal care appliance. In one scenario, the customer wants a new TV, mentions a relative’s recommended brand and a third-party membership, and leaves the budget and other requirements unsaid.
Visa calls it a first-of-its-kind benchmark, and its footnote says it is the first agentic commerce benchmark to require both discovery of every task-defined customer requirement and a correct purchase or no-purchase decision for a run to pass.
Olivia Moore, a partner at a16z by her X bio, shared figures from Visa’s report on 5 November: 22 agents tested across 15 shopping tasks, an average success rate of 57.2%, and two winners at 82%, Muse Wonder at $1.55 a run and Gemini 4 Argon at $0.54 a run. The Update box above checks those figures against Visa’s own results data.
AI agents failed 42.8% of Visa’s shopping tests
In more than half of those failures, the agent bought something anyway
Simon Taylor, who writes the Fintech Brain Food newsletter, posted on 6 November that in Visa’s tests Muse Wonder bought on 2 of its 8 failed runs, Gemini 4 Argon on 5 of 8 and Claude Fable 5.1 on all 11. The Update box above checks those counts against Visa’s published data.
The Results, as Visa Gives Them
Across testing, the models passed 57.2% of runs. The best performing models passed 82.2% of attempts and the worst 28.9%, about a 2.8-fold gap by our arithmetic.
The five models that uncovered the most hidden buying factors averaged a 70.2% pass rate; the five that uncovered the fewest passed 44.4% of the time.
The five models with the highest average cost per run passed 67.1% of runs, against 50.2% for the five least expensive, a 16.9-point gap by our arithmetic. Every model got three attempts at each scenario, and all three attempts passed just 34.2% of the time.
On the failed runs that still ended in a purchase, Visa says: “Either the AI recommended a purchase that did not meet the customer’s identified requirements, or it failed to uncover all the customer’s requirements.”
Update, 5 Nov, 23:36Z: which models won, and what each passed run cost. Visa’s published results data for the benchmark (its “October 2026 results snapshot”, 45 runs per model) ranks all 22 models. Meta’s Muse Wonder and Google’s Gemini 4 Argon tied at the top, each passing 37 of 45 runs (82.2%). Anthropic’s Claude Fable 5.1 came third at 75.6%. OpenAI’s best, GPT-6 Astra, passed 64.4%, level with Meta’s Muse Spark 1.3 and Claude Sonnet 5.5. The lowest score, 28.9%, went to Thinking Machines’ Inkling.
The two leaders got there at very different prices. Argon’s mean cost was $0.54 a run against $1.55 for Muse Wonder, and Visa’s cost-per-pass figure is $0.66 for Argon against $1.88 for Muse Wonder: by our arithmetic, Muse Wonder spent 2.9 times as much per successful purchase at the same pass rate. Argon was slower, at a mean 4.3 minutes a run against 2.1 minutes for Muse Wonder.
The ways models failed differ too. Visa counts a run as failed when the purchase missed a requirement, or when the model bought nothing although a fitting product was available. OpenAI’s four models ended 53 of their 180 runs in that second outcome, by our count of Visa’s data; Argon never did. Olivia Moore, a partner at Andreessen Horowitz, highlighted the two winners and their per-run costs on X.
Data: Visa Agentic Commerce Benchmark results (October 2026 snapshot). Cost per pass is Visa’s field; the 2.9x comparison is our arithmetic.
Why Visa Says It Matters
“AI agents have the potential to power the next era of commerce, but trust is key to making it work,” said Rubail Birwadker, Global Head of Growth at Visa.
The release cites Visa’s Trust Index, released in September 2026, which found that 72% of U.S. consumers had used an AI assistant, while only 23% trusted AI to handle payment transactions on their behalf.
Visa says it plans to add scenarios and product categories, and to extend testing to checkout, payment and order placement. It has shared the benchmark with leading model providers for feedback, and people can try the test themselves on Visa Payments Frontier.
Update, 6 Nov, 15:15 UTC: which models bought when they failed. Visa’s published results data, its October 2026 snapshot of 990 runs, also records how each failed run ended. In 229 runs, 23.1% of the total by our arithmetic, the model completed a purchase that missed at least one of the customer’s requirements.
The two leaders failed in different ways. Meta’s Muse Wonder failed 8 of its 45 runs and bought on 2 of them; Google’s Gemini 4 Argon also failed 8 and bought on 5. Anthropic’s Claude Fable 5.1 failed 11 runs, and all 11 ended in a purchase that missed a requirement, by our count of Visa’s data. Simon Taylor of Fintech Brain Food highlighted the same split on X on 6 November.
That matters for how shoppers react. In a YouGov survey of more than 2,000 UK adults for payments firm ACI Worldwide, published in June, 60% said they would stop using an AI agent after just one mistake.
Data: Visa Agentic Commerce Benchmark results (October 2026 snapshot); ACI Worldwide/YouGov. Counts and the 23.1% are our arithmetic from Visa’s data.
Connecting the Coverage
Our related pieces on AI agents that shop and act:
As we read Visa’s release, the test grades two things before any payment step: whether the model uncovers every requirement, and whether it buys or declines correctly.
As we read the cost figures, the five most expensive models passed 16.9 points more often than the five cheapest, by our arithmetic, but Visa notes that price did not guarantee success.
As we read the consistency figure, a model passed all three of its attempts at a scenario in only 34.2% of model-and-scenario pairs, across all models and scenarios.
Rubail Birwadker, Global Head of Growth at Visa, in the 5 November release
“AI agents have the potential to power the next era of commerce, but trust is key to making it work”
Three Implications
FOR MERCHANTS Visa says 54% of failed runs still ended in a purchase that missed or did not uncover the customer’s requirements.
FOR MODEL MAKERS Visa says it has shared the benchmark with leading model providers for feedback.
FOR PAYMENTS Visa says it will extend testing to checkout, payment and order placement.
The framework says: “The surface, not the model, governs autonomy.”
As we read it, Visa’s benchmark grades the agent’s behaviour in a shopping conversation, not a model’s raw score on a static test. The framework does not discuss Visa or shopping benchmarks.
What Is Not Established
The release does not name the 22 models or the 12 labs, or say which model scored 82.2% and which 28.9%. We have not read the whitepaper. This first round tested needs discovery and the buy-or-decline decision; checkout and payment were not tested yet, per Visa.
This Week In The AI Supercycle
Anthropic’s Reality Check, The Capital Squeeze, & The New World Order
This week’s issue: for two years the AI build-out was priced as a growthstory. This week it started being priced as a credit story.
In Visa’s first round, the models passed 57.2% of shopping runs, and 54% of the failed runs still ended in a purchase, about 23% of all runs by our arithmetic. Also on 5 November, two agent-shopping startups raised $30 million between them, as we reported, and on 4 November Uber counted four agent-booked rides in a week of about 320 million trips.
A note on sourcing. The benchmark figures, quotes and plans are from Visa’s own press release of 5 November 2026; the funding and Uber figures are from our earlier reports. Run counts, the share of runs that ended in a purchase after a failed test, and the gaps are our arithmetic.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
