A Google research paper posted to arXiv on 10 November scores Claude Opus 5.5 at 88.8% execution accuracy on a new benchmark for enterprise data agents, ahead of DeepSeek V4.1 Flash at 77.7% and Gemma 4 26B-A4B at 47.1%. All three ran as agents with files and a command line. The paper, by eight Google authors led by Jiani Zhang and Alon Halevy, describes SynthCorp, a platform that generates the test data by simulating a company’s business on a virtual calendar.
The first release has seven data estates and 256 tasks across five domains. By our arithmetic from the paper’s Table 7, the gap between Opus and Gemma widens from 21.3 percentage points when each model sees only the database schema to 41.7 points when each runs as an agent with files and a command line.
Business Pill · AI Agents Explained
An evergreen Business Engineer explainer on what AI agents are and how they act with tools. It teaches the general idea only and does not discuss the SynthCorp paper.
The key insight:Google’s SynthCorp paper scores Claude Opus 5.5 at 88.8% on 256 generated data-agent tasks, and by our arithmetic its lead over Gemma 4 26B-A4B grows from 21.3 points with the schema only to 41.7 points when both act as agents.
The authors write that current benchmarks for data agents “saturate quickly and cannot target specific capabilities”. The paper says existing ones give agents fixed databases and static task sets that models saturate quickly, cannot be customised to test a specific skill, and start from a static snapshot that “does not record how or why the data reached that state”. In a real company, the paper argues, a hard question often turns on that history, such as which table replaced a deprecated one in a migration.
SynthCorp works in four steps, according to the paper. It turns a scenario into reusable business event types, schedules routine and disruption events on a virtual calendar, executes the calendar so that relational tables, a document store and a data catalog stay consistent, and records every operation in a simulation log. Tasks and their gold answers are then derived from that log, without manual annotation.
The largest estate, Retail-1000, holds 1.4K tables, 1.5M rows and 1.4K documents built from 834 event instances over a 28-day horizon, according to the paper’s Table 4. The authors generated six estates with Gemini 4 Argon and Retail-1000 with Gemini 3.8 Flash.
Execution accuracy on all 256 SynthCorp tasks for three models, with the database schema only and as an agent with files and a command line, from Table 7 of Google’s SynthCorp paper (arXiv 2611.14765, mean of three runs). The 21.3-point and 41.7-point gaps between Claude Opus 5.5 and Gemma 4 26B-A4B are our arithmetic.
What Was Posted on X
Great paper from Google on building benchmarks for data agents.
Static data-agent benchmarks saturate quickly and can’t target a specific skill.
SynthCorp generates the data instead. It simulates routine business events and disruptions on a virtual calendar, and those events fill relational tables, a document store, and a data catalog that stay consistent with each other.
Every operation goes into a simulation log, so the gold answer for each task is derived from the log and checked without manual annotation.
The first release has seven data estates with up to 1,000 tables and 256 tasks across five domains. Tasks that chain several business events together still make state-of-the-art agents fail.
DAIR.AI, an AI research account, summarised Google’s SynthCorp paper on 13 November. We used the post as a lead; every number in this piece comes from the paper itself.
How Opus, DeepSeek and Gemma Scored
The paper tests three reasoning-enabled models in four settings, three runs per task. With the schema only, Opus scores 54.6%, DeepSeek 41.9% and Gemma 33.3%. Adding the top retrieved document lifts them to 71.2%, 56.2% and 42.3%. As agents with search tools, they reach 86.6%, 76.6% and 54.6%; with files and a command line, 88.8%, 77.7% and 47.1%.
The paper says one retrieved document adds at most 1.5K input tokens per run yet raises accuracy by 9 to 17 points, the cheapest gain across settings. Per correct answer with search tools, it reports Opus needs 102K input tokens, against 266K for Gemma and 360K for DeepSeek.
The authors audited the failures. Format issues cause 56% of Opus’s failed runs and 61% of DeepSeek’s in the files-and-command-line setting, the paper says; counting those as correct raises accuracy to 94% for Opus and 90% for DeepSeek. We covered Mercor’s test that put Fable 5.2 ahead of OpenAI.
Where Every Agent Still Failed
The hardest tasks chain several business events. In one Retail-1000 task, a 26 August event overwrote loyalty tiers in place; Opus counted members by the current tier in all 12 runs and missed the history table, and DeepSeek also failed all 12 runs, according to the paper. On the 11 multi-rule composite tasks of Retail-1000, accuracy with search tools falls from 91% for Opus to 76% for DeepSeek and 33% for Gemma.
The paper names three failure patterns: missing a second rule in another document, querying a partial or stale table instead of the authoritative one, and applying incomplete date or status filters. On 13 November, an AI research account on X summarised the paper; we used the post as a lead and read the paper itself. We also covered an HKU and Salesforce paper that trains the agent and its harness together.
The Structural Read
As we read it, the paper’s numbers say most of the work in enterprise data questions sits outside the database schema. One retrieved document adds 9 to 17 points for at most 1.5K input tokens, the paper reports, and the remaining failures it audits are missed rules, stale tables and incomplete filters, not SQL syntax.
Zhang et al., SynthCorp, Google, arXiv, 10 November 2026
“Current benchmarks for LLM-driven data agents saturate quickly and cannot target specific capabilities.”
Three Implications
FOR TEAMS BUILDING DATA AGENTS The paper reports that one retrieved document raises accuracy by 9 to 17 points, the cheapest gain across its settings.
FOR MODEL BUYERS Per correct answer with search tools, the paper reports Opus needs 102K input tokens, against 266K for Gemma and 360K for DeepSeek.
FOR BENCHMARK MAKERS The paper says format issues cause 56% of Opus’s failed runs in the files-and-command-line setting.
The framework opens: “The next professional jump is not a better prompt.”
As we read it, SynthCorp’s Table 7 puts numbers on that line for data work: the same model gains 34.2 points, by our arithmetic, when it moves from a schema-only prompt to an agent with files and a command line. The framework predates the paper and does not discuss it.
Accuracy with search tools on the 11 multi-rule composite tasks of SynthCorp’s Retail-1000 estate, from Section 5.4 of Google’s paper (arXiv 2611.14765).
What Is Not Established
The paper is a first arXiv version, and we found no peer review in what we read. It tests three models only, and the scores come from Google’s generated estates, not a company’s real data; the authors say every simulation has a boundary. The version we read does not link a public code release for SynthCorp. We did not rerun any test.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
Google’s SynthCorp paper replaces fixed benchmark databases with simulated business histories, and on its first 256 tasks Claude Opus 5.5 leads at 88.8% as an agent with files and a command line. The gap to the smallest model nearly doubles, from 21.3 to 41.7 points by our arithmetic, once models can search and act. On the 11 multi-rule composite tasks of its largest estate, Gemma scores 33% with search tools, against 91% for Opus.
A note on sourcing. We read Google’s SynthCorp paper on arXiv (version 1, submitted 10 November 2026) in full; every score here comes from its tables and text. We first saw the paper in a post on X by DAIR.AI, which we used only as a lead. The 21.3-point and 41.7-point gaps are our arithmetic from the paper’s Table 7. We did not contact Google or the authors and did not rerun any test.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
