Microsoft and Hugging Face published a joint post on 3 October 2026 introducing ThinkingBox, a benchmark that grades AI agents on the database state they leave behind. The post says Claude Opus 5.5 and Claude Opus 5 each pass 241 of 507 tasks on all 20 attempts, at estimated costs of $7.80 and $13.30 per dependable task.
A one-minute explainer of the idea behind this story: consistency. It teaches the concept, not this story’s figures.
The key insight: The joint Microsoft and Hugging Face post says breadth and consistency pull apart: how often an agent solves a task at least once is not how often it solves it on every attempt. It gives Claude Opus 5 at 79.09% of tasks solved at least once but 47.53% completed on every attempt, and Kimi-K3 at 93.89% against 13.41%. This publication reports the post’s figures and does not judge the benchmark.
The joint Microsoft and Hugging Face post says ThinkingBox grades agents on terminal backend state and side effects across 507 stateful business workflows, each run 20 times. It says 477 of the 507 tasks are graded on state alone and 30 add response rubrics.
Its opening example is a retail support case. The post says an agent made nine well-formed tool calls, read the refund policy correctly, and still closed a ticket as solved, when with the carrier exception still open the required end state was on hold. The post says the executable check that fails is a single field.
From the post’s Table 3: the estimated cost of the full 20-run campaign divided by the tasks a model passed on all 20 attempts. Claude Opus 5.5 and Claude Opus 5 pass the same 241 tasks, at $7.80 and $13.30. The figures are the authors’ estimates at list rates.
The post says that across 121,680 valid trials with 12 models, 79,853 attempts failed the executable checks. This publication’s own arithmetic: that is about 65.6% of those trials.
Of those failures, the post says 67.24% still terminated cleanly, invoked a state-changing tool and reported no final tool error. It says the checks found wrong field values in 77.61% of them, unintended extra effects in 43.30% and missing required effects in 25.36%, and that these findings overlap.
The post also assigns each failed trace one diagnostic signature: tool usage 79.9%, wrong state updates 10.3%, incomplete user resolutions 7.0% and no state-changing action 2.9%. It describes these as unweighted averages of per-model shares and not unique causal explanations.
One Attempt Versus Twenty
The post says Claude Opus 5.5 leads overall at 67.16% pass@1, ahead of Claude Opus 5 at 66.50%. It then asks how much of each score survives 20 repeats. It says GPT-6 Astra retains 78% of its single-attempt rate, Claude Opus 5.5 and Claude Opus 5 each retain 71%, and GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%.
The post says breadth and consistency pull apart. Kimi-K3 solves 476 of 507 tasks at least once, 93.89%, with only 31 defeating it entirely, but just 68 tasks, 13.41%, succeed in all 20 attempts.
Claude Opus 5, the post says, solves 79.09% at least once, with 106 tasks defeating it entirely, yet completes 47.53% of the benchmark on every attempt. The post says Claude Opus 5.5 solves more tasks at least once and passes exactly the same number on all 20 attempts as Claude Opus 5: 241.
What a Dependable Task Costs
The post prices each model’s recorded token usage at undiscounted list rates available on OpenRouter+, excluding endpoints that declare quantization. It calls the result a comparative efficiency index, not an invoice. Cost per dependable task is the estimated cost of the full 20-run campaign divided by the tasks passing all 20 attempts.
By that measure the post’s Table 3 puts GPT-5.4 lowest at $6.80, with 128 tasks passing 20 of 20. GPT-6 Astra follows at $7.45 with 231 tasks and Claude Opus 5.5 at $7.80 with 241. GPT-5.6 Sol is at $9.76 with 82 tasks. Claude Opus 5 is at $13.30 with 241.
The post puts Opus 5.5’s 20-run cost at $1,880.77 and Opus 5’s at $3,206.00. This publication’s own arithmetic: for the same 241 tasks that is $5.50 more per dependable task and $1,325.23 more per campaign for Claude Opus 5. The post says Opus 5.5 therefore dominates Opus 5 on that measure.
The post draws a second contrast. GPT-5.6 Sol has the lowest cost per single success at $0.127, and costs $9.76 per dependable task. On the same single-success measure the post gives Claude Opus 5.5 $0.276 and Claude Opus 5 $0.475.
The Structural Read
The post separates three measures. The first is the single-attempt score, which it reports as pass@1: it says Claude Opus 5.5 leads overall at 67.16% against 66.50% for Claude Opus 5. The second is how much of that score survives 20 repeats: the post says GPT-6 Astra retains 78% of its single-attempt rate, Claude Opus 5.5 and Claude Opus 5 each retain 71%, and GLM-5.1, Kimi-K2.6 and DeepSeek-V4-Pro each keep about 8%. The third is the number of tasks passed on all 20 attempts, where the post puts Claude Opus 5.5 and Claude Opus 5 at the same 241 of 507.
The post prices that third measure. It divides the estimated cost of the full 20-run campaign by the tasks passed on all 20 attempts, calls the result a comparative efficiency index rather than an invoice, and puts Claude Opus 5.5 at $7.80 and Claude Opus 5 at $13.30 per dependable task. The costs are the authors’ estimates at list rates.
The post also describes how the agents fail the checks. It says 67.24% of the failed trials still terminated cleanly, invoked a state-changing tool and reported no final tool error, and that the checks found wrong field values in 77.61% of the failures. It describes its diagnostic signatures as unweighted averages of per-model shares and not unique causal explanations.
“The Agent Said It Was Done. The Database Disagreed.”
Three Implications
ONCE IS NOT EVERY TIME The post gives Claude Opus 5 at 79.09% of tasks solved at least once and 47.53% completed on every attempt, and Kimi-K3 at 93.89% and 13.41%. This publication reports the figures and does not judge the benchmark.
A TIE ON TASKS IS NOT A TIE ON COST The post says Claude Opus 5.5 and Claude Opus 5 pass the same 241 tasks on all 20 attempts, at $7.80 and $13.30 per dependable task, and that Opus 5.5 therefore dominates Opus 5 on that measure. The costs are the authors’ estimates at list rates.
THE CHECK IS THE END STATE The post says ThinkingBox grades agents on the database state they leave behind, and gives an example in which an agent closed a ticket as solved when the required end state was on hold. This publication did not run the benchmark.
What Is Not Established
The post is a joint piece by Microsoft and Hugging Face about a benchmark presented in the authors’ own paper. This publication did not run the benchmark, did not read the underlying paper or dataset, and did not seek a response from Microsoft, Hugging Face or any model maker.
The costs are the authors’ estimates at list rates, not what any customer paid. The post also says it has not measured the lift from its suggested steps, such as classifying tool errors and requiring human approval for changes that are costly to reverse.
Everything above comes from the joint post, with the arithmetic marked as this publication’s own.
Business Engineer Framework
The Map of AI — The Stack Beneath the Applications
The Map of AI places more than 200 companies across nine layers of the stack, from silicon to application. It shows the layers that sit beneath the applications people use every day.
Microsoft and Hugging Face’s post says Claude Opus 5.5 and Claude Opus 5 each pass 241 of 507 tasks on all 20 attempts, at estimated costs of $7.80 and $13.30 per dependable task. The costs are the authors’ estimates at list rates, and this publication verified none of the figures.
91,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.
A note on sourcing. We drew on the joint Microsoft and Hugging Face blog post “The Agent Said It Was Done. The Database Disagreed.” dated 3 October 2026. Most figures come directly from that post, whose authors treat costs as estimates at list rates; the percentages and dollar totals marked as ours are our own arithmetic derived from those figures. We did not run the benchmark, we have not checked any of the post’s claims independently, and we did not contact Microsoft, Hugging Face or any model maker. Nothing here is a forecast or investment advice.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
Scroll to Top
Discover more from FourWeekMBA
Subscribe now to keep reading and get access to the full archive.
