A paper posted to arXiv on November 4 audits nine public benchmarks that grade whether AI systems apply written rules to tax, legal and business cases, and finds that correcting one scorer raised accepted answers on the 96 LegalBench SARA numeric cases from 8–9 to 87–91 for each of three models, according to the paper.
Averaged across the three models, accepted answers went from about 9% to about 92% of the 96 cases, by our arithmetic, with the model responses and the reference answers unchanged.
Business Pill · The Model and the Harness
A short Business Engineer explainer on the difference between a model and the system around it. It teaches the general idea only and does not discuss this paper.
The key insight: A November 4 audit of nine public benchmarks that grade whether AI systems apply written rules to tax, legal and business cases reports that correcting the LegalBench SARA scorer raised accepted answers from 8–9 to 87–91 of 96 for each of three models, with responses and references unchanged. Averaged, that is about 9% to 92%, by our arithmetic.
The paper, “Follow the Policy, Fail the Benchmark: Auditing and Revising Policy-Based Decision Benchmarks”, lists Yanru Xiao of Amazon Web Services and Yifei Cai of Iowa State University as authors.
The authors write that they “audit nine public policy-based decision benchmarks through source tracing, program execution, and controlled checks”. They report “unsupported targets, reference errors that survive replay or propagate as solved steps, and evaluators that reject acceptable answers or credit incomplete decisions”.
The benchmarks named in the paper are RuleArena, OpenExempt, LegalBench SARA, DeonticBench, ConditionalQA, ShARC, PolicyGuardBench, WorkArena++ and TheAgentCompany.
LegalBench SARA numeric cases accepted out of 96 for three models, with the original scorer and after the scorer fix, from Table 1 of Xiao and Cai, arXiv 2611.07042 (4 November 2026). Responses and reference answers are unchanged. The averages, about 9% to 92%, are our arithmetic.
What Was Posted on X
New Amazon paper finds that benchmarks used to grade AI models on tax, legal and business rules often mark correct answers wrong, because the tests’ own answer keys or grading code contain mistakes.
Benchmarks like RuleArena and LegalBench test whether models apply written rules to tax, legal, and business cases. If the reference answer or the scorer is wrong, the score measures the benchmark’s bugs, not the model.
They audited 9 public benchmarks by tracing references back to the source rules, rerunning reference programs, and testing scorers with controlled answers. Then they rescored the same saved model responses after each fix.
LegalBench’s tax scorer deleted decimal points, so $259,487.13 was read as 25,948,713. Fixing it raised accepted answers from 8 or 9 to between 87 and 91 out of 96 for each of 3 models.
Before trusting a low score on a rule-following benchmark, read the scorer code and check a few failed answers by hand.
A post on X by @rohanpaul_ai on 8 November summarises the paper and calls it an Amazon paper; the paper lists one author at Amazon Web Services and one at Iowa State University. The post’s figures match the paper: the LegalBench tax scorer deleted decimal points, and correcting it raised accepted answers from 8 or 9 to between 87 and 91 out of 96 for each of 3 models. Its closing advice to read the scorer code and check failed answers by hand is the poster’s own.
The Decimal Point
In one LegalBench SARA case the reference answer is $259,487 and a model response starts $259,487.13. The paper says the scorer “applies .replace(“.”, “”)” before extracting the number, so the decimal point is deleted. The authors keep the decimal point.
Keeping the decimal point alone lifts accepted answers to 87 of 96 for Haiku 5.5 (from 8), 83 for GPT6 Luna (from 9) and 84 for Gemma 4 31B (from 8), according to the paper’s Table 1. The paper says “Most of the gain comes from preserving decimal amounts during parsing”.
A second scorer correction, for exact zero amounts, “adds four accepted answers per model on the zero-reference cases”, taking the totals to 91, 87 and 88 of 96.
The Tax Floor
In RuleArena Tax, the paper says the released reference program omits a required zero floor on line 22 of the tax computation, so a response that applies the rule is marked wrong. Restoring the floor “changes 27 of 300 references”.
On the 300 tax cases the paper reports accepted answers rising from 209 to 224 for Haiku 5.5, 241 to 261 for GPT6 Luna and 195 to 220 for Gemma 4 31B. The paper describes the correction as narrowing “an observed model gap”: “The Haiku–Gemma gap falls from 14 to four accepted cases; model order is unchanged”.
RuleArena Tax cases accepted out of 300 for three models before and after restoring the line-22 zero floor, from Table 1 of Xiao and Cai, arXiv 2611.07042 (4 November 2026). The paper says the Haiku–Gemma gap falls from 14 to four accepted cases and model order is unchanged.
As we read the paper, the low original scores on these cases measured the scorer as much as the models: the same saved responses were accepted once the decimal point was kept.
As we read the tax result, the reference correction moves all three models up but keeps their order, which the paper states directly.
As we read the scope, the authors are careful to attribute no effect to published scores that reused the audited components, so the paper is evidence about the benchmarks rather than a ranking of models.
Yanru Xiao and Yifei Cai, arXiv 2611.07042
“In both cases, the reported score reflects not only model behavior but also defects in the benchmark.”
Three Implications
FOR BENCHMARK USERS The paper reports scorers that reject acceptable answers and references that misstate required decisions in public rule-following benchmarks.
FOR MODEL COMPARISONS On RuleArena Tax the paper says the Haiku–Gemma gap falls from 14 to four accepted cases after the correction, with model order unchanged.
FOR THE BENCHMARKS The authors present BAR (Benchmark Audit and Revision), a versioned collection of audit evidence and revisions for reevaluation.
The essay observes: “Model weights are increasingly fungible — Sonnet, GPT, Gemini, and Grok cluster within a few points on benchmarks that mattered last year.”
As we read it, when leading models sit within a few points of each other, a scorer defect that moves accepted answers by tens of points, as this paper reports, can matter more than the gap between models. The essay does not discuss this paper or these benchmarks.
What Is Not Established
The paper rescores three models on two of the nine benchmarks; it does not report rescored results for every benchmark or for other models. It does not say how many published leaderboard results used the affected scorer versions; the authors write that they “attribute no effect to those published scores”.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
A November 4 audit of nine rule-following AI benchmarks reports that correcting one LegalBench scorer raised accepted answers from 8–9 to 87–91 of 96 for each of three models, about 9% to 92% on average by our arithmetic, and that restoring one tax rule changed 27 of 300 RuleArena references.
A note on sourcing. Findings and figures are from Xiao and Cai, “Follow the Policy, Fail the Benchmark”, arXiv 2611.07042, posted 4 November 2026, read on arXiv; we did not contact the authors. The averages are our arithmetic from the paper’s Table 1.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
