AWS Emulator, the local copy of AWS that agents build, is graded by fidelity tests. AWS defines one as “a short program that uses an AWS SDK to run a workflow against an AWS service and assert how AWS behaves”, and AWS’s Matthew Goldberg calls the tests “one of our most fundamental guardrails in our system”.
The method also produces a public score. By our join of the 11 November score file to the test suite’s own labels, the emulator passed 68.0% of control-plane tests, 54.2% of data-plane tests and 0 of 156 tests that speak a database or broker protocol.
Business Pill · The AI Audit Problem
An evergreen Business Engineer explainer on why checking AI work can cost more than making it. It teaches the general idea only and does not discuss AWS Emulator.
The key insight: As we read it, a fidelity suite moves the hard part of agent-written software from writing code to defining the reference it must match: AWS uses the real cloud, Anthropic used a known-good compiler, and both publish pass rates against that reference.
AWS’s fidelity guide says the emulator passes a test “when its behavior matches AWS for every call in the workflow, including the status code, the resources that it creates, and other side effects”. If AWS returns an error, the emulator passes only if it returns the same error.
Its example is a test that tries to create a DynamoDB table with a name that is too short and asserts that the call is rejected with the error code and message AWS returns. AWS builds and verifies its tests in the US West (Oregon) Region, and a fidelity score is the number of tests a service passes out of its total, “such as 40 of 50”.
The tests live in a separate evaluation-suite repository, which says “AI agents author, manage, and maintain it”. Each test is a short Python program using boto3, with a runner that returns passed, failed or error.
Share of AWS Emulator behavioral fidelity tests passed in release v1.0.322 (scored 11 November 2026), grouped by the plane and data-plane labels in the aws-emulator-evaluation-suite manifest. We matched 29,255 of 32,402 behavioral tests to a label; a data-plane test can carry more than one data-plane label. Pass rates are our arithmetic.
The Reference Is the Real Thing
The method needs something to compare against. Goldberg wrote in an About Amazon post that “the AWS API Models are our spec and the actual AWScloud is our reference implementation”, and that agents “verify each one against real AWS before submitting”.
Goldberg credits the idea to an Anthropic project in which agents built a C compiler. That Anthropic engineering post of 5 February 2026 used existing compiler test suites and, for the Linux kernel, “GCC as an online known-good compiler oracle to compare against”. It reports a 99% pass rate on most compiler test suites, including the GCC torture test suite.
How the Agents Use the Tests
AWS describes three teams of agents: “One team manages a test suite; one team finds issues in the emulator; and one team fixes those issues”. The finders replay the tests and file issues for failures, and each fix must pass a gate that includes agents that “ensure that the change did not regress fidelity”.
The fidelity guide adds that “Fidelity tests that pass don’t regress”, while the failing count can rise as new tests find old differences. Our read of the repository’s history counted 12,708 fixed and 6,544 open issue files at its newest commit, with a median of 21 hours from filing to fix.
What the Labels Show, by Our Count
Each test in the suite carries labels from a published taxonomy. The control plane covers creating, updating, deleting or describing resources; the data plane covers using them, such as data written or messages sent. We matched 29,255 of the 32,402 behavioral tests in the v1.0.322 score file to those labels.
By our arithmetic, the emulator passed 12,796 of 18,818 control-plane tests and 5,399 of 9,955 data-plane tests. Data-plane tests through the signed AWS API passed 58.2%, those that contact a non-API hostname 25.2%, and those that use a native client protocol none of 156. That ordering matches AWS’s own statement that the emulator “matches AWS most closely for control plane APIs”.
Rejection tests, where AWS refuses a request and the emulator must refuse it with the same error, passed 60.6%. Tests that wait out a real AWS-defined period, such as a TTL or an alarm period, passed 42.8%.
Where the Method Stops
A suite measures only what it asserts. The Anthropic post warns that “it is easy to see tests pass and assume the job is done, when this is rarely the case”. AWS’s guide says to compare scores only on the same version of the fidelity tests, and calls a score “a snapshot”.
As we read the label counts, the emulator does best where AWS behaviour is a request and a response, 68.0% of control-plane tests, and worst where a test leaves the AWS API, 25.2% through other endpoints and none of 156 through native protocols, by our arithmetic.
As we read both posts, the suite is also the agents’ work queue: AWS’s finders file issues from failing tests, and the Anthropic post describes each agent picking a different failing test.
Anthropic Engineering, 5 February 2026
“So it’s important that the task verifier is nearly perfect, otherwise Claude will solve the wrong problem.”
Three Implications
FOR TEAMS ADOPTING AGENT-BUILT TOOLSAWS publishes scores with each release and the tests themselves, including the ones the emulator does not pass yet, so a team can check the services it uses.
FOR BUILDERS OF AGENT SYSTEMS Goldberg wrote that agents verify each test against real AWS before submitting it, and the Anthropic post used GCC as a known-good oracle when tests ran out.
FOR READERS OF SCORESAWS’s guide says to compare scores only on the same version of the fidelity tests, because later versions add tests and can retire outdated ones.
The framework argues that, in two frontier labs’ reports on agent usage, “their own measurement apparatus broke”.
As we read it, fidelity suites are one answer to that problem for agent-written software: the measurement is published and rerunnable against a reference. The framework predates AWS Emulator and does not discuss it.
What Is Not Established
Our label counts use the evaluation suite’s current manifest, which may be a different version of the tests from the one behind the v1.0.322 scores, and 3,147 behavioral tests did not match a label. The 10,535 tests of kind “wire” in the score file are not described in the pages we read, so we left them out of the label counts.
We have not run the suite ourselves. We did not contact AWS or Anthropic for this piece.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
A fidelity test pins agent-written code to a reference: real AWS for AWS Emulator, a known-good compiler for Anthropic’s project. By our count, AWS’s published scores show the emulator passing 68.0% of control-plane tests and 54.2% of data-plane tests, a record anyone can rerun with the public suite. Our launch piece and pricing comparison with LocalStack cover the product.
95,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.
A note on sourcing. We read AWS’s fidelity guide for AWS Emulator, the aws-emulator-evaluation-suite README and its taxonomy file, Matthew Goldberg’s About Amazon post and AWS’s launch post of 11 November 2026, and Anthropic’s engineering post of 5 February 2026 on its agent-built C compiler. The pass rates by label are our own join of the score file attached to AWS Emulator release v1.0.322 with the suite’s manifest.jsonl, downloaded on 11 November 2026, matched by test ID.
Repository counts come from our read of the public aws/aws-emulator git history. We did not contact AWS or Anthropic for this piece.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
