Amazon Web Services on Wednesday introduced the Universal Agent Evaluation Framework, or UAEF, in an AWS Machine Learning Blog post. It is an open-source system, under the Apache 2.0 license, that scores AI agents with 36 built-in metrics across 7 dimensions, whatever framework they were built with.
AWS said customers across 12 organizations in telecom, healthcare and financial services used it to compare agent versions before release. By our count of the project’s metrics catalog, 24 of the 36 built-in metrics, two in three, are scored by an LLM judge on Amazon Bedrock.
Business Pill · The Model and the Harness
An evergreen Business Engineer explainer on why products built on the same model differ: the harness around the model. It teaches the general idea only and does not discuss this news.
The key insight:AWS introduced UAEF, an open-source Apache 2.0 evaluator that scores AI agents with 36 built-in metrics across 7 dimensions, whatever framework built them. By our count of its metrics catalog, 24 of the 36 are scored by an LLM judge on Amazon Bedrock and 12 compute directly from the trace.
The blog post says teams build metrics, scoring logic and result storage against one framework’s trace format, and “they must rebuild them when they switch frameworks or run agents on more than one”. UAEF uses adapters to convert each run into one canonical record, the AgentTrace, which is the only thing the rest of the system scores.
The awslabs/uaef repository lists adapters for LangGraph, LangChain, AWS Bedrock, Langfuse, Strands and AgentCore, plus a generic JSON adapter for custom frameworks. The repository includes the Python library, the infrastructure for a deployable service with a web interface, and runnable notebooks, AWS said.
UAEF built-in metrics per dimension and how many of each are scored by an LLM judge on Amazon Bedrock. Source: UAEF metrics catalog; counts are our tally of the catalog table.
The Count: 24 of 36 Metrics Call a Judge
The metrics catalog marks each built-in metric as deterministic or LLM-judged. Tool calling and performance, 4 metrics each, are all deterministic. Response quality and reasoning, 4 each, are all LLM-judged. Responsible AI splits 3 judged to 1 deterministic, and multi-agent splits 2 and 2.
Multi-turn conversation carries 12 of the 36 metrics, one third of the set, and 11 of them are LLM-judged, by our count. Across all seven dimensions, 12 metrics compute directly from the trace and 24 prompt a judge model on Bedrock.
The blog post says deterministic metrics “reproduce exactly on repeated runs”, while judge metrics cover qualities such as relevance and safety. It also warns that scoring every agent on every dimension “wastes time and Amazon Bedrock invocations”. AWS credentials are required for the judge and for storage, the README says.
How It Compares With AgentCore Evaluations
AWS’s managed service, Amazon Bedrock AgentCore Evaluations, became generally available in March with 13 built-in evaluators, AWS said at the time. UAEF’s 36 built-in metrics are about 2.8 times that number, by our arithmetic. UAEF can also read traces from agents running on AgentCore.
Stored runs become experiments, with metadata in Amazon DynamoDB and full results in Amazon S3. The comparison engine flags a regression or improvement at a default threshold of 5 percent, and grades regressions as minor, moderate or severe. On cost and scale in multi-agent systems, see Amazon’s own multi-agent token study.
The Structural Read
As we read the catalog, the framework-neutral layer sits in the adapters, while two thirds of the scoring runs through Bedrock: the README says AWS credentials are needed for the judge metrics and for DynamoDB and S3 storage.
AWS Machine Learning Blog, 11 November 2026
“they must rebuild them when they switch frameworks or run agents on more than one”
Three Implications
FOR TEAMS ON SEVERAL FRAMEWORKS UAEF lists adapters for LangGraph, LangChain, AWS Bedrock, Langfuse, Strands and AgentCore, plus a generic JSON adapter, per its README.
FOR EVALUATION COSTAWS says scoring every agent on every dimension wastes time and Bedrock invocations; 24 of the 36 built-in metrics call a Bedrock judge, by our count.
FOR AWS’S OWN LINEUP AgentCore Evaluations lists 13 built-in evaluators, AWS said in March; UAEF lists 36 built-in metrics, about 2.8 times as many by our arithmetic.
The framework argues: “the conversational interface was scaffolding. The harness is the building.”
As we read it, an evaluator that scores agents across harnesses is tooling for that layer, and AWS has routed most of its scoring through its own model service. The framework predates this news and does not discuss it.
UAEF built-in metrics by scoring method. Source: UAEF metrics catalog; the split is our count of the catalog table.
What Is Not Established
AWS did not name the 12 organizations or give their results in the post we read. The post does not give Bedrock judge prices or the number of judge calls per evaluation, so the cost of a full run is not stated. The repository was created in September, its GitHub page shows.
This Week In The AI Supercycle
The Great Convergence of Capital, Power, and Geopolitics
This week’s issue: for two years the AI build-out was priced as a supply story, and last week as a credit story. This week it was priced as a permission story, and the answer arrived as a cost, not a ban.
AWS has published an open-source agent evaluator with 36 built-in metrics, and by our count 24 of them rely on an LLM judge running on Amazon Bedrock. The framework sits beside AWS’s managed AgentCore Evaluations service, which lists 13 built-in evaluators.
A note on sourcing. UAEF’s design, the 12-organization figure and the 5 percent default threshold are from AWS’s Machine Learning Blog post of 11 November 2026. Adapter list and requirements are from the awslabs/uaef README; per-dimension counts are our tally of the project’s metrics catalog. The 13 AgentCore evaluators are from AWS’s March 2026 announcement. Ratios are our arithmetic. AWS was not contacted for this piece.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
