Google Test: Best AI Engineer Agent Hits 50.6% at $17 a Task

The strongest AI agent configuration tested on a new benchmark of professional engineering work scored a mean of 50.6%, according to a paper by researchers at Google Cloud AI Research posted on arXiv on 21 October 2026. It spent 27M tokens and 63 minutes per task on average. That configuration, Claude Opus 5.5 running in the Claude Code harness, cost $17.18 per task, the paper says.

The benchmark, called PLE (Physical-Engineering’s Last Exam), has 85 tasks across electrical, civil, structural and mechanical engineering, “where solving a task means completing a realistic, billable engineering deliverable.” The authors tested 14 frontier models in seven publicly available agent harnesses and write that the results show “the substantial gap between current agents and autonomous physical engineering design.”

Business Pill · COST PER TASK EXPLAINED

A short Business Engineer explainer of pricing AI by the task rather than by the token. It teaches the general idea only and says nothing about this paper.

What the Test Asks Agents to Do

The paper describes the work as “modelling physics and legal code requirements across a multi-format project corpus, optimising material and installation costs, and issuing 2D/3D engineering design packages that a licensed reviewer would sign off on.” Its examples include sizing steel roof decking under snow-drift loads, laying out a phased campus as a 3D model and sizing low-voltage switchboards and feeder circuits.

Of the 85 tasks, 44 were written end to end by practising engineers and 41 were co-authored: engineers wrote the problem statements, and an adversarial agent pipeline built the project environment, grader and reference deliverables before experts validated them. By discipline, 30 tasks are mechanical, 27 electrical, 18 civil and 10 structural. The authors say they will release 29 of the 85 tasks with their graders and keep the rest private for a leaderboard.

What Was Posted on X

DAIR.AI shared the paper on X, writing that for professional engineering deliverables the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task; the summary is DAIR.AI’s own.

Scores and Cost per Task

The paper reports both a mean score and an inference cost per task for each model and harness pair. It writes that GPT 6 Astra in Codex “comes within 2.3 points of the top score at about half the cost”, and that “Higher cost does not always yield a higher score”: the eight most expensive configurations cost $27–54 per task yet scored 33.8%–49.3%.

Source: Han et al., “PLE: Physical-Engineering’s Last Exam”, Google Cloud AI Research, arXiv 2610.30537, 21 October 2026. Mean task score over three runs on all 85 tasks; cost is the paper’s inference cost per task.

By the paper’s own ranking of each model’s best harness, scores fall from 50.6% for Claude Opus 5.5 and 48.3% for GPT 6 Astra to 45.2% for Claude Sonnet 5.5 and 43.2% for Gemini 4 Argon. Qwen3 Coder 480B, MiniMax M2, Kimi K2 Thinking and Llama 4 Maverick “stay below 7%”. The authors write that “The model matters far more than the harness”.

Where the Agents Fall Short

The top configuration has a median task score of 43.5% and scores below 50% on 48 of the 85 tasks, the paper says. Mechanical (40.5%) and structural (46.6%) work were the hardest, and civil the easiest (69.9%). Co-authored tasks took far more work: 39.3M tokens and 97 minutes per attempt, against 15.6M tokens and 31 minutes for an expert-authored task.

The main failure is meeting every hard rule at once. On co-authored tasks, the top configuration’s deliverable items satisfy 91% of their constraints on average, “yet only 41.3% of the items satisfy every hard constraint”. Across all runs of all model and harness pairs, 23.2% scored 0%.

What Was Posted on X

DAIR.AI shared the paper on X on 23 October, writing that “the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task.” The post is DAIR.AI’s own summary of the paper.

Connecting the Coverage

In our 27B agent-interface piece, a Simular paper reported a 27-billion-parameter open model fully solving 21.3% of OSWorld V2 desktop tasks at an estimated $1.10 per task. In our harness scaling piece, a paper found that training an agent across 18 harness configurations beat training through one. In our Steinberger token-spend piece, OpenAI’s Peter Steinberger estimated spending around $10 million in tokens last month. What’s new here, as we read it, is a price tag on a billable professional deliverable: the paper puts the best result at $17.18 and 63 minutes of agent time per task, for a mean score of 50.6%.

The Structural Read

As we read it, the score measures whole deliverables, not answers: the paper says solving a task means completing a realistic, billable engineering deliverable, graded against physics and code requirements.

As we read it, the gap is in the last few rules: on co-authored tasks the top configuration’s items satisfy 91% of their constraints on average, but only 41.3% satisfy every hard constraint, the paper says.

As we read it, cost and score do not move together: the paper says the eight most expensive configurations cost $27–54 per task yet scored 33.8%–49.3%, below the $17.18 top result.

Three Implications

FOR FIRMS BUYING AI AGENTS The paper says results on PLE should always be reported with both the model and the harness.

FOR ENGINEERING FIRMS The paper says the main obstacle remains producing a design that satisfies every hard constraint while still optimising its cost.

FOR BENCHMARK READERS The paper says rubric items were judged by Claude Opus 5.5 and reports a 2.3-point combined self-preference on rubric-graded tasks.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Agentic Harness War.

The framework describes OpenAI’s bet as “pull every workflow into one agent, own the harness”, and says “Anthropic is betting on many harnesses, embedded where work already lives.”

As we read it, this paper measures both sides on the same professional work and finds the model mattered more than the harness, while the harness still changed cost: Claude Opus 5.5 cost $17.18 per task in Claude Code and $35.15 in Codex. The framework was published in July 2026 and does not discuss this paper.

What Is Not Established

The scores come from the authors’ own grading. Rubric items were judged by Claude Opus 5.5, and when the authors re-graded runs with other judges, Claude Code still ranked first, but they report “a 2.3-point combined self-preference” on the rubric-graded tasks. The cost figures are the paper’s inference costs, not a full price for engineering work, and the paper does not compare them with what a human engineer would charge for the same deliverable.

The authors note that engineers were paid on average $4,663 per expert-authored task; that is the cost of building the test, not of doing the job. We did not run the benchmark. Nothing here is investment advice.

The Bottom Line

On 85 professional engineering tasks graded against physics and building-code requirements, the best AI agent configuration Google Cloud AI Research tested scored a mean of 50.6% at $17.18 and 63 minutes per task. Only that one configuration passed 50%, and the paper says its losses stem from hard constraints its designs did not fully meet.