We’re proud to have collaborated on a new benchmark built by @GoogleAI: Physical-Engineering’s Last Exam (PLE).
Physical engineers design the skyscrapers, power grids, and chip fabs we rely on, yet no existing benchmark tested whether AI agents could do that work end to end.
Together with Google and practicing engineers, we built 85 long-horizon tasks across mechanical, electrical, civil, and structural engineering. On the private test set, all frontier models score below 50%:
1. Claude Opus 5.5: 48.3% 2. GPT 6 Astra: 45.8% 3. Claude Sonnet 5.5: 43.5% 4. Gemini 4 Argon: 40.6% 5. Claude Opus 5: 40.0%
On the tasks we co-authored, the top agent satisfied 91% of individual constraints, but only 41.3% of its deliverable items passed every hard safety check.
Learning how AI fails at physical engineering can help us improve future models and build a safer world.
This benchmark is also an open competition. Bring your own agent setup and submit it for evaluation on our private test set at http://ple-bench.com
Mercor, the data vendor that supplied the practising engineers, announced its collaboration with Google on PLE on 3 November. Its post lists the five native-harness scores and the 91% versus 41.3% hard-constraint finding that this piece checks against Google’s paper and leaderboard.
Google researchers have published Physical-Engineering’s Last Exam (PLE), a benchmark of 85 long-horizon tasks in mechanical, electrical, civil and structural engineering. Mercor, which supplied the practising engineers, announced the collaboration on X on 3 November.
On the 56 hidden test tasks, no AI agent reaches 50%. Claude Opus 5.5 in Claude Code leads the native-harness ranking on Google’s leaderboard at 0.483, ahead of GPT 6 Astra at 0.458. Google’s own Gemini 4 Argon is fourth at 0.406 on its native harness.
The paper’s sharpest number is about safety rules. On co-authored tasks, the best agent’s deliverable items meet 91% of their constraints on average, yet only 41.3% of items meet every hard constraint. That is a gap of 49.7 percentage points, by our arithmetic.
Business Pill · The Model and the Harness
A Business Engineer explainer of why products built on the same model can differ: the harness around the model. It teaches the general idea only and does not discuss PLE or engineering.
The key insight: As we read it, the best agent on Google’s engineering exam rarely fails outright but still misses hard constraints: on co-authored tasks its items meet 91% of their constraints on average, while only 41.3% of items meet all of the hard ones, a 49.7-point gap by our arithmetic.
What Google Built
The paper, posted to arXiv by authors from Google Cloud AI Research and Google, says solving a task means “completing a realistic, billable engineering deliverable”. Agents must produce files such as engineering memos, calculation workbooks, equipment schedules or 2D/3D drawings.
The benchmark’s site, ple-bench.com, lists 30 mechanical, 27 electrical, 18 civil and 10 structural tasks. Each task runs in an isolated container with no internet access and a six-hour limit, and each model and harness pair runs three times.
The tasks come by two routes. Expert-authored tasks are written end to end by practising engineers from the expert pool of Mercor, which the paper calls “our data vendor”. Co-authored tasks start from engineers’ problem statements, and an agent pipeline builds each project environment and grader before Mercor’s experts validate or harden it.
The paper says a public set of 29 tasks will be released with their graders. The rest form the hidden test set behind the leaderboard, which Mercor describes as an open competition.
PLE leaderboard scores on the 56 hidden test tasks, each model on its native harness, mean of three runs. Source: Google Research PLE (3 Nov 2026), ple-bench.com leaderboard; rubric-graded tasks are judged by Claude Opus 5.5. The 8.3-point top-to-fifth spread is our arithmetic.
The Leaderboard
On the hidden test set, the native-harness scores are: Claude Opus 5.5 in Claude Code 0.483, GPT 6 Astra in Codex 0.458, Claude Sonnet 5.5 in Claude Code 0.435, Gemini 4 Argon in Antigravity 0.406 and Claude Opus 5 in Claude Code 0.400. Mercor’s post lists the same five figures as percentages. On any harness, the top pair is Claude Opus 5.5 in OpenHands at 0.486.
Anthropic models take three of those five places. The spread from first to fifth is 8.3 points, by our arithmetic.
Cost differs more than score. The leaderboard puts Claude Opus 5.5 in Claude Code at $17.12 per task and 63 minutes, GPT 6 Astra in Codex at $9.53 and 26 minutes, and Gemini 4 Argon in Antigravity at $7.70 and 73 minutes.
The paper reports its own figure on all 85 tasks: Claude Opus 5.5 in Claude Code averages 50.6%, spending 27 million tokens and 63 minutes per task. That is why some reports quote 50.6% and others 48.3%.
Where the Agents Fail
The paper says the top configuration “rarely fails outright, scoring 0% in only 11 of its 255 runs”. Its losses come from hard constraints instead. Under the paper’s scoring, an item earns objective credit only if it satisfies every hard constraint.
Mercor’s post puts the same finding this way: the top agent “satisfied 91% of individual constraints, but only 41.3% of its deliverable items passed every hard safety check”.
For Claude Opus 5.5 in Claude Code, the paper reports mechanical work as the hardest discipline at 40.5% and civil as the easiest at 69.9%. Across all runs of all 86 model and harness pairs, 23.2% score zero.
The paper also finds the model matters far more than the harness. On all 85 tasks, it reports a 50-point spread across models’ best scores, against a 0.7 to 12.1 point spread across harnesses for a fixed model.
Who Grades the Work
The leaderboard says Claude Opus 5.5 grades the tasks that use a rubric. The paper tested that choice by re-grading Claude Opus 5.5’s runs with Claude Opus 5 and Gemini 4 Argon. It reports that Claude Code stays the top harness for Claude Opus 5.5 under every judge, and measures “a 2.3-point combined self-preference” on the 44 rubric-graded tasks.
The Business Lever
As we read it, the lever is billable engineering hours. The paper cites US Census data showing that US engineering-services firms report over $300 billion in yearly revenue, and it frames each task as a deliverable a licensed reviewer would sign off on. As we read it, the near-misses are the point: under the paper’s scoring an item that meets most of its constraints earns no objective credit unless it meets every hard one, and only 41.3% of the top agent’s items on co-authored tasks did.
As we read the leaderboard, Google built the exam and its own Gemini 4 Argon ranks fourth on its native harness, behind two Anthropic models and one OpenAI model. It is also the cheapest of those five at $7.70 per task.
As we read the paper, Claude Opus 5.5 both leads the leaderboard and grades its rubric tasks. The paper’s own re-grading with Claude Opus 5 and Gemini 4 Argon measured a 2.3-point combined self-preference, and it says Claude Code stays the top harness for Claude Opus 5.5 under every judge.
As we read Mercor’s post, the hidden test set is also an open competition: Mercor says anyone can bring an agent setup and submit it for evaluation on the private test set.
PLE paper, Google Cloud AI Research and Google (arXiv 2610.30537)
“Even the strongest configuration reaches a mean score of only 50.6%, while spending 27M tokens and 63 minutes per task on average, demonstrating the substantial gap between current agents and autonomous physical engineering design”
Three Implications
FOR ENGINEERING FIRMS The paper frames each task as a deliverable a licensed reviewer would sign off on. On the hidden test set the best native-harness score is 0.483.
FOR AI LABS The paper reports that the model matters far more than the harness: a 50-point spread across models against 0.7 to 12.1 points across harnesses for a fixed model.
FOR GOOGLEGoogle’s Gemini 4 Argon ranks fourth on its native harness at 0.406, at $7.70 per task, on a benchmark built by Google researchers.
The framework argues: “The surface, not the model, governs autonomy.”
As we read it, PLE puts a number on the other side of that question for engineering scores: the paper finds the model matters far more than the harness, though the harness still moves cost per task. The framework is about how much autonomy users hand to agents, not about benchmark scores, and it does not discuss engineering.
What Is Not Established
The leaderboard ranks models by their best harness, and on that view Claude Opus 5 (0.426) edges ahead of Gemini 4 Argon (0.411). The fourth place in our headline is on each model’s native harness, the view Mercor’s post uses; the paper’s best-harness ranking on all 85 tasks also puts Gemini 4 Argon fourth.
The 91% and 41.3% figures cover co-authored tasks and one configuration, Claude Opus 5.5 in Claude Code. Our 49.7-point gap subtracts two different measures, constraints met and items fully passing, so it is a comparison, not a pass rate. The benchmark scores files, not built structures; a score here is not a licence to practise. We did not contact Google or Mercor.
This Week In The AI Supercycle
Anthropic’s Reality Check, The Capital Squeeze, & The New World Order
This week’s issue: for two years the AI build-out was priced as a growthstory. This week it started being priced as a credit story.
Google’s PLE benchmark asks AI agents to deliver real engineering work across four disciplines, and on its hidden test set the best agent, Claude Opus 5.5, scores 0.483. Google’s Gemini 4 Argon is fourth on its native harness. The top agent meets 91% of its constraints but fully passes only 41.3% of its items, a 49.7-point gap by our arithmetic.
95,000+ executives read Business Engineer for the AI strategy frameworks cited by ChatGPT, Claude, and Perplexity.
A note on sourcing. This piece is based on the PLE paper by Google researchers (arXiv 2610.30537, version 2), the leaderboard and site at ple-bench.com, read on 3 November, and Mercor’s post on X of 3 November. The 49.7-point gap and the 8.3-point spread are our arithmetic. We did not contact Google or Mercor.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
