Only 81 of 522 prominent AI benchmarks, or 16%, report a separate result for any organizational condition of work, according to a new paper from Mercor Research and Copenhagen Business School. Fewer than a quarter, 118 or 23%, even claim to build one into their tasks.
The authors, Victor Barres, Aden Barton, Nicolai J. Foss and Henrik M. Nielsen, call the missing piece “organizational difficulty”. Mercor’s blog post puts it plainly: “benchmarks don’t measure jobs; they measure tasks”. The trend is moving: 42 benchmarks published in 2026 claim an organizational dimension, against 6 in 2023, 7.0 times as many by our arithmetic.
Business Pill · Span of Control
A short explainer of span of control: why a small decision can take a week when it has to climb layers of managers. It teaches the general idea only and does not discuss this paper.
The key insight: As we read it, the paper separates two questions that benchmark headlines tend to merge: whether a model can do a task, and whether it can do that task inside an organization. 81 of 522 prominent benchmarks measure any organizational condition, and the 42 new ones claiming one in 2026 are 7.0 times the 6 from 2023, by our arithmetic.
The paper separates two kinds of difficulty. Technical task difficulty is the difficulty of a task “once the goal, the inputs and the authority to act are given”. Organizational task difficulty is “the difficulty that comes from how a task relates to other people and activities”. The blog says “The two are orthogonal”.
Its example is a software engineer fixing a customer-reported bug. The code change is small, but the logs sit with a hard-to-reach support team, the billing team owns the code and must approve, the product manager doubts it is a bug, and the account manager has already promised a fix this week. In the blog’s words, “the software engineering itself is quite easy”.
The taxonomy has four layers: the structure of work, coordination requirements, governance mechanisms and performance outcomes. The first layer has six dimensions: task interdependence, information structure, authority structure, temporal structure, incentive structure and environmental dynamism. The authors describe it as an organizational counterpart to O*NET’s task database.
AI models are rapidly improving on benchmark tasks, and that progress is often read as an answer to a bigger question: can AI do your job
But benchmarks donโt measure jobs; they measure tasks. Most economically valuable work takes place within companies, where people must coordinate, decide and negotiate to get things done. Jobs are messy in a way that tasks are not.
In a new paper, Mercor Research’s @aden_barton and @victorbarres_, along with @NicolaiFoss and Henrik M. Nielsen, develop a vocabulary for this messiness, which they call organizational difficulty, and apply it to ~500 prominent AI benchmarks.
Mercor announced the paper on X on 5 November. The post says benchmarks measure tasks rather than jobs, that most economically valuable work takes place within companies, and that the authors apply their idea of organizational difficulty to about 500 prominent AI benchmarks; the paper itself counts 522.
What the 522 Benchmarks Show
The authors tagged benchmarks from three groups: the Artificial Analysis Intelligence Index and Epoch AI’s Benchmarking Hub, oral, spotlight or award papers at six major machine-learning conferences, and benchmarks that at least two of five AI labs (OpenAI, Anthropic, Google, xAI and Meta) report in model releases. Each paper was scored on whether it mentions, claims or measures each dimension.
60% of the 522 mention at least one dimension, 23% claim one and 16% measure one. Task interdependence is the rarest: claimed by 4% and measured by 1%. Of the 19 benchmarks that claim it, the paper says “only two, TheAgentCompany and SWE-Lancer” involve colleagues in an organization; the rest involve a simulated user, a human collaborator or other agents in a game.
Authority, when it appears, is “almost always represented as a written policy in the prompt”, the blog says. Knowledge held by colleagues appears in only 4 of the 66 benchmarks that claim information structure. Of the 118 that claim any dimension, 81 measure one, about 69% by our arithmetic.
The paper names two benchmarks built around real professional work, GDPval and the Remote Labor Index, and says “neither claims any of the six organizational dimensions”, because in both the agent works alone with the materials supplied up front. It adds: “This does not mean they are unrealistic”.
Why It Matters for AI and Jobs
Benchmark gains are often read as a sign that AI can do whole jobs. The paper says benchmarks are often taken as evidence that AI can do entire jobs, while most evaluate atomic tasks. The blog says that without progress on organizational tasks, “the productivity gains from model progress may be limited”. It links this to agents joining organizations as coworkers, naming Claude Tag and Devin.
The trend is rising from a low base. By year of publication, the share of prominent benchmarks claiming an organizational dimension went from 14% in 2023 to 23% in 2024 and 2025 and 33% in 2026, per the blog’s chart. The 42 such benchmarks from 2026 match the 42 from all of 2025, from 129 benchmarks instead of 186, by our arithmetic.
Mercor sells human data and evaluation work to AI developers and runs its own APEX benchmarks, according to its website, so it has a business interest in new kinds of evaluation. Foss and Nielsen are at Copenhagen Business School. The paper says the authors used Claude Opus 5.5 to code each benchmark’s paper, then refined definitions by reading cases. It was posted to arXiv on 2 November and on Mercor’s blog on 5 November.
Connecting the Coverage
Our related pieces on AI at work and how it is measured:
As we read the paper, a high benchmark score says the technical part of a task is solved once the goal, the inputs and the authority to act are supplied.
As we read the counts, the industry measures the condition it can write into a prompt: authority, when it appears, is almost always a written policy.
As we read the by-year chart, organizational content is rising from a low base: a third of 2026 benchmarks claim a dimension, against 14% in 2023.
Mercor Research, blog post, 5 November 2026
“Work is fundamentally organizational.”
Three Implications
FOR READERS OF AI JOB FORECASTS The paper says benchmarks are often taken as evidence that AI can do entire jobs, while most measure single tasks done alone.
FOR COMPANIES DEPLOYING AGENTS The blog names agents joining organizations as coworkers, such as Claude Tag and Devin, which face approvals, colleagues and deadlines.
FOR BENCHMARK BUILDERS The authors offer the taxonomy as a shared vocabulary; its tags for coordination, governance and outcomes are exploratory.
The framework says: “The first is what your underlying technology can do. The second is what your users can actually do with it.”
As we read it, the Mercor paper measures one source of that gap inside companies: benchmarks track the first curve on tasks done alone, while the second runs through colleagues, approvals and deadlines. The framework does not discuss this paper.
What Is Not Established
The paper counts what benchmarks represent; it does not test how well current models handle organizational conditions. It does not estimate how many jobs or what share of work are organizationally difficult. The blog calls the taxonomy “preliminary”, and its tags for layers 2 to 4 are described as exploratory.
This Week In The AI Supercycle
Anthropic’s Reality Check, The Capital Squeeze, & The New World Order
This week’s issue: for two years the AI build-out was priced as a growthstory. This week it started being priced as a credit story.
Most of the AI tests the industry reports measure a task done alone, with the information and authority supplied: 81 of 522 prominent benchmarks measure any organizational condition, according to Mercor Research and Copenhagen Business School. The count of new benchmarks that claim one rose from 6 in 2023 to 42 in 2026, 7.0 times as many by our arithmetic.
A note on sourcing. Figures and quotations are from the Mercor Research and Copenhagen Business School paper on arXiv (2611.03494) and Mercor’s blog post of 5 November 2026; by-year shares are read from the blog’s chart. Ratios are our arithmetic.
Gennaro Cuofano is a CRO and tech executive who has worked in AI since late 2015, starting with bringing NLP, natural-language generation, voice and chatbot products to the marketing industry. His background is in law and finance: he holds a Master's degree in Law and an International MBA with an emphasis on corporate finance (LUISS Business School and the University of San Diego, 2012), and worked as a financial analyst at a real-estate investment firm in San Diego and as an assistant controller. His work focuses on business model strategy, business engineering and, more broadly, structural analysis: how companies make money, read from their own filings. He created FourWeekMBA and leads research there and at The Business Engineer, his newsletter on AI and business strategy, with over 95,000 subscribers and more than 1,000 published analyses. His writing has also appeared on Entrepreneur (for example, How to Digitize a Family Business in Europe) and HackerNoon. Find him on LinkedIn and Substack. See how we source.
