Editors Preferred Claude’s Paper Reviews in 87% of Picks

Three former editors of major scholarly journals preferred a review written by Claude Opus 4.8 over a paid human expert’s review in 26 of 30 blind picks, or 87%. That is according to a working paper by Gary King and Steven Worthington of Harvard’s Institute for Quantitative Social Science, dated 22 October 2026. An OpenAI model, GPT-5.5, acting as a second judge, preferred the AI review in all 132 pairs where it reached a decision.

Peer review is the step in which experts read a research paper before publication and point out its errors and weaknesses. The authors write that “AI-generated article reviews are now better, by every measure we could construct, than reviews written by well-compensated human experts”. The paper also says what it did not test: whether the advice made papers better, and how editors should decide what to publish.

Business Pill · AI AGENTS

A short Business Engineer explainer on AI agents that do work rather than answer questions. It teaches the general idea only and says nothing about the study in this piece.

What the Paper Found

• Editors. Three former editors of leading journals independently judged 10 randomly selected pairs, one AI and one human review in random order, without knowing that one review was written by AI; they picked the AI review 87% of the time, the paper reports.
• Accuracy. The paper reports that 5.5% of the AI reviews’ claims about the manuscript were judged incorrect, “about a third of the human reviews’ 15.9%”, and that the AI reviews cited no fabricated sources.
• Agreement. The editors agreed with each other 80% of the time but agreed with the AI judge 87% of the time on average, according to the paper.
• Length. On 44 pairs where the two reviews were of similar length, the AI review was still preferred in all 44, the authors write.

Source: Gary King and Steven Worthington, Institute for Quantitative Social Science, Harvard University, working paper dated 22 October 2026. All figures are the paper’s.

What Was Posted on X

AI commentator Rohan Paul’s post on 23 October summarised the study: Claude Opus 4.8 reviews against paid rapidPeer human reviews, judged blind by GPT-5.5 and three former journal editors, with the AI review picked in all 132 decided pairs by GPT-5.5 and 87% of the time by the editors. His closing line, that teams can let AI do the first full review pass and keep a human for the final decision, is his own reading.

Ethan Mollick wrote on X that, by every measure, Opus 4.8 beat human experts as peer reviewers in a study of 124 Harvard social science papers, and that academics need to adapt science to AI fast.

How the Test Worked

The human reviews came from rapidPeer, a Harvard service that gives university faculty double-blind reviews of their paper drafts before journal submission. The corpus holds 124 paper submissions and 134 human reviews collected between 2019 and 2026, because 10 submissions received two reviews, the paper says. Reviewers were paid, and 88% of the 74 authors who answered a survey rated their review as high or very high quality, according to the paper.

For each submission, the authors generated one AI review with Claude Opus 4.8 at its maximum reasoning setting, once with the same guidelines the human reviewers got and once with a tuned prompt and web search. A model from a different family, GPT-5.5, judged each pair. In blind pairwise judging, the judge sees two reviews of the same paper side by side without being told who wrote them and picks the better one; here each comparison ran in both orders, and a win counted only when the same review won both times.

A summary of the paper posted on X by AI commentator Rohan Paul says Claude “reviewed the same 1124 papers”. The paper itself describes 124 paper submissions and 134 human reviews.

What the Authors Propose

The authors list options for journals rather than a single recommendation. They include asking authors to run repeated AI reviews before submission, publishing an official AI review prompt, and requiring an AI review plus the author’s response with each submission. They also name a risk: “AI benefits us individually but risks making the collective efforts of science more homogenous and ordinary.”

The paper also notes that its results speak to helping authors improve their papers, and that they “do not speak to the second purpose, which is on what basis editors should make publication decisions”.

Connecting the Coverage

Our Sakana reviewer piece reported on 10 October that Sakana AI’s review system caught 73.43% of contradictions its researchers had planted in papers’ main claims, on a synthetic test. Our proofs piece reported on 23 October that Theorem’s cost model for AI-written code proofs starts at about $40 per kilobyte of machine code. What’s new: this paper compares AI reviews with real, paid human reviews of the same unpublished drafts.

Our expert-pay piece reported that Bloomberg found recruiters offering up to $400 an hour to civil rights attorneys to help train AI systems. Our Tao piece reported that Terence Tao told a Caltech lecture on 9 October that mathematics has moved from proof scarcity to proof abundance.

The Structural Read

As we read it, the human side of the test was strong by design: the reviews were paid, came from a Harvard pre-submission service, and 88% of 74 surveyed authors rated theirs high or very high quality, the paper says.

As we read it, the editors’ agreement figures matter as much as the win rate: the paper reports that the editors agreed with each other 80% of the time and with the AI judge 87% of the time on average.

As we read it, the paper separates two jobs that peer review usually bundles: giving authors constructive advice, which it tested, and deciding what to publish, which it says its results do not speak to.

Three Implications

FOR JOURNALS The paper lists options from asking authors to run AI reviews before submission to requiring an AI review and the author’s response with each paper.

FOR RESEARCHERS In the paper’s tests, the AI reviews made about a third as many incorrect claims about the manuscript as the human reviews, 5.5% against 15.9%.

FOR ANYONE BUYING EXPERT CRITIQUE The paper’s design compared one AI review per paper with a paid human review of the same draft, judged blind by an AI model and by three former editors.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Four Intelligence Moats.

The framework describes “The Verifier Moat” as “built by RLVR on domain-specific reward signals, building into a new asset class for reasoning-heavy verticals.”

As we read it, a review that checks a paper’s claims against the paper is a verifier in that sense, and the paper’s accuracy test measures how good the check is. The framework was published in July 2026 and does not discuss peer review.

What Is Not Established

The editors judged only 10 pairs, and the larger tests rely on an AI judge. The papers came from one service used by university faculty, and the authors did not measure whether revisions based on the reviews improved the papers. The paper says its supplementary appendix will be available on the authors’ site; we read the 22 October version of the paper. We did not contact the authors. Nothing here is investment advice.

The Bottom Line

In the Harvard working paper, former journal editors picked Claude’s reviews over paid human reviews in 87% of blind picks, and the AI reviews made about a third as many incorrect claims. The authors say the result is about helping authors improve papers, not about how editors should decide what to publish.