Claude, GPT Found 2x to 3.6x the Errors Peer Reviewers Did

Fifteen large language models, including Anthropic’s Claude and OpenAI’s GPT models, found more of the major errors inserted into three clinical-trial reports than human peer reviewers did, according to a medRxiv preprint posted on 28 October. The models’ major-error detection scores ranged from 53% to 96%. Untrained human reviewers in an earlier experiment on the same reports scored 27%.

By our arithmetic, the lowest-scoring model found 1.96 times the human share (53/27) and the highest found 3.56 times it (96/27). The preprint has not been peer reviewed.

Business Engineer Course · The FDE Playbook

An evergreen Business Engineer course on forward-deployed engineering, how AI gets put to work inside organisations. It teaches the general idea only and does not discuss peer review or this preprint.

What the Preprint Tested

The authors are Dominikus Brian of OrganoBit BioIntelligence, Xavier Isidro Pereira Hernández and Jeffrey Koury of ResearchHub Foundation, Hao Yin of Western University and Ruslan Rust of the University of Southern California. They write that in “an earlier experiment with errors inserted into three reports of randomized clinical trials, untrained human reviewers achieved a mean major-error detection score of 27%”.

They then “tested whether 15 large language models (LLMs), including four open-weight models, detect the same errors”. To reduce the risk that the models had memorised the old papers, they “repeated the test on 58 randomized clinical trials published in 2026, including all eligible phase 3 trials”. According to the abstract, “1,796 reviews were scored by two blinded LLM judges”.

What Was Posted on X

Andrew Curran writes that even little Haiku found about twice as many major errors as human reviewers.

The Results

On the original reports, the abstract says, “the models’ major-error detection scores ranged from 53% to 96%”. On the 2026 trials, “scores ranged from 58% to 99%, and from 66% to 89% for the open-weight models”.

The abstract names “Opus 5.5” as the model with the highest major-error detection score, 99%, and says “GPT-6.1 Sol and GPT-6 Luna offered cost-effective alternatives with scores of 94% and 86%, respectively”. Mean review times “ranged from 1.4 to 16.3 minutes”.

In a chart Rust posted with his thread on X, Anthropic’s Haiku 4.5 sits lowest of the 15 models on the original reports. Andrew Curran wrote: “Even little Haiku found about twice as many major errors as human reviewers.”

Where Humans Missed Most

Rust wrote on X that the errors humans miss most “are the ones that change how a trial reads”. He listed human detection rates from the earlier experiment: “Text and table disagree: humans caught it 7% of the time. Reversed comparison: 10%. Abstract contradicts the results: 25%. Patient numbers don’t add up: 32%.”

He added that “The four leading models caught all four in every single review of the 2026 trials”.

The Policy Debate

Rust framed the study against “debate this week on whether journals should ban AI in peer review”. His conclusion on X: “Journals that ban AI in review are choosing the 27% and are basically not detecting 3 out of 4 major errors.”

On confidentiality, he wrote that for unpublished work, “open-weight models you can run on your own computer found up to 89% (Kimi K3)”. On cost, he wrote that “GPT-6.1 Sol found 94% of major errors for about 10 cents per review”.

The Structural Read

As we read the preprint, the comparison that holds is on the three original reports, where humans and models faced the same inserted errors; the 2026-trial scores have no human counterpart in the study.

As we read the abstract, the models were not error-free: they flagged 14% to 29% of the error types in control reports that had no inserted errors.

As we read the disclosures, four of the five authors are affiliated with ResearchHub Foundation, which the paper says is implementing AI-assisted review.

Three Implications

FOR JOURNALS Rust wrote on X that journals that ban AI in review are choosing the 27%.

FOR TRIAL READERS The preprint says the most frequent false flags on control reports were conclusions not supported by the results and missing reasons why screened patients were not enrolled.

FOR MODEL CHOICE The abstract says GPT-6.1 Sol and GPT-6 Luna offered cost-effective alternatives with scores of 94% and 86%.

The Business Engineer Lens

This story maps onto the Business Engineer framework The Product Overhang Doctrine.

The framework describes the gap this way: “They are not the same curve, and the gap between them — what the platform makes possible versus what the product surfaces — has a name.”

The framework is the owner’s general argument about capability and adoption, not evidence about this preprint; it does not discuss peer review.

What Is Not Established

The models were not error-free. The abstract says “LLMs still flagged 14% to 29% of the error types in control reports without inserted errors”, most often conclusions not supported by the results.

The reviews were scored by LLM judges, not by human editors. The 27% human figure comes from an earlier experiment on the original reports only, so the 2026-trial scores are not compared with it here. The authors disclose that four of them “are affiliated with ResearchHub Foundation, whose open-science platform offers incentives for human peer review and is implementing AI-assisted review”.

This connects to our recent coverage of how AI shapes expert judgement: Anthropic’s study of analysts siding with Claude Code and AI-written text reaching 24% of web pages.

The Bottom Line

In a preprint posted on 28 October, all 15 AI models tested found more of the inserted major errors in three trial reports than untrained human reviewers did in an earlier experiment: 53% to 96%, against 27%. By our arithmetic that is 1.96 to 3.56 times the human share. The models also flagged errors that were not there in 14% to 29% of error types on control reports.