AI Agents Reproduced 2,226 ICML Papers in Just 19 Days

Machine learning conferences have grown faster than anyone's capacity to check their contents. ICML 2026 accepted 6,352 papers. Between July 15 and August 2, a Hugging Face community challenge attempted to reproduce a third of them — and published every logbook.

• Coverage: 2,226 papers attempted, roughly 34% of the accepted proceedings
• Participation: 1,221 community participants producing 6,816 public reproduction logbooks and 2,962 cloud jobs
• Verification: 35,908 individual claims evaluated; 266 papers fully reproduced with every claim verified
• Timeline: 19 days, from July 15 to August 2, 2026

How a 19-Day Audit of a Conference Works

The challenge, led by Abubakar Abid with more than 63 contributors from Hugging Face and the wider community, began by indexing all accepted ICML 2026 papers and extracting their core scientific claims. Participants then pointed AI coding agents — Claude Code, Codex, Cursor and Pi among them — at those claims and attempted to actually run the experiments.

Every attempt was documented in a Trackio logbook, published openly. An automated judge built on the open-weights GLM-5.2 model then assigned each claim one of four verdicts: verified, falsified, supported only at toy scale, or inconclusive.

The result is, as far as anyone can tell, the largest open claim-by-claim audit of a machine learning conference ever attempted. Fifty-one percent of examined papers had at least one claim verified, and 266 came through with every extracted claim confirmed by a real experiment.

What Did the Reproduction Challenge Find?

The honest headline is mixed, and the organisers presented it that way. Alongside the 266 fully reproduced papers and 632 partially reproduced with nothing falsified, 23% of examined papers had at least one claim falsified or contested, and 49 had all claims falsified.

Those numbers deserve careful reading rather than alarm. A falsified verdict from an automated judge is a signal to look closer, not a verdict on the research — reproduction attempts fail for many reasons that have nothing to do with the original work, including missing hyperparameters, unavailable data, and compute budgets that do not match the paper's. The organisers' own conclusion is that humans remain essential for steering agent work, questioning assumptions, and judging the irreducibly subjective parts of a result.

What the exercise genuinely establishes is that the audit is now affordable. Checking a third of a major conference used to be unthinkable; here it took nineteen days and a volunteer community.

Why Agent-Driven Reproducibility Could Become Standard

The value is not the pass rate. It is that 3,978 individual claims are now backed by a public logbook someone else can inspect and rerun. That is a different epistemic footing than a PDF and a promise, and it accrues to the papers that pass.

The tooling to support this is arriving in parallel — Claude Science, an AI workbench built for reproducible research, points at the same need from the authoring side. And the fact that the automated judge ran on an open-weights model matters: an audit infrastructure that depends on a closed API is an audit infrastructure someone else controls. The 900 million downloads Gemma has accumulated across open models is the supply side of that same story.

For anyone tracking AI research and open models, the interesting prediction is structural. If reproduction logbooks become an expected artefact alongside the paper, the incentive gradient in ML publishing shifts — quietly, and for the better.

Sources: Hugging Face blog — August 13, 2026; MetaTalks — August 2026; How AI Works — August 2026.