MixedReplication game
Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
- Analytic flexibility in silicon samples: re-examining Argyle et al. Study 316 Sep 2025
What was attempted
In randomised replication games, 288 researchers formed 103 teams in three arms: human-only, AI-assisted (ChatGPT as a collaborator) and AI-led (ChatGPT drives, with minimal human oversight). Each team had seven hours to computationally reproduce a few pre-selected results from a published paper, find coding errors, and propose robustness checks. Twelve studies were used across events. Some papers contained coding errors the organisers had found earlier and not published. The published date above is the compile date of the accepted PNAS manuscript, not the journal issue date.
What happened
- Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).
- Major coding errors found per team: human-only 1.70, AI-assisted 0.74, AI-led 0.23.
- Minor errors: 1.00, 1.40 and 0.69; differences were not statistically significant.
- AI-led teams proposed weaker robustness checks, six supplied none rated as good, and took longer.
Why it failed or succeeded, per the authors
- Participants in AI-led teams reported prompt fatigue, hallucinated reasoning paths, truncated context windows and long debugging loops.
- AI-assisted teams outsourced routine tasks such as boilerplate code and file paths, and kept conceptual control.
- The authors say the AI-led arm is probably a lower bound, since better prompting, newer models and supervision could raise it.
Checks used
- Randomised assignment to arms, with a one-hour ChatGPT training for the AI arms.
- Reproduction counted only if the team reproduced the pre-selected results.
- Transcripts of ChatGPT use retained, plus focus-group material for explanation.
- The authors conclude that "expert human judgment currently remains indispensable for reliable empirical verification".
Limits of this case
Only ChatGPT, only seven hours, and a small non-random set of twelve studies. The models used were 2024-era GPT-4 and GPT-4o; later agentic tools do much better on similar tasks (see socsci-repro-bench.md).