# Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams

- Outcome: Mixed
- By: Abel Brodeur and colleagues
- Field: economics and political science (quantitative social science)
- Published: 2026-03-26
- Source: https://repository.essex.ac.uk/43026/

## What was attempted

In randomised replication games, 288 researchers formed 103 teams in three arms: human-only, AI-assisted (ChatGPT as a collaborator) and AI-led (ChatGPT drives, with minimal human oversight). Each team had seven hours to computationally reproduce a few pre-selected results from a published paper, find coding errors, and propose robustness checks. Twelve studies were used across events. Some papers contained coding errors the organisers had found earlier and not published. The published date above is the compile date of the accepted PNAS manuscript, not the journal issue date.

## What happened

- Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).
- Major coding errors found per team: human-only 1.70, AI-assisted 0.74, AI-led 0.23.
- Minor errors: 1.00, 1.40 and 0.69; differences were not statistically significant.
- AI-led teams proposed weaker robustness checks, six supplied none rated as good, and took longer.

## Why it failed or succeeded, per the authors

- Participants in AI-led teams reported prompt fatigue, hallucinated reasoning paths, truncated context windows and long debugging loops.
- AI-assisted teams outsourced routine tasks such as boilerplate code and file paths, and kept conceptual control.
- The authors say the AI-led arm is probably a lower bound, since better prompting, newer models and supervision could raise it.

## Checks used

- Randomised assignment to arms, with a one-hour ChatGPT training for the AI arms.
- Reproduction counted only if the team reproduced the pre-selected results.
- Transcripts of ChatGPT use retained, plus focus-group material for explanation.
- The authors conclude that "expert human judgment currently remains indispensable for reliable empirical verification".

## Limits of this case

Only ChatGPT, only seven hours, and a small non-random set of twelve studies. The models used were 2024-era GPT-4 and GPT-4o; later agentic tools do much better on similar tasks (see `socsci-repro-bench.md`).
