Partly succeededReplication study
Reproducing 48 social-science papers from the methods description alone
Also partly succeeded
- LLM reanalysis of 180 social and behavioural science studies11 Jun 2026
- ReplicatorBench: agents replicating social and behavioural science claims with new data11 Feb 2026
- Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini6 Feb 2026
- GPT-4o agent team reproducing five Alzheimer's disease studies29 May 2025
- PaperBench: agents replicating 20 ICML 2024 papers from scratch2 Apr 2025
What was attempted
The authors asked whether agents can reproduce results without the original code. A system extracts a structured description of each paper's methods, masks the reported numbers, and gives an agent only that description and the original data. Agents never see the original code, results or paper. Four agent scaffolds and four LLMs were run on 48 papers that the Institute for Replication had verified as reproducible. Outputs were compared cell by cell with the originals, and an LLM traced each discrepancy to a root cause.
What happened
- Three best agents matched the sign of the original coefficient about 85 to 91% of the time.
- The best agent (GPT-5.4 on OpenCode) was within 1.96 original standard errors over 80% of the time; it beat Claude Code (Opus 4.6), GPT-5.3 Codex and GLM-5.
- Scaffold mattered: GPT-5.4 on OpenCode beat GPT-5.4 on Codex CLI and on mini-SWE-Agent, with much greater token use.
- Root-cause labels: most divergences were attributed to paper under-specification, fewer to agent errors, with the share of agent errors lower for stronger agents.
Why it failed or succeeded, per the authors
- Errors come mostly from papers under-specifying their methods and, to a lesser extent, from agents misreading or ignoring the method description. The abstract says "agents can largely recover published results".
- Better results came with more token use and longer analysis runs.
Checks used
- Strict information isolation between the agent and the original code and results.
- Deterministic, cell-level comparison against the published tables.
- A human audit of the LLM root-cause step: of 53 flagged divergences, 49 confirmed as meaningful (92%), but only 33 of 49 error-source labels confirmed (67%), 10 unsure. A different LLM judge changed the agent-error share.
Limits of this case
Papers were pre-selected as verified-reproducible and the data were given, so this tests re-implementation, not discovery of data. Root-cause shares come from an LLM judge and are indicative only, as the authors say.