Partly succeededReplication study
LLM reanalysis of 180 social and behavioural science studies
Also partly succeeded
- Reproducing 48 social-science papers from the methods description alone23 Apr 2026
- ReplicatorBench: agents replicating social and behavioural science claims with new data11 Feb 2026
- Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini6 Feb 2026
- GPT-4o agent team reproducing five Alzheimer's disease studies29 May 2025
- PaperBench: agents replicating 20 ICML 2024 papers from scratch2 Apr 2025
What was attempted
An agentic pipeline received each study's original dataset, one focal statistical claim and, in the main condition, the relevant text of the article. It then wrote and ran its own analysis code, five runs per study, and the result was converted to a Cohen's d. The corpus was 180 published studies with predefined claims. For a subset the authors also compared the LLM with human reanalyses from the Multi100 collaboration. Three information conditions were tested: full paper, paper without methods, and abstract only.
What happened
- For 11 of the 180 studies the pipeline could not produce a usable effect size.
- For the rest, the LLM reached the same qualitative conclusion as the original in 80% of cases, and recovered the original effect size within ±0.05 in Cohen's d in 24%.
- In the subset with human reanalyses: same conclusion 95% for the LLM against 83% for human analysts; effect size within ±0.05 in 40% against 28%.
- Information condition barely mattered: within ±0.05 in 24% (full paper), 24% (no methods) and 22% (abstract only); within ±0.20 in 50%, 52% and 46%.
- GPT-5.5 and GLM-5.1 were run as robustness checks on the same task.
Why it failed or succeeded, per the authors
- The authors say the human comparison is descriptive: human analysts may have chosen their own defensible specifications, so higher LLM agreement could reflect a narrower, paper-anchored path rather than greater accuracy.
- Different defensible analytic choices can move effect sizes, in both humans and the LLM.
- They conclude "LLMs can support systematic audits of empirical results rather than substitute expert judgment".
Checks used
- Comparison against the original effect size at two tolerances (±0.05 and ±0.20) and on qualitative verdict (supports, opposite, inconclusive).
- Five independent runs per study; three information conditions; two additional models.
- Human reanalysis effect sizes from Multi100 for a subset.
Limits of this case
The study is a reproduction from original data with the claim handed to the model, so success is easier than a blind replication; and the strict tolerance rate of 24% shows exact recovery of effect sizes is rare. Only part of the supplement was read.