# LLM reanalysis of 180 social and behavioural science studies

- Outcome: Partly succeeded
- By: Tobias Holtdirk and colleagues
- Field: psychology, political science, economics
- Published: 2026-06-11
- Source: https://arxiv.org/abs/2606.13670

## What was attempted

An agentic pipeline received each study's original dataset, one focal statistical claim and, in the main condition, the relevant text of the article. It then wrote and ran its own analysis code, five runs per study, and the result was converted to a Cohen's d. The corpus was 180 published studies with predefined claims. For a subset the authors also compared the LLM with human reanalyses from the Multi100 collaboration. Three information conditions were tested: full paper, paper without methods, and abstract only.

## What happened

- For 11 of the 180 studies the pipeline could not produce a usable effect size.
- For the rest, the LLM reached the same qualitative conclusion as the original in 80% of cases, and recovered the original effect size within ±0.05 in Cohen's d in 24%.
- In the subset with human reanalyses: same conclusion 95% for the LLM against 83% for human analysts; effect size within ±0.05 in 40% against 28%.
- Information condition barely mattered: within ±0.05 in 24% (full paper), 24% (no methods) and 22% (abstract only); within ±0.20 in 50%, 52% and 46%.
- GPT-5.5 and GLM-5.1 were run as robustness checks on the same task.

## Why it failed or succeeded, per the authors

- The authors say the human comparison is descriptive: human analysts may have chosen their own defensible specifications, so higher LLM agreement could reflect a narrower, paper-anchored path rather than greater accuracy.
- Different defensible analytic choices can move effect sizes, in both humans and the LLM.
- They conclude "LLMs can support systematic audits of empirical results rather than substitute expert judgment".

## Checks used

- Comparison against the original effect size at two tolerances (±0.05 and ±0.20) and on qualitative verdict (supports, opposite, inconclusive).
- Five independent runs per study; three information conditions; two additional models.
- Human reanalysis effect sizes from Multi100 for a subset.

## Limits of this case

The study is a reproduction from original data with the claim handed to the model, so success is easier than a blind replication; and the strict tolerance rate of 24% shows exact recovery of effect sizes is rare. Only part of the supplement was read.
