MixedReplication study
LLMs as simulated participants: replicating 156 psychology and management experiments
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Auditing three Kosmos radiation-biology claims against null models17 Nov 2025
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
What was attempted
The authors reran 156 scenario-based experiments from top social-science journals, 2015 to 2024, with LLMs playing the participants. Each model saw only one experimental condition at a time, so it was blind to the design and hypothesis. The 156 studies (50 articles from five management and psychology journals, randomly selected) gave roughly 620 to 680 main effects and 130 to 165 interaction effects, depending on the model (DeepSeek could not process images, so its set was smaller). A replication counted if the LLM sample preserved the direction and statistical significance of the original effect.
What happened
- Main effects replicated at 72.7% (GPT-4), 80.6% (Claude) and 76.2% (DeepSeek); interaction effects at 45.7%, 61.7% and 62.9%.
- LLM effect sizes were much larger than human ones, with Fisher Z values about 2 to 3 times higher.
- Where original studies reported null findings, LLMs produced significant results 68 to 83% of the time.
- Replication rates were lower for race-related effects (GPT-4: 41.5% against 76.8%) and gender-related effects (57.8% against 74.3%). Claude showed no significant difference for race or gender.
- 35.3% of studies needed prompt adaptations to keep the models engaged with the manipulation.
Why it failed or succeeded, per the authors
- LLMs may produce cleaner data with less noise, which narrows confidence intervals but risks overestimating effects.
- Lower rates for socially sensitive topics suggest models do not capture culturally sensitive social reasoning.
- The authors say "LLMs show significantly lower replication rates for studies involving socially sensitive topics such as race, gender and ethics".
Checks used
- Rate of direction and significance preserved against the original published statistics.
- Comparison across three LLMs and across study features (journal, sample, platform, publication year, citations).
- A contamination argument: the papers were behind subscription journals, a preliminary test showed GPT-4 could not describe them accurately, and publication-year and openness effects were non-significant.
Limits of this case
"Replication" here means an LLM-simulated sample reproducing a human study's effect, not an agent re-running a study. It does not test what the effect would be on real people. The count of studies differs across versions and secondary sources (154 or 156 appear), and this record follows the version read.