Partly succeededBenchmark
ReplicatorBench: agents replicating social and behavioural science claims with new data
Also partly succeeded
- LLM reanalysis of 180 social and behavioural science studies11 Jun 2026
- Reproducing 48 social-science papers from the methods description alone23 Apr 2026
- Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini6 Feb 2026
- GPT-4o agent team reproducing five Alzheimer's disease studies29 May 2025
- PaperBench: agents replicating 20 ICML 2024 papers from scratch2 Apr 2025
What was attempted
Unlike code-rerun benchmarks, this one asks agents to replicate a claim: find and retrieve new data, design the analysis, run it, and judge whether the claim replicates. It has 19 observational studies from the Center for Open Science's SCORE project, including human-verified non-replicable ones. Agents are scored on about 1,568 rubric checkpoints across extraction and data retrieval, design and execution, and interpretation. The authors tested their ReplicatorAgent with four LLMs and varied the programming language and level of code access.
What happened
- Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).
- Data retrieval was the weak stage: macro F1 about 8 to 19 across settings, and the agent found all of a study's data sources (hit@all) in only 5 to 16% of studies.
- GPT-5 and o3 had far fewer input-data errors than GPT-4o in execution (6 against 211 for GPT-5 and GPT-4o).
Why it failed or succeeded, per the authors
- Agents "struggle with retrieving new data, necessary to replicate a claim", while designing and running analyses went well.
- GPT-5-mini failed more on syntax and library use than on data handling.
- Stronger models also put more effort into execution than design, which the authors do not count as a failure.
Checks used
- Ground truth from human-verified replicable and non-replicable outcomes, with listed data-source URLs for the retrieval task.
- LLM-based scoring of stages (LLMEval), with 95% confidence intervals across runs; failures counted by category from run logs.
Limits of this case
Only 19 studies, all drawing on web-retrievable data; the authors note that primary data collection is out of scope. Read via the main text and tables, not every appendix.