FailedBenchmark

Add a note

REPRO-Bench: agents assessing reproducibility of 112 social-science papers

What was attempted

Agents were given a social-science paper (PDF) and its reproduction package, and asked to rate its reproducibility from 1 (not reproducible) to 4 (fully reproducible). Ground truth was taken from published reproduction reports by social scientists: 112 task instances across Stata, R and Python. The task is to execute the package and check consistency with the paper's claims, not only to rerun the code.

What happened

  • Best existing agent (CORE-Agent with GPT-4o): 21.4% accuracy, below the 25% expected by guessing among four scores.
  • The authors' REPRO-Agent, with a four-phase template, a fallback score and few-shot examples of common errors, reached 36.6%, a 71% relative gain.
  • The authors report that the three agents did about as well on packages with several data formats (52% average accuracy) as on single-format packages (54%).

Why it failed or succeeded, per the authors

  • Four failure types: (1) wrong comparison of reproduced and reported results; (2) Stata scripts whose errors go to log files, not the terminal, so the agent misreads silence; (3) failed library installation; (4) files not in the execution directory, the most frequent type.
  • Agents often concluded data were missing without searching the package.
  • Reading whole code files instead of relevant sections made inconsistency detection hard.

Checks used

  • Human reproduction reports as ground truth, annotated by the authors in a consensus process.
  • Accuracy measured on the 1–4 score.

Limits of this case

All agents used one 2024 model (GPT-4o); the later SocSci-Repro-Bench and Kohler et al. results (see socsci-repro-bench.md, kohler-read-paper-write-code.md) show much higher rates with newer coding agents. The abstract reports an "accuracy of only 21.4%".

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page