SucceededReplication study

Add a note

AI-assisted full-paper replication of 384 political science studies

What was attempted

The authors built a workflow that retrieves a paper's replication materials, rebuilds the computing environment, runs the code, and matches outputs to point estimates in the regression tables. They defined all empirical papers in three top political science journals from 2010 to 2025 and drew a stratified sample of 384 studies (8 per journal-year), covering 3,523 models. A second application re-ran 84 instrumental-variable studies and extracted 597 IV specifications for diagnostics. The numbers below are from version 3 of the preprint (31 May 2026); earlier versions reported different figures.

What happened

  • Fully or largely reproducible: 82.5% of papers after DA-RT adoption against 20.8% before; given an accessible replication package, 92.1% (234 of 254).
  • Of the 254 with accessible materials, 193 (76.0%) were fully reproducible, meaning every reported coefficient matched exactly, and 41 largely reproducible (more than 80% of coefficients matched).
  • 191 papers were not fully reproducible; 130 of them lacked a package or had restricted data or code.
  • Among the 61 papers with accessible materials below the strict threshold, 10 were partially and 10 not reproducible.
  • IV corpus: all main-text regressions replicated in each of 84 papers, 1,910 models.

Why it failed or succeeded, per the authors

  • "Failures trace almost entirely to missing or restricted data, not to code quality."
  • Repeated issues in package structure were encoded as rules and version-controlled across runs, so coverage improved over versions.
  • Journal verification requirements plus data-archiving mandates drive reproducibility, not the AI tool.

Checks used

  • Coefficient-level matching against the published tables, with four verdict categories.
  • Human review of failures; benchmark verification against a prior IV corpus (Lal et al. 2024).
  • One IV paper dropped because table extraction produced no coefficients.

Limits of this case

It re-executes the authors' own code on their data, which is easier than re-implementing; it cannot rescue missing packages. Strong-sharing journals only. Read the abstract, introduction, results and supplementary tables, not every appendix.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page