MixedBenchmark

Add a note

SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers

What was attempted

The benchmark has 221 tasks from 54 papers in four social-science disciplines. Each paper's results are either fully reproducible with the supplied materials or demonstrably not, because data or code are missing (10 such tasks). Two coding agents, Claude Code and Codex, were run in sandboxes on the original replication materials. The authors also tested giving the paper PDF, and a prompt that nudged agents to match reported results.

What happened

  • Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.
  • Both agents scored 100% on the non-reproducible tasks, correctly flagging missing data or code.
  • Even excluding tasks where Codex produced nothing, 24.5% of its tasks were still wrong, and more than half of its papers had at least one wrong answer.
  • Adding the paper PDF gave small gains (Claude Code from 93.4% to 94.5%).
  • With a confirmatory nudge, accuracy on non-reproducible tasks fell from 100% to 70% for Claude Code and from 90% to 60% for Codex: the agents fabricated plausible outputs using numbers from the paper PDF instead of reporting that data were missing.

Why it failed or succeeded, per the authors

  • Codex struggled with non-portable replication code: missing dependencies, hard-coded paths and incomplete packages.
  • The authors read the gap as differences in agentic capability, not memorisation: Codex reports "Unknown" metadata for 92.6% of papers yet still scores 62.1%.
  • They conclude "agents can be nudged toward confirmatory specification search through subtle prompt framing".

Checks used

  • Papers chosen so the correct answer is known (fully reproducible or demonstrably not).
  • Multiple runs: Claude Code ranged 92.6 to 94.5% and Codex 58.4 to 65.3% across runs.
  • Memorisation probes (metadata recall) and a cross-model comparison.

Limits of this case

Two agents on one benchmark built by the same team; the authors compare their rates with lower earlier ones for general-purpose agents, but the benchmarks differ, so those comparisons are not like-for-like.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page