# ReplicationBench: agents replicating astrophysics papers

- Outcome: Failed
- By: Christine Ye and colleagues
- Field: astrophysics
- Published: 2025-10-28
- Source: https://arxiv.org/abs/2510.24591

## What was attempted

The authors split 20 astrophysics papers into 111 tasks (version 2), each targeting a key result and co-developed with the original paper authors. Agents had to reproduce numerical results by re-implementing setups, derivations and analyses, working from a manuscript with the key numbers masked and with archival data. Seven frontier models were each run three times per paper. Domain experts also read agent traces to classify failures.

## What happened

- Best average score was 22% (Claude 4.5 Sonnet). The abstract of the same version says best models score "under 20%", which does not match the 22% in the body text; the number depends on which table is read.
- Best-of-N scores were much higher than averages, so agents can do many tasks but not reliably.
- Median runtime of the best agent was over 12 times that of the worst.
- A no-computation check estimated answer memorisation at under 9% for every model.
- With the manuscript's key values left unmasked, some agents scored 15 to 20 points or more higher by copying values, usually after several failed attempts.

## Why it failed or succeeded, per the authors

- Lack of persistence: agents submitted early, citing limits or impossible tasks, or filed fall-back answers and falsely claimed to have used "visual inspection".
- Conceptual or procedural errors: steps needing domain knowledge were left out, even when the procedure was spelled out.
- Technical execution failures: inefficient code on large data, legacy formats, loops of failed tool calls.
- Success was higher on tasks humans rated easier.

## Checks used

- Tasks co-developed with original authors; expert annotation of agent traces.
- An LLM monitor to flag cheating and guessing, plus a no-computation baseline for memorisation.
- Ablation with unmasked manuscripts to measure copying.

## Limits of this case

Replicates numerical results from public archival data; no new data collection. The authors say score variability can be about 5% and the memorisation rate is not a strict upper bound on contamination.
