# LLM workflow run over 4,452 economics replication packages

- Outcome: Mixed
- By: Matthew Schwartz and colleagues
- Field: economics
- Published: 2026-09-01
- Source: https://www.nber.org/papers/w35782

## What was attempted

The authors describe an open-source workflow that gives an LLM a published article's replication package and asks it to do three things: reproduce the original calculations and flag discrepancies from the published findings, run automated sensitivity analyses, improve the calculations with a different implementation or algorithm, and develop extensions that fit the article's assumptions but are not in it. It was run across 4,452 replication packages from five economics journals. The date above is the NBER issue month (September 2026), with the day set to the first.

## What happened

- Discrepancies were flagged in 3,460 articles or their appendices.
- Computation time was cut by more than a factor of ten in 496 articles, with similar or greater accuracy.
- Extensions were developed for 923 articles.
- The NBER page does not report how many packages reproduced cleanly.

## Why it failed or succeeded, per the authors

Not stated on the page read; no failure analysis was available in the abstract. The page states that the LLMs were used in the analysis and writing, and that one author worked as a contractor for Anthropic.

## Checks used

The abstract does not say whether flagged discrepancies were audited by a human. The working paper itself was not read.

## Limits of this case

Abstract-level record only. A flagged discrepancy is not a confirmed error. Without the full paper it is unclear how many flags are real, how many reflect missing materials, and which model was used. Treat the counts as unverified until the paper is read.
