# Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini

- Outcome: Partly succeeded
- By: Scott Cunningham and colleagues
- Field: political communication / computational text analysis
- Published: 2026-02-06
- Source: https://causalinf.substack.com/p/claude-code-15-the-results-are-in

## What was attempted

Cunningham asked Claude Code to replicate the text-classification part of Card et al.'s PNAS paper on 140 years of immigration rhetoric in congressional and presidential speeches. The original used a fine-tuned RoBERTa classifier. He replaced it with zero-shot labels from gpt-4o-mini through the OpenAI Batch API. Claude Code handled the infrastructure, found and downloaded the original replication data, and ran the pipeline over about 305,000 speech segments. The post is part two of a pair; part one described the setup.

## What happened

- The LLM labels agreed with the original classifier's labels in 69% of cases.
- The headline substantive findings held: Democrats and Republicans diverge sharply from the 1970s onward under both classifiers, and the country-of-origin ordering (Italy, China, Mexico) held.
- Direct polarity flips between pro and anti were about 4%; where classifiers disagreed the LLM mostly moved speeches to neutral (33% of pro and 44% of anti speeches reclassified as neutral).
- Run cost $10.99 and about 4.6 hours in total (2.6 hours of batch runtime, against 24 hours he had expected).
- He reports a lapse in his own protocol: a "Referee 2" review was run inside the same session, and a reader caught it.

## Why it failed or succeeded, per the author

- Cunningham reads the move toward neutral as the LLM being cautious on ambiguous speeches.
- He argues the 69% figure should be read against modest agreement among human coders (Krippendorff's alpha 0.48).

## Checks used

- Transition matrix between original and LLM labels, and comparison of the substantive conclusions across both classifiers.
- Human annotator agreement as a benchmark.
- No separate validation step is described in the excerpt read.

## Limits of this case

The Substack page was paywalled after the summary, so the first-hand text read covers only the date, the title, the 69% agreement, the cost and the time; the other figures come from a third-party summary of the post (a wiki, velikov-mihail.github.io/ai-econ-wiki). It replicates a classifier-based result, not a full reproduction of the paper. A blog report, not peer reviewed.
