Partly succeededReplication study

Add a note

Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini

What was attempted

Cunningham asked Claude Code to replicate the text-classification part of Card et al.'s PNAS paper on 140 years of immigration rhetoric in congressional and presidential speeches. The original used a fine-tuned RoBERTa classifier. He replaced it with zero-shot labels from gpt-4o-mini through the OpenAI Batch API. Claude Code handled the infrastructure, found and downloaded the original replication data, and ran the pipeline over about 305,000 speech segments. The post is part two of a pair; part one described the setup.

What happened

  • The LLM labels agreed with the original classifier's labels in 69% of cases.
  • The headline substantive findings held: Democrats and Republicans diverge sharply from the 1970s onward under both classifiers, and the country-of-origin ordering (Italy, China, Mexico) held.
  • Direct polarity flips between pro and anti were about 4%; where classifiers disagreed the LLM mostly moved speeches to neutral (33% of pro and 44% of anti speeches reclassified as neutral).
  • Run cost $10.99 and about 4.6 hours in total (2.6 hours of batch runtime, against 24 hours he had expected).
  • He reports a lapse in his own protocol: a "Referee 2" review was run inside the same session, and a reader caught it.

Why it failed or succeeded, per the author

  • Cunningham reads the move toward neutral as the LLM being cautious on ambiguous speeches.
  • He argues the 69% figure should be read against modest agreement among human coders (Krippendorff's alpha 0.48).

Checks used

  • Transition matrix between original and LLM labels, and comparison of the substantive conclusions across both classifiers.
  • Human annotator agreement as a benchmark.
  • No separate validation step is described in the excerpt read.

Limits of this case

The Substack page was paywalled after the summary, so the first-hand text read covers only the date, the title, the 69% agreement, the cost and the time; the other figures come from a third-party summary of the post (a wiki, velikov-mihail.github.io/ai-econ-wiki). It replicates a classifier-based result, not a full reproduction of the paper. A blog report, not peer reviewed.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page