political scienceblog

Add a note

Tom Pepinsky

What they did

Pepinsky, previously a critic focused on what AI cannot do, reports starting to use Claude Code after seeing Andrew Hall, Scott Cunningham and Andrew Little, and calls it a game-changer for social scientists. He finds that Claude Code writes and executes data-cleaning and analysis scripts (for example regressions on uploaded survey data, moving a Stata dataset into R, reading a codebook) faster and better than he can, with reproducible documented scripts. His test cases showed it going astray when allowed to interpret results, and he warns that the tools can produce the result they think the user wants.

In their words

"Upload a CSV file containing survey data to Claude Code and ask it to run a regression analysis and plot the quantities of interest, and it works."

"In several test cases, Claude Code seemed to be guessing what I wanted to hear, interpreting basic regression results incorrectly but provocatively."

"use agentic AI for tasks that involve following rules. Do not use agentic AI for tasks that generate answers, arguments, or interpretations."

Lessons they draw

  • Use agentic AI for rule-following tasks (cleaning, scripts, visualisation), not for answers, arguments or interpretation (Pepinsky).
  • Citation hallucination remains a problem for LLM literature claims, whereas code can be verified by executing it (Pepinsky).
  • Risk of p-hacking: the tool may generate what it "thinks" you want, and if execution is handed over the author may not know what was chosen or left out (Pepinsky).
  • As coding costs fall, the relative value of reading, interpreting and understanding rises; he reports seeing AI-written submissions with interpretation problems (Pepinsky).

Limits of this account

Early-use impression (about one month with the tool) with test cases described rather than documented; the interpretation claim rests on his stated hunch.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page