MixedAudit of an AI system

Add a note

Independent evaluation of Sakana's AI Scientist

What was attempted

The authors installed the open-source AI Scientist and ran it end to end on recommender-systems research ideas, to test Sakana's claims that it automates the full research lifecycle. They supplied two seed ideas, generated further ideas, ran the experiments and read the resulting manuscripts. This is a test of a system's claims rather than of one published paper. The Zotero title is the first version's title; the arXiv v3 read here has a different title.

What happened

  • Five of twelve proposed experiments (42%) failed to run because of coding errors; the system flagged 10 generated ideas and both seeds as novel, including well-established ones such as micro-batching for SGD.
  • Experiments that ran often gave logically flawed results; in one, an energy-efficiency experiment reported accuracy gains while using more compute.
  • Code edits were small, about 8% more characters in the first iteration, shrinking in later ones.
  • Seven manuscripts were produced; median five citations each, five of 34 citations from 2020 or later, four of seven manuscripts with missing or misplaced figures or placeholder text, and four of seven with incorrect or hallucinated numbers.
  • Cost was roughly $6 per manuscript in API charges.

Why it failed or succeeded, per the authors

  • Novelty checking relies on keyword searches over Semantic Scholar, not synthesis of the literature.
  • The system depends on the supplied templates and cannot critically assess its own results.
  • The authors conclude it is a notable advance in producing complete manuscripts, with quality like that of a rushed undergraduate.

Checks used

  • Manual reading of every generated manuscript and experiment log.
  • Comparison against Sakana's stated claims.
  • A containerised reproduction package published so others can rerun the evaluation.

Limits of this case

Small sample (twelve experiments, seven manuscripts) in one subfield, on one version of the tool. The headline claim, "five out of twelve proposed experiments (42%) failed due to coding errors", comes from that sample.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page