# Towards end-to-end automation of AI research

- Authors: Chris Lu, Cong Lu, Robert Tjarko Lange, Yutaro Yamada, Shengran Hu, Jakob Foerster, David Ha, Jeff Clune
- Venue: Nature 651, 914–919
- Published: 2026-03-25 (online); issue 2026-03-26
- Source: https://doi.org/10.1038/s41586-026-10265-5

# The AI Scientist (Lu et al., *Nature*, 2026)

## What the paper does

A team from Sakana AI, the University of British Columbia, the Vector Institute and Oxford (FLAIR) describes **The AI Scientist**, a pipeline of language-model agents that proposes a machine-learning research idea, checks it against the literature (Semantic Scholar), writes and runs the experiments, plots the results, writes a LaTeX paper and then has it reviewed by a second component, **The Automated Reviewer**. It only does computational experiments in machine learning. There are two modes: a template-based one that extends human-supplied starter code, and a template-free one that writes its own code through a parallel tree search (Methods). The template-free version used OpenAI o3 for ideas and code critique, Claude Sonnet 4 for code, GPT-4o for figures and best-first node selection, and o4-mini for review. A full template-free run takes "from several hours to over 15 h".

The headline test (section "Human evaluation results"): three generated papers were submitted, blind, to the ICLR 2025 "I Can't Believe It's Not Better" workshop, with the organisers' consent and ethics approval. One of the three scored 6, 7 and 6 (mean 6.33), above the workshop's average acceptance threshold; it was withdrawn by pre-agreed protocol. The other two did not meet the bar.

## Results worth quoting

**How results were checked, and by whom**

- *Workshop reviewers (external, human).* 43 papers were reviewed; the three AI submissions were among them. Reviewers were told some submissions were AI-generated but not which, and could opt out. The accepted-level paper "ranked among the top 45%" of submitted papers (Fig. 2 caption). The workshop accepted 70% of submissions, against 32% for the ICLR 2025 main conference (Limitations).
- *The authors' own team (internal, human).* They concluded one paper met the workshop bar and none met the main-conference bar (Supplementary section C.2, as summarised in the main text).
- *The Automated Reviewer (internal, automated).* An ensemble of five o4-mini reviews plus a meta-review in which the model takes the role of area chair (Methods). It was checked against ICLR decisions from OpenReview: balanced accuracy 0.69 ± 0.04 on 2017–2024 papers and 0.66 ± 0.03 on 2025 papers published after the model's knowledge cutoff, against 0.66 for human reviewers in the NeurIPS 2021 consistency study (Table 1). The authors note the paper pools differ, so the comparison "is not exact".
- *Reading Table 1 more closely.* The automated reviewer's accuracy (0.65, 0.63) is below the humans' (0.73), and its false positive rate is high: 0.45 (before cutoff) and 0.52 (after) against 0.17 for humans, with a false negative rate of 0.17 against 0.52. It rejects too few bad papers. The text reports it as "comparable" on balanced accuracy and F1 and does not discuss this.
- *Scaling claims.* Paper quality scored by the Automated Reviewer rises with model release date (Fig. 1b; R = 0.517; n = 6 points per template-free model, n = 3 template-based) and with search budget (Fig. 3c). These are scores from the system's own reviewer, not from external reviewers.

**How many checked versus produced.** Three papers were checked externally. The text says the team "manually filtered the most promising outputs at each stage" and that three were selected on topic fit, whether the code "correctly implemented the proposed idea and ran without errors", and formatting. The number generated before selection is not in the main text (Supplementary section A.4; not read). One in three selected papers passed, so the pass rate over everything generated is unknown but lower.

**What the humans did.** They chose the workshop and prompted the system with its theme; filtered outputs at each stage; picked the three submissions; and, per the author contributions, three authors (R.T.L., Y.Y., S.H.) read and validated AI-generated papers to select the submissions and checked the code implementations behind them. They also ran the organiser communication, the ethics approval and the pre-planned withdrawal. The paper states that each submitted paper's workflow was "performed without any human modification", which is true of the text and code but not of the selection and checking around them.

## The authors' own caveats

- "Only one of three submissions was accepted"; the system "cannot yet meet the standards of top-tier publications nor even do so consistently for workshops".
- Failure modes they list: naive or underdeveloped ideas, incorrect implementations of the main idea, lack of methodological rigour, errors in experimental implementation, duplicated figures in main text and appendix, and "many types of hallucinations, such as inaccurate citations". Rates are in the Supplement, not the main text.
- The Automated Reviewer may be contaminated by training data: balanced accuracy fell from 69% to 66% after the cutoff, which the authors read as, at most, a minimal effect.
- Persistent problems named: systems being easily fooled and overconfidently wrong, which "would prevent us from reliably trusting the outputs of systems like The AI Scientist".
- Risks named: overwhelming peer review, inflating credentials, using others' ideas without credit, adding noise to the literature.
- Reproducibility: code for both versions is public (Apache 2.0). Fig. 1b and Fig. 3c give replicate counts. The three submitted papers and trajectories sit in the Supplement. Competing interests: most authors are Sakana AI employees or consultants; the principal investigator also has Google DeepMind and Vector Institute affiliations.

## Ideas that travel beyond machine learning

1. **Output must be checkable (supports, with a gap).** The reviewers and the Automated Reviewer assess the manuscript (the Automated Reviewer processes the PDF). The paper is only as checkable as its link to the code. The authors themselves read the code of candidate papers before submitting; that step is what made "the code correctly implemented the idea" a checked fact rather than a claim, and it was done by people.
2. **External checks the agent cannot alter (complicated).** The workshop review is a genuine external check, with a pre-registered withdrawal so the result could not be used to publish. But it covered three hand-selected papers. The check used to compare configurations and claim improvement over time (Fig. 1b, 3c) is the system's own Automated Reviewer, built by the same team, with a high false positive rate. Inside the pipeline, the status "non-buggy" means the code ran and a vision model found no problem with the plots; the authors still report incorrectly implemented ideas.
3. **"Done" and "verified" as claims (supports).** The write-up is generated from the agent's notes and the system "automatically corrects any compilation errors". A paper that compiles, cites and has plots is not shown to be correct. The paper's own failure list (duplicated figures, invented citations) is a list of confident outputs that were wrong.
4. **Silent omission (supports).** Selection of three from an unreported number, plus the stage-by-stage filtering, is invisible in the headline "passed peer review" result. The authors argue the papers "would still have been produced" without filtering; that is about cost, not about the pass rate.
5. **Councils and multiple models (supports, with a caveat).** Different model families fill different roles (o3, Sonnet 4, GPT-4o, o4-mini). The five-review ensemble and area-chair meta-review use one model, so the five reviews share its biases; the paper treats the ensemble as improving robustness and does not test independence.
6. **Disclosure (supports).** The study had consent of the venue and ethics approval, told reviewers that some papers were AI-generated, and withdrew all submissions "to avoid setting a precedent for publishing fully automated research before the scientific community has established clear standards for disclosure and evaluation". The reviewers knew the share but not which papers, and the co-authors who chose the papers were not blind.

## Limits for our purposes

Machine learning only, with experiments that run entirely on a computer and can be re-run cheaply; the reproducibility of a research claim is therefore far easier here than in most fields. The central result is one paper in one workshop at a 70% acceptance rate, so it is an existence demonstration, not a rate. Several of the checks that would matter most to a sceptical reader (failure-mode frequencies, how many papers were produced, the selection protocol) are in the Supplement, which this note does not cover. The scaling claims rest on the system's own reviewer.
