Researchers using AI agents

Progress in using AI agents to empower researchers is rapid but still very unevenly distributed. Many researchers have been reporting great benefits while others are still complaining about ChatGPT hallucinating references. This is a collection of public accounts by researchers who shared their usage and lessons learned. It was compiled by AI agents at the direction of Dominik Lukeš who also provided some of his own use cases.

Recent examples

Lessons from June 2026 onward

Choose a question worth answering

Before asking an agent to tackle a research problem, ask who would care about the answer and what it would tell them. Revisit those questions when the first results come in. A calculation can become feasible long after specialists know what answer to expect.

Examples and sources

Matthew Schwartz used BootLoops to test how well neutral biodiversity theory explained changes in a Panamanian forest. The ecologist James O’Dwyer told him that ecologists already knew the theory could not keep up with real forests. He suggested subtracting the neutral prediction and using what remained to study differences between species. That changed the question they put to the data.

Robert Lipshitz chose a question about Khovanov local equivalence groups that interested him, involved algebra he could check line by line, and had seemed too marginal for a graduate student to spend time on. He still leaves open whether the new algebraic structure has consequences for knots. His account gives a reason to pursue a modest question when the work becomes feasible and a reason to be precise about what answering it would establish.

Matthew Schwartz (2026-10-01) · Robert Lipshitz (2026-10-27)

Build the method and its checks together

An agent can help build the tools a research method needs. Decide while building how you will tell whether the method works. For a new search, compare what it returns with existing searches. For a simulation you want to repeat, review the setup before reusing it on new inputs.

Examples and sources

Aaron Tay built a way for an agent to search a library catalogue repeatedly, changing the query when it got nowhere. On selected old searches that had returned zero results, he compared that loop with the catalogue’s single-pass query expansion. The loop recovered some searches by fixing spelling or broadening the query. Tay also found that he still needed to validate links in the final answer. His examples came from a selected set of catalogue failures.

Ke Xiao and colleagues used Codex to help prepare a package of simulation settings, meshing procedures and analysis scripts. They kept review records with the package, then reused it on 140 versions of an injector design. This is a report on one engineering study.

In an earlier project, Joshua Sternfeld compared his agent-built newspaper search with the collection’s own website. The agent had set a limit on how much of each page it searched, so the new tool missed matches that the website found.

Aaron Tay (2026-06-24) · Ke Xiao, Han Li, Teng Zhang, Yangchen Xu, Runze Mao, Zhi X. Chen (2026-07-15) · Joshua Sternfeld (2026-04-20)

Use a council of models

When I have a decision that matters, I give several models the same question and brief and ask them to answer separately. I compare their reasons, especially where they disagree, before deciding what to do.

Examples and sources

Chris Blattman describes how he uses separate agents to develop a plan and how he brings in fresh reviewers after giving his own judgement. Blattman’s council guide

John H Boyer’s instruction files for his Greek readers require work by a Claude agent to be reviewed by GPT or Grok, and work by Codex to be checked by Claude before it is committed. They record that Grok’s content checks twice found classes of defect that the other models had missed. This comes from his project’s instruction files.

Two papers describe limits. A primer on agentic AI systems for scientific discovery in Nature Reviews Methods Primers (Feng and colleagues, published 22 October 2026) lists multi-agent coordination among its failure modes: adding agents can make results worse. Feng and colleagues’ primer Lu and colleagues’ paper in Nature on automating AI research (March 2026) uses an automated reviewer built from five reviews by the same model plus a meta-review. Because one model wrote all five, they share its biases, and the paper does not test whether the reviews are independent.

Earlier in 2026, Matthew Schwartz supervised Claude Code through a quantum field theory calculation and published his account on 23 March 2026 as a guest post on Anthropic’s website. The project plan was built by merging outlines from three models. He reports that GPT and Claude caught each other’s errors, and that GPT solved the hardest integral, which Claude then incorporated. This is his own report of one project, hosted by the vendor.

Tiago Ventura’s slides for a faculty retreat in 2026 report that cross-checking with a second model improved a long instruction file that Claude Code had mostly written. He also reports running a tournament across models among the top candidate papers for a replication, on the grounds that different models have different evaluation biases. The slides are terse bullets, so there is little detail.

Try a council

Pick a decision about a research design or an interpretation. In fresh sessions, give each model the same question, evidence and criteria. Ask each to recommend an approach and explain what would change its recommendation. Keep the first answers separate. When you compare them, look for the assumptions or evidence that account for any disagreement. If you have access to one model, Blattman’s starter kit shows how to give separate runs different reviewing roles. Blattman’s starter kit

My agents do this through my model council skill. It asks for my approval first, gets separate answers from other models, and keeps the disagreements and the evidence behind any recommendation.

Models can make the same mistake. Check any claim that affects your decision against the source or data, even if the models agree. For a result you can test, start with a known result and keep an independent check.

Chris Blattman (2026-10) · John H Boyer (2026-11-20) · Matthew Schwartz (2026-03-23) · Tiago Ventura (2026)

Start with a known result and keep an independent check

Try the method first on a result you already know. Before moving on to new work, decide how you will check an answer when there is no published result to compare it with. Where possible, make that check independent of the code that produces the answer.

Examples and sources

Matthew Schwartz started the work that became BootLoops by asking Claude to implement methods from his papers and related work. He reports that it reproduced a result from one of his papers before he asked it to look for unsolved calculations. For the new calculations, the method let him compare exact formulas with high-precision numerical evaluations.

In Terence Tao’s formalisation of the proof of Sendov’s conjecture, Python code generated certificates and Lean checked them. Lean can reject a faulty certificate without relying on the generator to detect the fault. The project also tested the check itself by deliberately altering certificates. Part of that test turned out to be weaker than expected.

Francis DiTraglia used Stata’s ivreg2 as the reference for his R port. The R tests read results that Stata had exported to CSV and compared coefficients, covariance matrices and diagnostics. The expected results came from Stata, so the same checks could be rerun after changes to the R code. Validation guide

Matthew Schwartz (2026-10-01) · Terence Tao (2026-08-13) · Francis DiTraglia (2026-07-21)

Keep enough of the work to resume, audit and undo it

Keep the original material, the last usable output and a record of what failed. If a long run stops or a correction goes wrong, you need to know what you can reuse and what you have to do again. A saved decision can also keep a new session from repeating a failed approach.

Examples and sources

In my radio archive project, records with invalid source references were held back. The recovery run left out claims with invalid references and rebuilt summaries from the evidence that remained. We kept the original failures and saved editorial corrections separately from the model output.

Thomas D. Grant starts each project with a set of files that have separate jobs: one holds what was agreed, one the open questions, and one the sources found and what has been checked in each. His slides say each file answers a different question when the project resumes. He saves corrections to the model’s prose as rules with examples, and he split the rules file when it grew to 44 rules.

John H Boyer’s instruction files for his Greek readers keep a dated list of failures and what each taught. They also keep hand-over notes per track of work, after one session overwrote another’s notes in a shared file.

The write-up of Terence Tao’s Lean formalisation records dead ends, including times the agent inserted proof placeholders against the project’s rules. Chris Blattman saves corrections that should affect future work in the instructions or examples the agent will use next time. He also removes old rules when they start getting in the way.

Dominik Lukeš (2026-10-24) · Terence Tao (2026-08-13) · Chris Blattman (2026-10) · Thomas D. Grant (2026-11-20) · John H Boyer (2026-11-20)

Routine research work is a useful place to start

Try an agent on a recurring task whose output you can check against the project files, such as preparing for a meeting from past notes or flagging inconsistencies in a manuscript. A task you do often gives you a reason to spend time explaining it and a chance to see whether the next attempt takes less work to check.

Examples and sources

Chris Blattman uses notes and recent updates to prepare for research team meetings. He asks what needs deciding and what information is missing. Some of his recurring tasks took weeks to set up and initially took as long as doing them himself.

Robert Lipshitz uses Claude Code to check LaTeX papers for typos and mathematical inconsistencies and put the findings in a checklist. He sometimes repeats the same request because a second pass can find different things. The guide describes one mathematician’s working practice, including the checks he still has to do.

Chris Blattman (2026-10) · Robert Lipshitz (2026-07)

Guides, slides and tools

Workflows for research

Tools and worked examples

Workshop recordings and slides

How researchers used OpenAI’s tools

The accounts

41 accounts across 28 disciplines. Read or search the full accounts.

June 2026 onward

philosophy

John H Boyer

Claude Code (Fable as orchestrator with Opus and Sonnet subagents) Codex (GPT-5.6) Grok 4.5

Boyer works on Aristotle. He writes on his site that for years he wanted what no one site offers: the TLG has the Greek texts, search and parsing but no translations, Perseus has translations but no good way to read them beside the Greek, and the Loeb has…

law (international law)

Thomas D. Grant

Codex saved Markdown instruction files (AGENTS.md and others) a source wiki on Andrej Karpathy's pattern

Grant works on international law, and his slides show his work in Codex. The programme abstract says he has been experimenting with prompts, context management and harness-building for seven months across a range of projects. The slides list seven kinds of…

mathematics

Robert Lipshitz

Claude Code (Fable 5, Opus 5, Opus 5.5, Sonnet subagents)

Lipshitz chose an algebraic problem about Khovanov local equivalence groups that he thought Claude had a chance of solving and that he could check line by line. He reports that Claude did much of the exploratory research. He began with Fable 5, then used…

digital archiving

Dominik Lukeš

Codex (GPT-6.1 Sol) Qwen3-ASR 1.7B Gemma4 31B

I used Codex and local models to build a searchable catalogue of my radio recordings. The catalogue links summaries to passages in the transcripts so the wording and attribution can be checked.

physics

Matthew Schwartz

Claude Code (Fable 5) BootLoops

Schwartz used Claude to assemble BootLoops, a set of numerical and symbolic tools that began with calculations in particle physics. As a first test, he asked Fable 5 to implement methods from his published work and related papers. He reports that Claude…

economics

Chris Blattman

Claude Code Codex

Blattman describes an AI setup, started in January 2026, that acts as executive assistant, project manager and research collaborator for his large research teams. He uses it to prepare for meetings from notes and updates, to work through backlogs such as…

law (constitutional law, standing)

William Baude

Claude Code ("Fable 5.1", Max effort) Pangram AI-text detector

Baude opened Claude Code, tuned to "Fable 5.1" at Max effort, and asked it to expand the previous day's blog post into a law review article with no further input. He had written the idea himself and the model drew on his earlier writing on standing. By…

literary studies, library and information science (eighteenth-century book history)

Lawrence Evalyn

Claude Code

Evalyn works on the eighteenth-century practice of publishing books "by subscription". A 1970s to 1990s project collected 8,330 subscription lists in printed bibliographies, with no digital data preserved. He scanned a few pages and, in about three hours…

philosophy

Simon Goldstein

Claude Code / Claude Deep Drafter (his own agent pipeline)

Goldstein gave Claude a one-paragraph thesis and general instructions on writing philosophy papers; Claude did the literature review and wrote outlines and drafts, and he commented on every round and made many corrections. The resulting paper, "Epistocracy…

mathematics

Terence Tao

Claude Code (Opus 5) Lean 4 GPT (for suggested approaches)

A repository write-up ("Making of") presents a curated transcript of a four-day conversation (10-13 August 2026) between Tao and Claude (Opus 5, in Claude Code) that produced a Lean formalisation tied to his blog post on the proof of Sendov's conjecture.…

philosophy

Gus Skorburg

Claude Cowork Obsidian Readwise Reader

Skorburg describes three months of rebuilding his reading and writing around an Obsidian vault fed by everything he reads and annotates in Readwise, with Claude Cowork reading and writing in the vault. Each text he reads is routed as a clean copy to a…

economics

Francis DiTraglia

Claude Code Stata R

Francis DiTraglia reports using Claude Code and the GPL-licensed Stata source to port ivreg2 to R. The R package, ivreg2r, was published on CRAN on 21 July 2026. Package and development notes · CRAN record

engineering

Ke Xiao, Han Li, Teng Zhang, Yangchen Xu, Runze Mao, Zhi X. Chen

GPT-5.5 in Codex Pi coding agent (with four LLM backends) OpenFOAM-7

The authors propose a "case-bundle" operating model for coding agents in OpenFOAM-based CFD, with a Build mode (agent-assisted case development under engineering review) and a Replay mode (applying a reviewed case to new variants). In a study screening…

mathematics

Robert Lipshitz

Claude Code (Sonnet 4.6/5, Opus 4.8/5) VS Code

Lipshitz wrote a practical guide, dated around 30 July 2026 and updated on 28 October 2026, to using Claude Code for a research mathematician's non-theorem work. His typical prompt asks Claude to read a section of a .tex file for typos and mathematical…

economics (econometrics)

Pedro H. C. Sant'Anna

Claude Code slash commands and skills 18 custom sub-agents

On his page "My Claude Code Setup", Sant'Anna describes how he uses Claude Code for academic work beyond code: slides, papers, data analysis and more. He works goal-first: he describes a task, Claude picks a skill, and a review loop of specialised agents…

library and information science

Aaron Tay

Codex (GPT-5.5, extra-high effort; GPT-5.4-mini, low effort) Claude (model not named) Primo MCP server (forked)

Tay forked an open-source Primo MCP server so an agent could search his institution's library catalogue, then took selected 2023 searches that had returned zero results and ran them through 2026 models. He compared the agent loop with the vendor's own…

economics

Scott Cunningham

Claude Code YouTube series recording the collaboration

In an interview written up by Aniket Panjwani, Cunningham describes a public collaboration with Caitlin Myers (Middlebury) on a new paper that is ideated, conceptualised and written with Claude Code. Claude Code collected and merged the data. Myers, more…

medicine

Adam Johnson

Claude Code Docker with the OHDSI HADES stack Synthea (synthetic data)

Johnson presented to the OHDSI community how he uses Claude Code to develop transportable analytic code for prognostic modelling on OMOP data, constrained by a CLAUDE.md that enforces OHDSI conventions. The environment is a Docker template with the HADES…

Earlier accounts and uncertain dates

history, digital history

Joshua Sternfeld

Claude Code Claude desktop app (Claude Pro) Claude Sonnet 4.6

Sternfeld had Claude Code build, from a short prompt pointing at the API's README, an MCP server over Chronicling America (nearly 24 million digitised newspaper pages), with twelve tools including full-text search and retrieval of page OCR text. A first…

linguistics

Jia Yu, Weiwei Yu, Pengfei Xiao, Fukun Xing

Claude Opus via a command-line coding environment Model Context Protocol CQP-indexed corpora (Gutenberg, CLMET)

The authors link an LLM agent (Claude Opus in a command-line coding environment) to a CQP-indexed corpus through MCP and give it the single instruction to investigate English intensifiers in a 5-million-token Gutenberg corpus. The agent produced a…

neuroscience

Seyed Yahya Shirazi

Claude Code custom plugins ("Research Skills" marketplace)

Shirazi describes moving from first experiments with Claude Code in June 2025 to running multiple concurrent sessions from November 2025 across 92 projects, with more than 14,500 prompts in six months. He captured domain procedures, such as formatting an…

economics

Paul Goldsmith-Pinkham

Claude Code Cowork GitHub

This is a newsletter summary of an eight-episode Markus' Academy mini-series in which Goldsmith-Pinkham teaches Claude Code to applied economists, covering getting started, data analysis, planning, sub-agents, large datasets and verification. In an…

history

Cameron Blevins

Claude Code Gemini 3.1 Pro

Blevins returned to 1882 railway mail-transit tables he had abandoned fourteen years earlier and used Claude Code and Gemini over about a month, in half a dozen sessions of one to two hours, to build an interactive visualisation, "How Fast Was the Mail?".…

physics

Siddharth Mishra-Sharma

Claude Code (Opus 4.6) tmux on an HPC cluster with SLURM JAX

Mishra-Sharma describes setting a Claude Code agent to build a differentiable cosmological Boltzmann solver (CMB physics) over several days, in a task outside his core domain. The setup used a root instruction file with the plan, a changelog as persistent…

physics

Matthew Schwartz

Claude Code (Opus 4.5, VS Code extension) GPT-5.2 Gemini 3.0

Schwartz supervised Claude through a real quantum field theory research calculation at the level of a second-year graduate student, under strict rules: text prompts only, no editing files himself. The plan of 102 tasks across seven stages was built by…

computer science

Max Zimmer, Nico Pelleriti, Christophe Roux, Sebastian Pokutta

Claude Code Codex CLI OpenCode

The authors present a framework of methodological rules, written as agent prompts, that turns CLI coding agents into research assistants running inside a sandboxed container. Case studies in deep learning and mathematics report autonomous sessions that…

philosophy

Seth Lazar

Claude Code OpenClaw-derived retrieval tricks Slack

Lazar describes building lab research infrastructure around an agent, Minty: when a paper is added to the lab Slack, the agent tracks down the PDF and adds it to a growing searchable corpus. Minty also searches social media, arXiv, Substacks and RSS feeds,…

climate science

Zeke Hausfather

Claude Code (Opus 4.6) Gemini 3.1 web app Gemini Deep Research

Hausfather describes using Claude Code to write code for reduced-complexity climate model runs, saying projects that took hours or days ran in minutes while he still had to help create plans, develop tests and review results. He lists uses that work well…

higher education, science communication (not stated on the page; unverified)

Sam Illingworth

Claude Code custom slash-command skill file (/peer-review)

Illingworth explains how an academic can run Claude Code in a folder where a paper lives so that it can read the manuscript, check references and find inconsistencies between methods and results. His worked example is a skill file he wrote and says he runs…

chemistry

Benjamin J. Shields

Claude Code MCP server (cmxflow)

Shields built cmxflow, a Python package of composable cheminformatics blocks exposed through an MCP server, to see how useful agentic systems can be for cheminformatics and drug discovery. In a transcript, Claude Code builds two virtual-screening workflows…

genomics

Rob Patro

Claude (Sonnet, Opus) Claude Code (Opus 4.6) GitHub Copilot (web, CLI, VS Code)

Patro, a self-described skeptic, describes a series of ports and features done with AI coding agents in his lab's genomics software: the mim index core, a Cuttlefish in-memory tile collation feature (about five times faster), sshash and piscem translated…

political science

Tom Pepinsky

Claude Code

Pepinsky, previously a critic focused on what AI cannot do, reports starting to use Claude Code after seeing Andrew Hall, Scott Cunningham and Andrew Little, and calls it a game-changer for social scientists. He finds that Claude Code writes and executes…

qualitative methods, information systems (e-commerce interview data)

Rayed AlGhamdi

Claude Code

AlGhamdi took qualitative interview data from a doctoral study on e-commerce, originally coded by a human in NVivo in 2012, and ran four independent thematic analyses with Claude Code in 2026. Two prompting strategies were compared, general prompts and…

history, digital humanities

Shawn Graham

Claude Code pytesseract PaddleOCR

Graham followed course materials from Anastasia Salter's agentic-coding-for-humanities session (MLA, January 2026) to prompt Claude Code with a question he and his child had: did the local paper, the Shawville Equity, print anything about the Whitechapel…

computer science

Tim Dettmers

Claude Code

Dettmers reports eight months of using agents, mainly Claude Code, to automate his own work as a professor who writes little code; most use is for writing, including grant proposals and area-chair meta-reviews. For meta-reviews, an agent logs into…

political science

Andy Hall

Claude (AI coding assistant, Claude Code)

Hall took his own 2020 paper on vote-by-mail, handed it to an AI coding assistant and asked it to replicate the findings from scratch and extend the analysis with new data. He commissioned an independent audit: a UCLA PhD student, Graham Straus, updated…

digital humanities, international relations (United Nations General Assembly speeches)

Koen van Gilst

Claude Code (model not named) SQL database Python

The site presents itself as a case study in using Claude Code on a humanities corpus: more than seventy-five years of UN General Assembly speeches. By his account the agent wrote the SQL, the Python extraction scripts and the React pages while he supplied…

other

Kevin Yang

Claude Code Cursor Jupyter

Yang used Claude Code for several weeks to perform the data analysis for a paper he was working on and estimates it generated over 98 percent of the code. His setup pairs Claude Code in a terminal with Cursor for reviewing generated code, Jupyter for…

microbiology

José R. Penadés

Google AI co-scientist

Penadés's team tested Google's AI co-scientist system, at Google's request, on a question the lab had already answered experimentally but not yet published. The system returned five hypotheses, and the most plausible coincided with the lab's own hypothesis…

psychology

Matthias Mittner

Claude Opus 4.5 in Cursor (Plan Mode) PsychoPy EyeLink

Mittner documents a 60-90 minute session in which Claude Opus 4.5 in Cursor produced a 646-line PsychoPy N-back experiment with EyeLink eye-tracker integration for pupillometry. His workflow had four phases: Plan Mode report, careful plan review with…

political science

Tiago Ventura

Claude Code GPT (cross-checking)

Ventura's faculty-retreat slides list six uses of Claude Code: developing ideas, a retrieval-based literature notes base, a data-to-paper pipeline, coding support on a 23-country randomised trial, four skills for pre-submission review, and routine writing…

Replication attempts with agents and LLMs

economics

Mixed

LLM workflow run over 4,452 economics replication packages

Discrepancies were flagged in 3,460 articles or their appendices.

scientific machine learning (PINNs, SINDy, physics-informed field theory)

Mixed

Coding agents replicating four scientific machine learning papers with an evidence workflow

All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.

psychology, political science, economics

Partly succeeded

LLM reanalysis of 180 social and behavioural science studies

For 11 of the 180 studies the pipeline could not produce a usable effect size.

social sciences (four disciplines, 13 domains)

Mixed

SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers

Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.

economics and political science

Partly succeeded

Reproducing 48 social-science papers from the methods description alone

Three best agents matched the sign of the original coefficient about 85 to 91% of the time.

economics and political science (quantitative social science)

Mixed

Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams

Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).

political science

Succeeded

AI-assisted full-paper replication of 384 political science studies

Fully or largely reproducible: 82.5% of papers after DA-RT adoption against 20.8% before; given an accessible replication package, 92.1% (234 of 254).

social and behavioural science

Partly succeeded

ReplicatorBench: agents replicating social and behavioural science claims with new data

Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).

political communication / computational text analysis

Partly succeeded

Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini

The LLM labels agreed with the original classifier's labels in 69% of cases.

radiation biology / genomics

Mixed

Auditing three Kosmos radiation-biology claims against null models

DNA damage response versus p53 response (4 cell lines): Spearman rho -0.40, empirical p = 0.756; indistinguishable from random gene sets. Judged false.

metabolomics, materials science, neuroscience, statistical genetics

Mixed

Kosmos reproducing unpublished and preprinted findings (developer-run)

Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.

astrophysics

Failed

ReplicationBench: agents replicating astrophysics papers

Best average score was 22% (Claude 4.5 Sonnet). The abstract of the same version says best models score "under 20%", which does not match the 22% in the body text; the number depends on which table is read.

social psychology / computational social science

Mixed

Analytic flexibility in silicon samples: re-examining Argyle et al. Study 3

Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.

economics and political science

Failed

REPRO-Bench: agents assessing reproducibility of 112 social-science papers

Best existing agent (CORE-Agent with GPT-4o): 21.4% accuracy, below the 25% expected by guessing among four scores.

biomedical informatics / Alzheimer's disease

Partly succeeded

GPT-4o agent team reproducing five Alzheimer's disease studies

On average the agents approximately reproduced 53.2% of findings per study; in 3 of 5 studies a majority of findings was approximately reproduced.

machine learning

Partly succeeded

PaperBench: agents replicating 20 ICML 2024 papers from scratch

Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.

natural language processing

Failed

SciReplicate-Bench: reproducing algorithms from 36 NLP papers

Best result: Claude 3.7 Sonnet with Sci-Reproducer, 39.0% execution accuracy; all models were below 40%.

machine learning / recommender systems

Mixed

Independent evaluation of Sakana's AI Scientist

Five of twelve proposed experiments (42%) failed to run because of coding errors; the system flagged 10 generated ideas and both seeds as novel, including well-established ones such as micro-batching for SGD.

computer science, social science, medicine

Mixed

CORE-Bench: computational reproducibility from code and data, and its later 'solved' status

Best agent (CORE-Agent with GPT-4o): 60.0% Easy, 57.8% Medium, 21.5% Hard.

psychology and management

Mixed

LLMs as simulated participants: replicating 156 psychology and management experiments

Main effects replicated at 72.7% (GPT-4), 80.6% (Claude) and 76.2% (DeepSeek); interaction effects at 45.7%, 61.7% and 62.9%.