MixedReplication study 1 Sep 2026
Matthew Schwartz and colleagues economics
What happened Discrepancies were flagged in 3,460 articles or their appendices.
The authors describe an open-source workflow that gives an LLM a published article's replication package and asks it to do three things: reproduce the original calculations and flag discrepancies from the published findings, run automated sensitivity analyses, improve the calculations with a different implementation or algorithm, and develop extensions that fit the article's assumptions but are not in it. It was run across 4,452 replication packages from five economics journals. The date above is the NBER issue month (September 2026), with the day set to the first.
MixedReplication study 2 Jul 2026
Atharva Hans and colleagues scientific machine learning (PINNs, SINDy, physics-informed field theory)
What happened All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.
The authors wrote a workflow, packaged as a coding-agent skill, in which the agent records each computational claim of a paper as a target, rebuilds the method, runs experiments, and links outputs to provenance and to the paper's claim. Completion has to pass external validation checks, not the agent's own statement. They ran three independent replications from paper materials alone, with author code excluded, on each of four papers (PIFT, PINN-I, PINN-II, SINDy), twelve runs in all. At least one of the four papers (PIFT) is co-authored by one of the authors, so this is a mixed case of own and others' work; authorship of the other three was not checked.
Partly succeededReplication study 11 Jun 2026
Tobias Holtdirk and colleagues psychology, political science, economics
What happened For 11 of the 180 studies the pipeline could not produce a usable effect size.
An agentic pipeline received each study's original dataset, one focal statistical claim and, in the main condition, the relevant text of the article. It then wrote and ran its own analysis code, five runs per study, and the result was converted to a Cohen's d. The corpus was 180 published studies with predefined claims. For a subset the authors also compared the LLM with human reanalyses from the Multi100 collaboration. Three information conditions were tested: full paper, paper without methods, and abstract only.
MixedBenchmark 9 Jun 2026
Meysam Alizadeh and colleagues social sciences (four disciplines, 13 domains)
What happened Task-level accuracy: Claude Code 93.4%, Codex 62.1%. Paper-level (all tasks correct): 78.0% against 35.8%.
The benchmark has 221 tasks from 54 papers in four social-science disciplines. Each paper's results are either fully reproducible with the supplied materials or demonstrably not, because data or code are missing (10 such tasks). Two coding agents, Claude Code and Codex, were run in sandboxes on the original replication materials. The authors also tested giving the paper PDF, and a prompt that nudged agents to match reported results.
Partly succeededReplication study 23 Apr 2026
Benjamin Kohler and colleagues economics and political science
What happened Three best agents matched the sign of the original coefficient about 85 to 91% of the time.
The authors asked whether agents can reproduce results without the original code. A system extracts a structured description of each paper's methods, masks the reported numbers, and gives an agent only that description and the original data. Agents never see the original code, results or paper. Four agent scaffolds and four LLMs were run on 48 papers that the Institute for Replication had verified as reproducible. Outputs were compared cell by cell with the originals, and an LLM traced each discrepancy to a root cause.
MixedReplication game 26 Mar 2026
Abel Brodeur and colleagues economics and political science (quantitative social science)
What happened Reproduction rate: human-only 94% (31 of 33 teams), AI-assisted 91% (32 of 35), AI-led 37% (13 of 35).
In randomised replication games, 288 researchers formed 103 teams in three arms: human-only, AI-assisted (ChatGPT as a collaborator) and AI-led (ChatGPT drives, with minimal human oversight). Each team had seven hours to computationally reproduce a few pre-selected results from a published paper, find coding errors, and propose robustness checks. Twelve studies were used across events. Some papers contained coding errors the organisers had found earlier and not published. The published date above is the compile date of the accepted PNAS manuscript, not the journal issue date.
SucceededReplication study 17 Feb 2026
Yiqing Xu and colleagues political science
What happened Fully or largely reproducible: 82.5% of papers after DA-RT adoption against 20.8% before; given an accessible replication package, 92.1% (234 of 254).
The authors built a workflow that retrieves a paper's replication materials, rebuilds the computing environment, runs the code, and matches outputs to point estimates in the regression tables. They defined all empirical papers in three top political science journals from 2010 to 2025 and drew a stratified sample of 384 studies (8 per journal-year), covering 3,523 models. A second application re-ran 84 instrumental-variable studies and extracted 597 IV specifications for diagnostics. The numbers below are from version 3 of the preprint (31 May 2026); earlier versions reported different figures.
Partly succeededBenchmark 11 Feb 2026
Bang Nguyen and colleagues social and behavioural science
What happened Judging replicable versus non-replicable (version 3 numbers): GPT-5 accuracy 76.9% (macro F1 76.9), GPT-4o 61.5%, GPT-5-mini 48.7% (macro F1 47.9).
Unlike code-rerun benchmarks, this one asks agents to replicate a claim: find and retrieve new data, design the analysis, run it, and judge whether the claim replicates. It has 19 observational studies from the Center for Open Science's SCORE project, including human-verified non-replicable ones. Agents are scored on about 1,568 rubric checkpoints across extraction and data retrieval, design and execution, and interpretation. The authors tested their ReplicatorAgent with four LLMs and varied the programming language and level of code access.
Partly succeededReplication study 6 Feb 2026
Scott Cunningham and colleagues political communication / computational text analysis
What happened The LLM labels agreed with the original classifier's labels in 69% of cases.
Cunningham asked Claude Code to replicate the text-classification part of Card et al.'s PNAS paper on 140 years of immigration rhetoric in congressional and presidential speeches. The original used a fine-tuned RoBERTa classifier. He replaced it with zero-shot labels from gpt-4o-mini through the OpenAI Batch API. Claude Code handled the infrastructure, found and downloaded the original replication data, and ran the pipeline over about 305,000 speech segments. The post is part two of a pair; part one described the setup.
MixedAudit of an AI system 17 Nov 2025
Humza Nusrat and colleagues radiation biology / genomics
What happened DNA damage response versus p53 response (4 cell lines): Spearman rho -0.40, empirical p = 0.756; indistinguishable from random gene sets. Judged false.
The authors took three hypotheses that Kosmos produced from public gene-expression datasets and re-ran each analysis themselves. Each claim was tested against an empirical null: 10,000 random five-gene sets, 10,000 random genes, or 5,000 random 12-gene signatures. The aim was to see which AI-generated claims survived a simple falsification test.
MixedReplication study 4 Nov 2025
Ludovico Mitchener and colleagues metabolomics, materials science, neuroscience, statistical genetics
What happened Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.
The developers gave Kosmos open-ended objectives and datasets, then checked whether its reports matched findings that were unpublished, preprinted or published after the underlying models' cut-off. Three of the seven highlighted discoveries are framed this way: a metabolomics dataset on cooling-induced neuroprotection (later preprinted), a perovskite solar-cell fabrication dataset, and neuron-morphology data from eight connectomes (matching a bioRxiv preprint). Separately, expert scientists audited 102 statements from three reports for accuracy.
FailedBenchmark 28 Oct 2025
Christine Ye and colleagues astrophysics
What happened Best average score was 22% (Claude 4.5 Sonnet). The abstract of the same version says best models score "under 20%", which does not match the 22% in the body text; the number depends on which table is read.
The authors split 20 astrophysics papers into 111 tasks (version 2), each targeting a key result and co-developed with the original paper authors. Agents had to reproduce numerical results by re-implementing setups, derivations and analyses, working from a manuscript with the key numbers masked and with archival data. Seven frontier models were each run three times per paper. Domain experts also read agent traces to classify failures.
MixedAudit of an AI system 16 Sep 2025
Jamie Cummins and colleagues social psychology / computational social science
What happened Study 1: configurations varied substantially on all three criteria (recovering participant rankings, response distributions and between-scale correlations), and configurations that did well on one often did poorly on another.
Silicon samples are LLM-generated datasets meant to stand in for human respondents. In Study 1 the author generated 252 silicon-sample configurations (varying model, sampling parameters, prompt format and demographic detail) for a controlled case using two social-psychological scales. In Study 2 he re-examined Study 3 of Argyle et al. (2023), a published silicon-sample use case, with 66 alternative configurations. The question was how far defensible choices change conclusions about how well silicon samples match human data.
FailedBenchmark 25 Jul 2025
Chuxuan Hu and colleagues economics and political science
What happened Best existing agent (CORE-Agent with GPT-4o): 21.4% accuracy, below the 25% expected by guessing among four scores.
Agents were given a social-science paper (PDF) and its reproduction package, and asked to rate its reproducibility from 1 (not reproducible) to 4 (fully reproducible). Ground truth was taken from published reproduction reports by social scientists: 112 task instances across Stata, R and Python. The task is to execute the package and check consistency with the paper's claims, not only to rerun the code.
Partly succeededReplication study 29 May 2025
Nic Dobbins and colleagues biomedical informatics / Alzheimer's disease
What happened On average the agents approximately reproduced 53.2% of findings per study; in 3 of 5 studies a majority of findings was approximately reproduced.
The authors chose five highly cited papers that used the National Alzheimer's Coordinating Center "Quick Access" dataset and appeared reproducible from that dataset alone. A simulated research team of GPT-4o agents, built with AutoGen, wrote and executed code to reproduce each study's findings. The agents were given only the abstract, the methods section and the data dictionary. The authors extracted 35 key findings from the abstracts as the targets.
Partly succeededBenchmark 2 Apr 2025
Giulio Starace and colleagues machine learning
What happened Best agent on the main benchmark: Claude 3.5 Sonnet (New) with a basic scaffold, average replication score 21.0%. OpenAI o1 scored 13.2%; the other models scored under 10%.
Agents were given the text of 20 ICML 2024 Spotlight and Oral papers plus an addendum of clarifications. They had to write a codebase from scratch, run it, and produce a reproduce.sh entry point that regenerates the paper's empirical results. Each paper has a hierarchical rubric (8,316 gradable outcomes in total) that was co-developed with one of the original authors. Submissions were re-run and graded by an LLM judge against the rubric. The agents were barred from using the authors' own code repositories, which were blacklisted.
FailedBenchmark 31 Mar 2025
Yanzheng Xiang and colleagues natural language processing
What happened Best result: Claude 3.7 Sonnet with Sci-Reproducer, 39.0% execution accuracy; all models were below 40%.
The benchmark has 100 tasks from 36 NLP papers published in 2024. Each task asks a model to write the code for one algorithm described in a paper, inside the authors' repository, with test cases from the original implementation. The authors built Sci-Reproducer, with a Paper Agent that reads the paper and related literature and a Code Agent that retrieves repository dependencies and writes the code. Several reasoning and non-reasoning LLMs were tested.
MixedAudit of an AI system 20 Feb 2025
Joeran Beel and colleagues machine learning / recommender systems
What happened Five of twelve proposed experiments (42%) failed to run because of coding errors; the system flagged 10 generated ideas and both seeds as novel, including well-established ones such as micro-batching for SGD.
The authors installed the open-source AI Scientist and ran it end to end on recommender-systems research ideas, to test Sakana's claims that it automates the full research lifecycle. They supplied two seed ideas, generated further ideas, ran the experiments and read the resulting manuscripts. This is a test of a system's claims rather than of one published paper. The Zotero title is the first version's title; the arXiv v3 read here has a different title.
MixedBenchmark 17 Sep 2024
Zachary S. Siegel and colleagues computer science, social science, medicine
What happened Best agent (CORE-Agent with GPT-4o): 60.0% Easy, 57.8% Medium, 21.5% Hard.
CORE-Bench has 270 tasks drawn from 90 papers with public code, in Python and R. The agent must install dependencies, run the code, and answer questions about the outputs. There are three difficulty levels: results given (Easy), a run command given (Medium), or only a README (Hard). A task counts as solved if every answer falls within a prediction interval from three manual reproductions. The authors tested AutoGPT and their own CORE-Agent with GPT-4o and GPT-4o-mini.
MixedReplication study 29 Aug 2024
Ziyan Cui and colleagues psychology and management
What happened Main effects replicated at 72.7% (GPT-4), 80.6% (Claude) and 76.2% (DeepSeek); interaction effects at 45.7%, 61.7% and 62.9%.
The authors reran 156 scenario-based experiments from top social-science journals, 2015 to 2024, with LLMs playing the participants. Each model saw only one experimental condition at a time, so it was blind to the design and hypothesis. The 156 studies (50 articles from five management and psychology journals, randomly selected) gave roughly 620 to 680 main effects and 130 to 165 interaction effects, depending on the model (DeepSeek could not process images, so its set was smaller). A replication counted if the LLM sample preserved the direction and statistical significance of the original effect.