# Replication attempts

- [LLM workflow run over 4,452 economics replication packages](/replications/schwartz-andrews-shapiro-nber/index.md): The authors describe an open-source workflow that gives an LLM a published article's replication package and asks it to do three things: reproduce the original calculations and flag discrepancies from the published findings, run automated sensitivity analyses, improve the calculations with a different implementation or algorithm, and develop extensions that fit the article's assumptions but are not in it. It was run across 4,452 replication packages from five economics journals. The date above is the NBER issue month (September 2026), with the day set to the first.
- [Coding agents replicating four scientific machine learning papers with an evidence workflow](/replications/hans-bilionis-paper-replication-skill/index.md): The authors wrote a workflow, packaged as a coding-agent skill, in which the agent records each computational claim of a paper as a target, rebuilds the method, runs experiments, and links outputs to provenance and to the paper's claim. Completion has to pass external validation checks, not the agent's own statement. They ran three independent replications from paper materials alone, with author code excluded, on each of four papers (PIFT, PINN-I, PINN-II, SINDy), twelve runs in all. At least one of the four papers (PIFT) is co-authored by one of the authors, so this is a mixed case of own and others' work; authorship of the other three was not checked.
- [LLM reanalysis of 180 social and behavioural science studies](/replications/holtdirk-llm-reproducibility-assessment/index.md): An agentic pipeline received each study's original dataset, one focal statistical claim and, in the main condition, the relevant text of the article. It then wrote and ran its own analysis code, five runs per study, and the result was converted to a Cohen's d. The corpus was 180 published studies with predefined claims. For a subset the authors also compared the LLM with human reanalyses from the Multi100 collaboration. Three information conditions were tested: full paper, paper without methods, and abstract only.
- [SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers](/replications/socsci-repro-bench/index.md): The benchmark has 221 tasks from 54 papers in four social-science disciplines. Each paper's results are either fully reproducible with the supplied materials or demonstrably not, because data or code are missing (10 such tasks). Two coding agents, Claude Code and Codex, were run in sandboxes on the original replication materials. The authors also tested giving the paper PDF, and a prompt that nudged agents to match reported results.
- [Reproducing 48 social-science papers from the methods description alone](/replications/kohler-read-paper-write-code/index.md): The authors asked whether agents can reproduce results without the original code. A system extracts a structured description of each paper's methods, masks the reported numbers, and gives an agent only that description and the original data. Agents never see the original code, results or paper. Four agent scaffolds and four LLMs were run on 48 papers that the Institute for Replication had verified as reproducible. Outputs were compared cell by cell with the originals, and an LLM traced each discrepancy to a root cause.
- [Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams](/replications/i4r-ai-replication-games/index.md): In randomised replication games, 288 researchers formed 103 teams in three arms: human-only, AI-assisted (ChatGPT as a collaborator) and AI-led (ChatGPT drives, with minimal human oversight). Each team had seven hours to computationally reproduce a few pre-selected results from a published paper, find coding errors, and propose robustness checks. Twelve studies were used across events. Some papers contained coding errors the organisers had found earlier and not published. The published date above is the compile date of the accepted PNAS manuscript, not the journal issue date.
- [AI-assisted full-paper replication of 384 political science studies](/replications/xu-yang-political-science-replication/index.md): The authors built a workflow that retrieves a paper's replication materials, rebuilds the computing environment, runs the code, and matches outputs to point estimates in the regression tables. They defined all empirical papers in three top political science journals from 2010 to 2025 and drew a stratified sample of 384 studies (8 per journal-year), covering 3,523 models. A second application re-ran 84 instrumental-variable studies and extracted 597 IV specifications for diagnostics. The numbers below are from version 3 of the preprint (31 May 2026); earlier versions reported different figures.
- [ReplicatorBench: agents replicating social and behavioural science claims with new data](/replications/replicatorbench/index.md): Unlike code-rerun benchmarks, this one asks agents to replicate a claim: find and retrieve new data, design the analysis, run it, and judge whether the claim replicates. It has 19 observational studies from the Center for Open Science's SCORE project, including human-verified non-replicable ones. Agents are scored on about 1,568 rubric checkpoints across extraction and data retrieval, design and execution, and interpretation. The authors tested their ReplicatorAgent with four LLMs and varied the programming language and level of code access.
- [Scott Cunningham: Claude Code replicating a PNAS text-classification paper with GPT-4o-mini](/replications/cunningham-claude-code-pnas/index.md): Cunningham asked Claude Code to replicate the text-classification part of Card et al.'s PNAS paper on 140 years of immigration rhetoric in congressional and presidential speeches. The original used a fine-tuned RoBERTa classifier. He replaced it with zero-shot labels from gpt-4o-mini through the OpenAI Batch API. Claude Code handled the infrastructure, found and downloaded the original replication data, and ran the pipeline over about 305,000 speech segments. The post is part two of a pair; part one described the setup.
- [Auditing three Kosmos radiation-biology claims against null models](/replications/kosmos-radiation-biology-audit/index.md): The authors took three hypotheses that Kosmos produced from public gene-expression datasets and re-ran each analysis themselves. Each claim was tested against an empirical null: 10,000 random five-gene sets, 10,000 random genes, or 5,000 random 12-gene signatures. The aim was to see which AI-generated claims survived a simple falsification test.
- [Kosmos reproducing unpublished and preprinted findings (developer-run)](/replications/kosmos-reproduces-unpublished-findings/index.md): The developers gave Kosmos open-ended objectives and datasets, then checked whether its reports matched findings that were unpublished, preprinted or published after the underlying models' cut-off. Three of the seven highlighted discoveries are framed this way: a metabolomics dataset on cooling-induced neuroprotection (later preprinted), a perovskite solar-cell fabrication dataset, and neuron-morphology data from eight connectomes (matching a bioRxiv preprint). Separately, expert scientists audited 102 statements from three reports for accuracy.
- [ReplicationBench: agents replicating astrophysics papers](/replications/replicationbench-astrophysics/index.md): The authors split 20 astrophysics papers into 111 tasks (version 2), each targeting a key result and co-developed with the original paper authors. Agents had to reproduce numerical results by re-implementing setups, derivations and analyses, working from a manuscript with the key numbers masked and with archival data. Seven frontier models were each run three times per paper. Domain experts also read agent traces to classify failures.
- [Analytic flexibility in silicon samples: re-examining Argyle et al. Study 3](/replications/cummins-silicon-sample-flexibility/index.md): Silicon samples are LLM-generated datasets meant to stand in for human respondents. In Study 1 the author generated 252 silicon-sample configurations (varying model, sampling parameters, prompt format and demographic detail) for a controlled case using two social-psychological scales. In Study 2 he re-examined Study 3 of Argyle et al. (2023), a published silicon-sample use case, with 66 alternative configurations. The question was how far defensible choices change conclusions about how well silicon samples match human data.
- [REPRO-Bench: agents assessing reproducibility of 112 social-science papers](/replications/repro-bench/index.md): Agents were given a social-science paper (PDF) and its reproduction package, and asked to rate its reproducibility from 1 (not reproducible) to 4 (fully reproducible). Ground truth was taken from published reproduction reports by social scientists: 112 task instances across Stata, R and Python. The task is to execute the package and check consistency with the paper's claims, not only to rerun the code.
- [GPT-4o agent team reproducing five Alzheimer's disease studies](/replications/dobbins-alzheimers-agents/index.md): The authors chose five highly cited papers that used the National Alzheimer's Coordinating Center "Quick Access" dataset and appeared reproducible from that dataset alone. A simulated research team of GPT-4o agents, built with AutoGen, wrote and executed code to reproduce each study's findings. The agents were given only the abstract, the methods section and the data dictionary. The authors extracted 35 key findings from the abstracts as the targets.
- [PaperBench: agents replicating 20 ICML 2024 papers from scratch](/replications/paperbench/index.md): Agents were given the text of 20 ICML 2024 Spotlight and Oral papers plus an addendum of clarifications. They had to write a codebase from scratch, run it, and produce a reproduce.sh entry point that regenerates the paper's empirical results. Each paper has a hierarchical rubric (8,316 gradable outcomes in total) that was co-developed with one of the original authors. Submissions were re-run and graded by an LLM judge against the rubric. The agents were barred from using the authors' own code repositories, which were blacklisted.
- [SciReplicate-Bench: reproducing algorithms from 36 NLP papers](/replications/scireplicate-bench/index.md): The benchmark has 100 tasks from 36 NLP papers published in 2024. Each task asks a model to write the code for one algorithm described in a paper, inside the authors' repository, with test cases from the original implementation. The authors built Sci-Reproducer, with a Paper Agent that reads the paper and related literature and a Code Agent that retrieves repository dependencies and writes the code. Several reasoning and non-reasoning LLMs were tested.
- [Independent evaluation of Sakana's AI Scientist](/replications/sakana-ai-scientist-audit/index.md): The authors installed the open-source AI Scientist and ran it end to end on recommender-systems research ideas, to test Sakana's claims that it automates the full research lifecycle. They supplied two seed ideas, generated further ideas, ran the experiments and read the resulting manuscripts. This is a test of a system's claims rather than of one published paper. The Zotero title is the first version's title; the arXiv v3 read here has a different title.
- [CORE-Bench: computational reproducibility from code and data, and its later 'solved' status](/replications/core-bench/index.md): CORE-Bench has 270 tasks drawn from 90 papers with public code, in Python and R. The agent must install dependencies, run the code, and answer questions about the outputs. There are three difficulty levels: results given (Easy), a run command given (Medium), or only a README (Hard). A task counts as solved if every answer falls within a prediction interval from three manual reproductions. The authors tested AutoGPT and their own CORE-Agent with GPT-4o and GPT-4o-mini.
- [LLMs as simulated participants: replicating 156 psychology and management experiments](/replications/cui-silicon-subjects-156-experiments/index.md): The authors reran 156 scenario-based experiments from top social-science journals, 2015 to 2024, with LLMs playing the participants. Each model saw only one experimental condition at a time, so it was blind to the design and hypothesis. The 156 studies (50 articles from five management and psychology journals, randomly selected) gave roughly 620 to 680 main effects and 130 to 165 interaction effects, depending on the model (DeepSeek could not process images, so its set was smaller). A replication counted if the LLM sample preserved the direction and statistical significance of the original effect.
