MixedReplication study

Add a note

Coding agents replicating four scientific machine learning papers with an evidence workflow

What was attempted

The authors wrote a workflow, packaged as a coding-agent skill, in which the agent records each computational claim of a paper as a target, rebuilds the method, runs experiments, and links outputs to provenance and to the paper's claim. Completion has to pass external validation checks, not the agent's own statement. They ran three independent replications from paper materials alone, with author code excluded, on each of four papers (PIFT, PINN-I, PINN-II, SINDy), twelve runs in all. At least one of the four papers (PIFT) is co-authored by one of the authors, so this is a mixed case of own and others' work; authorship of the other three was not checked.

What happened

  • All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.
  • Of 39 scalar anchor-run observations (13 standardised anchors, three runs each), 37 fell inside the paper-anchored threshold; two did not (a Schrödinger relative L2 error of 4.8 × 10^-2 against a 1% threshold, and a Navier–Stokes coefficient error of 16.4% against 10%).
  • Elapsed time ranged from 1.2 to 13.0 hours; posterior median 6.9 hours for PINN-II and 5.0 for PINN-I against 2.2 for PIFT and 1.9 for SINDy.
  • Reproduced PINN tables kept the reported accuracy patterns, but individual cell values did not match every printed digit.
  • For a cylinder-wake target the paper materials omit the simulation trajectories and preprocessing, so the matched evidence supports the wake structure and trajectory geometry, not exact coefficient equality.

Why it failed or succeeded, per the authors

  • The authors say a plain prompt does not reliably preserve progress or check that generated evidence supports claims, which the external gate fixes.
  • "repeated runs differ in how papers are divided into targets, in numerical fidelity", and in effort and accepted evidence rules, even when all pass.
  • Papers without seeds, sampled training sets or preprocessing details limit exact replication.

Checks used

  • Workspace validation gate with hashes and provenance for every matched target.
  • A separate paper-anchored scalar threshold analysis, independent of the per-target acceptance rule.
  • Three independent runs per paper.

Limits of this case

Four papers and one agent (Codex with GPT-5.4); "matched" means the target-specific acceptance rule was met, which can be looser than the paper-anchored threshold. The authors' own workflow is being evaluated on a small set including at least one paper by an author.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page