# Coding agents replicating four scientific machine learning papers with an evidence workflow

- Outcome: Mixed
- By: Atharva Hans and colleagues
- Field: scientific machine learning (PINNs, SINDy, physics-informed field theory)
- Published: 2026-07-02
- Source: https://arxiv.org/abs/2607.02134

## What was attempted

The authors wrote a workflow, packaged as a coding-agent skill, in which the agent records each computational claim of a paper as a target, rebuilds the method, runs experiments, and links outputs to provenance and to the paper's claim. Completion has to pass external validation checks, not the agent's own statement. They ran three independent replications from paper materials alone, with author code excluded, on each of four papers (PIFT, PINN-I, PINN-II, SINDy), twelve runs in all. At least one of the four papers (PIFT) is co-authored by one of the authors, so this is a mixed case of own and others' work; authorship of the other three was not checked.

## What happened

- All twelve workspaces passed the completion gate and all 158 recorded targets were matched with report coverage.
- Of 39 scalar anchor-run observations (13 standardised anchors, three runs each), 37 fell inside the paper-anchored threshold; two did not (a Schrödinger relative L2 error of 4.8 × 10^-2 against a 1% threshold, and a Navier–Stokes coefficient error of 16.4% against 10%).
- Elapsed time ranged from 1.2 to 13.0 hours; posterior median 6.9 hours for PINN-II and 5.0 for PINN-I against 2.2 for PIFT and 1.9 for SINDy.
- Reproduced PINN tables kept the reported accuracy patterns, but individual cell values did not match every printed digit.
- For a cylinder-wake target the paper materials omit the simulation trajectories and preprocessing, so the matched evidence supports the wake structure and trajectory geometry, not exact coefficient equality.

## Why it failed or succeeded, per the authors

- The authors say a plain prompt does not reliably preserve progress or check that generated evidence supports claims, which the external gate fixes.
- "repeated runs differ in how papers are divided into targets, in numerical fidelity", and in effort and accepted evidence rules, even when all pass.
- Papers without seeds, sampled training sets or preprocessing details limit exact replication.

## Checks used

- Workspace validation gate with hashes and provenance for every matched target.
- A separate paper-anchored scalar threshold analysis, independent of the per-target acceptance rule.
- Three independent runs per paper.

## Limits of this case

Four papers and one agent (Codex with GPT-5.4); "matched" means the target-specific acceptance rule was met, which can be looser than the paper-anchored threshold. The authors' own workflow is being evaluated on a small set including at least one paper by an author.
