MixedAudit of an AI system
Auditing three Kosmos radiation-biology claims against null models
Also mixed
- LLM workflow run over 4,452 economics replication packages1 Sep 2026
- Coding agents replicating four scientific machine learning papers with an evidence workflow2 Jul 2026
- SocSci-Repro-Bench: Claude Code and Codex reproducing 54 social-science papers9 Jun 2026
- Institute for Replication AI replication games: human-only, AI-assisted and AI-led teams26 Mar 2026
- Kosmos reproducing unpublished and preprinted findings (developer-run)4 Nov 2025
- Analytic flexibility in silicon samples: re-examining Argyle et al. Study 316 Sep 2025
What was attempted
The authors took three hypotheses that Kosmos produced from public gene-expression datasets and re-ran each analysis themselves. Each claim was tested against an empirical null: 10,000 random five-gene sets, 10,000 random genes, or 5,000 random 12-gene signatures. The aim was to see which AI-generated claims survived a simple falsification test.
What happened
- DNA damage response versus p53 response (4 cell lines): Spearman rho -0.40, empirical p = 0.756; indistinguishable from random gene sets. Judged false.
- OGT and CDO1 predicting radiation-response modules (16 breast cancer lines): OGT r = 0.23, empirical p = 0.341, not supported. CDO1 r = 0.70, empirical p = 0.0039, a clear outlier. Judged well supported.
- 12-gene prostate recurrence signature (248 patients): c-index 0.613 beat random signatures (p = 0.0166) but the hazard ratio was not unusual (p = 0.3738). Judged plausible but uncertain.
- Overall verdict: one supported, one ambiguous, "one false result (DDR to p53)".
Why it failed or succeeded, per the authors
- The false claim rested on very small n and a fluent mechanistic story that hid weak evidence.
- The ambiguous signature probably captures proliferation, which many random gene sets also capture.
- The supported CDO1 claim had the largest sample (16 lines) and combined data patterns with literature.
- The authors argue AI outputs should be treated as hypotheses to falsify, and propose a four-step audit checklist.
Checks used
- Empirical null distributions from random gene sets with fixed seeds.
- Datasets and Kosmos reports public; the analysis code is "available upon request".
Limits of this case
Three claims from one system and one field, audited by the same authors who chose the tests; no second auditor. Not a replication of a published paper but a re-analysis of AI outputs.