# Auditing three Kosmos radiation-biology claims against null models

- Outcome: Mixed
- By: Humza Nusrat and colleagues
- Field: radiation biology / genomics
- Published: 2025-11-17
- Source: https://arxiv.org/abs/2511.13825

## What was attempted

The authors took three hypotheses that Kosmos produced from public gene-expression datasets and re-ran each analysis themselves. Each claim was tested against an empirical null: 10,000 random five-gene sets, 10,000 random genes, or 5,000 random 12-gene signatures. The aim was to see which AI-generated claims survived a simple falsification test.

## What happened

- DNA damage response versus p53 response (4 cell lines): Spearman rho -0.40, empirical p = 0.756; indistinguishable from random gene sets. Judged false.
- OGT and CDO1 predicting radiation-response modules (16 breast cancer lines): OGT r = 0.23, empirical p = 0.341, not supported. CDO1 r = 0.70, empirical p = 0.0039, a clear outlier. Judged well supported.
- 12-gene prostate recurrence signature (248 patients): c-index 0.613 beat random signatures (p = 0.0166) but the hazard ratio was not unusual (p = 0.3738). Judged plausible but uncertain.
- Overall verdict: one supported, one ambiguous, "one false result (DDR to p53)".

## Why it failed or succeeded, per the authors

- The false claim rested on very small n and a fluent mechanistic story that hid weak evidence.
- The ambiguous signature probably captures proliferation, which many random gene sets also capture.
- The supported CDO1 claim had the largest sample (16 lines) and combined data patterns with literature.
- The authors argue AI outputs should be treated as hypotheses to falsify, and propose a four-step audit checklist.

## Checks used

- Empirical null distributions from random gene sets with fixed seeds.
- Datasets and Kosmos reports public; the analysis code is "available upon request".

## Limits of this case

Three claims from one system and one field, audited by the same authors who chose the tests; no second auditor. Not a replication of a published paper but a re-analysis of AI outputs.
