MixedReplication study

Add a note

Kosmos reproducing unpublished and preprinted findings (developer-run)

What was attempted

The developers gave Kosmos open-ended objectives and datasets, then checked whether its reports matched findings that were unpublished, preprinted or published after the underlying models' cut-off. Three of the seven highlighted discoveries are framed this way: a metabolomics dataset on cooling-induced neuroprotection (later preprinted), a perovskite solar-cell fabrication dataset, and neuron-morphology data from eight connectomes (matching a bioRxiv preprint). Separately, expert scientists audited 102 statements from three reports for accuracy.

What happened

  • Metabolomics run: nine of the top fifteen metabolites matched the original analysis exactly, with log10 fold changes closely matching (R2 = 0.998) and the same nucleotide-salvage conclusion.
  • Connectome run: Kosmos reproduced the two main quantitative results of the preprint (lognormal distributions; power-law relations between metrics).
  • Statement audit: 79.4% of statements accurate overall; 85.5% of data-analysis statements reproducible, 82.1% of literature statements validated, and 57.9% of interpretive synthesis statements accurate.
  • The seven highlighted discoveries include four novel ones that were not reproductions.

Why it failed or succeeded, per the authors

  • Success is attributed to a shared world model linking data-analysis and literature agents over about 200 rollouts.
  • Weakness is in interpretation: Kosmos tends to conflate statistical significance with scientific value, and invents unorthodox metrics.
  • In the metabolomics run its extra claim about de novo synthesis could not be settled, because the data were steady-state levels only.

Checks used

  • Expert scientists classified sampled statements as supported or refuted, without being told which came from which analysis type.
  • Human-validation plots compared against Kosmos output, and the original studies as reference.
  • "Independent scientists found 79.4% of statements in Kosmos reports to be accurate", per the abstract.

Limits of this case

Developer-run and selected: the seven highlighted cases were picked from collaborations, and failed or uninteresting runs are not itemised. See the independent audit of Kosmos in kosmos-radiation-biology-audit.md for a harsher picture.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page