FailedBenchmark

Add a note

SciReplicate-Bench: reproducing algorithms from 36 NLP papers

What was attempted

The benchmark has 100 tasks from 36 NLP papers published in 2024. Each task asks a model to write the code for one algorithm described in a paper, inside the authors' repository, with test cases from the original implementation. The authors built Sci-Reproducer, with a Paper Agent that reads the paper and related literature and a Code Agent that retrieves repository dependencies and writes the code. Several reasoning and non-reasoning LLMs were tested.

What happened

  • Best result: Claude 3.7 Sonnet with Sci-Reproducer, 39.0% execution accuracy; all models were below 40%.
  • Average execution accuracy across models with Sci-Reproducer was 0.235.
  • Without the agent to read literature and repository context, most LLMs scored below 0.1 execution accuracy; Sci-Reproducer added 0.181 on average.
  • When the authors added the missing or mismatched information from the papers as extra input, execution accuracy rose by 0.05 for GPT-4o-mini, 0.25 for DeepSeek-V3 and 0.04 for o3-mini-low.

Why it failed or succeeded, per the authors

  • Papers omit hyperparameters, numerical-stability tricks, implementation logic and coding strategy; the authors conclude "missing or inconsistent algorithm descriptions as key barriers to successful reproduction".
  • Models, like human developers, cannot see runtime information from a compiler in this setup, which limits debugging.

Checks used

  • Execution against test cases from the original repositories.
  • Reasoning-graph accuracy to score understanding, plus CodeBLEU and dependency recall.
  • Authors' manual comparison of every task's code against its paper description to find missing or mismatched details.

Limits of this case

A code-completion task inside a given repository, not a reproduction of a paper's results; only the abstract, introduction, results and analysis sections were read closely. The aggregator claim about syntax-error reduction was not checked and is not used.

Search the collection

↑ ↓ move · Enter open · Shift+Enter original source · Esc close · Search index for agents

Keyboard shortcuts

Single keys work when you are not typing and no dialog is open. Turn them off if stray key presses move the page. ⌘K / Ctrl+K and Esc keep working.

/
Search the collection
⌘K / Ctrl+K
Open or close search
?
Show these shortcuts
h
Home
l
Lessons
a
Accounts
r
Replications
p
Papers
g
Guides and tools
d
Timeline
b
Bookmark the record you are reading (press again to remove it)
s
Your saved list and notes
← →
Previous and next record on an account, lesson, replication or paper page, and in a quick look
m
Switch between system, light and dark theme
t
Back to the top of the page
Open full page