ScienceIDE turns 27 scientific codebases into 64 agent environments

A 45-author team has released ScienceIDE, infrastructure that converts 27 real scientific codebases into 64 executable environments where AI agents can be trained, graded and compared. The technical report was submitted to arXiv on 16 September 2026, and the authors are spread across the AItonomy Foundation, Oxford, Stanford, Princeton, Caltech and Berkeley. The work was initiated by PhAI Labs.

The mechanism is what separates this from another benchmark. Each environment packages a pinned upstream repository with its runtime, its own physics cases and a private verifier, so an agent edits source code and the verifier recompiles it and re-runs the science. The registry currently holds 2,812 manufactured tasks: 2,515 repair, 295 implementation and two acceleration. A companion collection supplies 1,076 executable checks of numerical outputs and physical invariants.

That design closes an obvious loophole. Reward comes from the simulation being numerically right again rather than from matching a reference diff, as the project’s GitHub repository puts it, and an untouched repository scores exactly zero. The upstream codes are named and real, including Athena++, MITgcm, PLUTO, PHANTOM, Gkeyll, Stim and pymatgen.

What the agents actually scored

For public evaluation the team carved out ScienceIDE-Hard, 85 validated tasks from 18 environments, split into 52 repair and 33 implementation problems. Fifteen models from eight providers ran through Codex, Claude Code or Gemini CLI, each with the same container and a one-hour episode budget. The primary metric is strict success, so a partial or budget-exhausted delivery counts as a failure.

Agent	Strict success	Est. cost per task	Output tokens per task
Fable 5.1	67.1%	$7.90	85.7k
Opus 5	64.6%	n/a	n/a
Astra	63.1%	$3.56	13.9k
Sol	55.0%	n/a	n/a
DeepSeek V4.1 Flash	36.0%	n/a	172.4k

The remaining eleven agents scored below 40%, and the authors warn against reading the top of that table as a settled ranking. Fable has a single-attempt estimate, and the repeat intervals for Opus and Astra overlap. Budget exhaustion alone reached a task-balanced 37.3% on the hard subset, and the authors note that the hard set probes horizon and scale only up to a one-hour budget and two coupled edit sites.

Spending more didn’t reliably buy accuracy. Astra averaged 9.4 minutes and 13.9k output tokens per task against Fable’s 16.8 minutes and 85.7k, while DeepSeek V4.1 Flash burned 172.4k output tokens for 36.0% success. Time changes the ordering too: at ten minutes Astra led on 49.6% to Fable’s 25.9%, and Fable only overtook it at roughly 31 minutes.

The team also used the environments as training data rather than just as a scoreboard. They fine-tuned three PhAI-IDE models at 4B, 9B and 72B on verified trajectories, and they report gains on held-out repair plus selected public benchmarks in code, reasoning and knowledge. Reinforcement learning on the verifier signal lifted a Qwen3.5-4B checkpoint from 0.357 to 0.857 mean verifier reward on held-out LAPS tasks after 30 steps, and from 0.286 to 0.571 on MITgcm-biogeo. That echoes the small-model GRPO gains Hugging Face reported earlier this month.

What the report does not show

The limitations section is unusually blunt, and it’s the part worth reading before anyone treats 67.1% as a measure of scientific ability. The evidence covers reference-verifiable software tasks, mostly repair and implementation in computational physics and geoscience, not open-ended discovery. Retrieval controls can’t exclude a model’s pretrained knowledge of public source code, so contamination isn’t ruled out.

Independent external audits and adversarial reward-hacking evaluations remain outstanding.

ScienceIDE technical report, section 5

Only part of the work is public so far. The repository ships 15 of the 64 environments and 30 of the 85 hard tasks, with the rest named but held back as a test set, and the paper lists a 1,000-environment release as a development target rather than a delivery. What’s already there is enough to reproduce the grading, though, because the published tasks carry their injected defect, the reference fix and the measurement records.

ScienceIDE landed in the same week as Paper2Agent, published in Nature on 16 September, which turns an individual paper into an agent that acts as a virtual corresponding author. Both attack the same gap from opposite ends, one making papers queryable and the other making code gradeable. What to watch is the 49 held-out environments, because the paper states plainly that neither its fine-tuning nor its reinforcement-learning results establish transfer to unseen codebases. Until someone shows that, it’s best read as a training substrate with a public scoreboard attached, which is a narrower thing than the AI-for-science headlines around it, such as DeepMind’s AlphaGenome Atlas.