Scientific-agent benchmarks need execution receipts, not just scores

The hard part of scientific-agent evaluation is specifying success

A scientific agent can produce fluent analysis, runnable code, polished plots, and a wrong conclusion in the same session. The useful benchmark therefore cannot ask only whether the answer sounds scientific. It must define the environment, required artifacts, acceptable numerical tolerances, and evidence that a result came from the intended workflow.

Terminal-Bench-Science 0.1, announced August 28, packages 70 workflows from the life, physical, Earth, mathematical, and engineering sciences. The project says 920 proposals produced 464 implementation approvals, 386 pull requests, and 70 release tasks. That funnel matters more than raw breadth: a research workflow must be interesting to a domain expert, difficult for a frontier agent, executable in a controlled terminal, and objectively gradable.

The public release ran every evaluated system three times per task. Claude Opus 5 with Claude Code led at 30.0%, followed by GPT-5.6 Sol with Codex at 22.4% and Claude Fable 5 with Claude Code at 21.4%. Those are current protocol results, not universal model rankings. The harness, reasoning configuration, sandbox, dependencies, task version, grader, retry policy, and cost all participate in the outcome.

Developer attention is real but bounded. The launch reached 116 points and 36 comments on Hacker News, while the public repository showed hundreds of stars and active task-review work during the August 30 check. That supports writing about the evaluation mechanism. It does not prove that laboratories have adopted the benchmark or that a 30% result predicts safe autonomous research.

A scientific task is a small reproducible system

Each useful task needs more than an instruction. It needs an initial filesystem or dataset state, required tools and dependencies, network policy, resource limits, a time budget, an expected artifact set, and a grader that can distinguish partial progress from a resolved workflow. The agent should encounter the same consequential constraints that a researcher would: imperfect data, coupled steps, scientific software, intermediate files, and numerical or structural acceptance criteria.

The grader should evaluate consequences rather than prose. A signal-processing task may compare recovered parameters within justified tolerances. A theorem task may require a machine-checkable proof. A simulation task may inspect generated files, conservation constraints, and summary statistics. A grader that checks only whether a filename exists rewards agents that learn the shape of success without doing the science.

Oracle runs are equally important. The repository recommends running oracle solutions repeatedly to confirm that tasks work in the chosen sandbox. If the reference solution flakes, the benchmark cannot tell whether an evaluated agent failed or the environment did. Treat oracle reliability as a release gate, not a setup convenience.

Run a pinned dataset before comparing systems

The public repository uses Harbor to materialize tasks and execute agents. For a reproducible comparison, pin the dataset release, Harbor version, agent adapter, model identifier, reasoning configuration, environment provider, concurrency, secrets policy, and retry behavior. The following command mirrors the documented shape but deliberately uses a pinned release placeholder that a team should resolve and record before execution:

uv tool install "harbor[modal,daytona]"

harbor run \
  -d terminal-bench-science/terminal-bench-science@v0.1.0 \
  --agent codex \
  --model openai/gpt-5.6-sol \
  --ak reasoning_effort=max \
  --n-concurrent 8 \
  --env modal

Start with the oracle on the same infrastructure. Run it at least as many times as the evaluated system on a small domain slice. Record failed environment setup separately from failed task execution. A dependency download timeout, unavailable license server, full disk, or missing GPU is infrastructure evidence, not scientific-agent evidence.

evaluation:
  dataset: terminal-bench-science@v0.1.0
  trials_per_task: 3
  domains: [earth-sciences, mathematical-sciences]
  timeout_seconds: 7200
  retries:
    environment_setup: 1
    agent_run: 0
  network: task-policy
  preserve:
    - command_log
    - final_files
    - grader_output
    - resource_usage
    - failure_class

Do not repair failed runs interactively. If an operator installs a missing package or clarifies an instruction for one model, the protocol changed. Either restart all compared systems under the corrected release or label the intervention as a separate assisted condition.

Interpret resolution rate as a system probability under one release

The official release also plots cost and token use against resolution. That changes the decision. GPT-5.6 Sol reportedly approached Fable 5's resolution at less than one-third of the total evaluation cost, while Fable used fewer tokens. A research platform may prefer the system with lower cost, a better domain slice, or more stable artifacts rather than the highest aggregate score.

Read per-task outcomes before making architecture changes. A model may fail because the harness cannot manage long-running processes, because it misreads a scientific file format, because the tool policy blocks a required action, or because it reaches a plausible but unverified conclusion. Those require different fixes. Model replacement is only one of them.

Store one evidence receipt per trial

{
  "task": "earth-sciences/inverse-problem-17",
  "task_release": "tb-science-v0.1.0",
  "task_digest": "sha256:...",
  "agent": "codex-0.42.1",
  "model": "gpt-5.6-sol",
  "reasoning": "max",
  "environment": "modal-image@sha256:...",
  "trial": 2,
  "started_at": "2026-08-30T01:12:03Z",
  "duration_seconds": 1874,
  "cost_usd": 18.42,
  "grader": "inverse-problem-grader@sha256:...",
  "resolved": false,
  "partial_checks": {"input_valid": true, "fit_converged": true, "residual_ok": false},
  "artifacts_digest": "sha256:...",
  "failure_class": "scientific-validation",
  "operator_intervention": false
}

This record lets a team re-grade preserved artifacts after a grader fix, distinguish model behavior from environment failure, and reproduce the exact run. Preserve the command log and final filesystem where privacy and licensing allow. Hash large or restricted artifacts and store them in controlled object storage rather than placing sensitive research data in a public benchmark record.

Add scientific review for tasks that matter to deployment. Automated graders can check equations, formats, tolerances, and known invariants. They cannot decide that a research question was well posed, a causal claim is justified, a proxy is scientifically meaningful, or an output is safe to use. Human domain review remains the final gate.

Four validity threats survive a clean leaderboard

Public-task contamination. The project warns that benchmark data should not appear in training corpora, but public tasks can still be indexed, copied, or optimized against. Track task exposure, add private holdouts, rotate variants, and distinguish public-development scores from blinded evaluation.

Task and grader defects. Scientific workflows are hard to specify. A grader may reward a shortcut, reject a valid alternative, or depend on brittle numerical tolerances. Version every repair, preserve historical results, and publish whether old artifacts were re-graded or systems were re-run.

Harness confounding. A leaderboard row combines model and agent. Prompt scaffolding, context management, retry logic, tool adapters, and filesystem behavior can move the score. Compare a model across harnesses and a harness across models before attributing the result.

Collection validity. Seventy carefully reviewed tasks are still a sample. TerminalWorld reports weak correlation across terminal-task collections, and K-Bench argues for examining what was delivered and claimed on real scientific requests. Use multiple benchmarks plus internal workflows; no single collection represents science.

Design comparisons that can survive a model or harness change

A useful comparison begins with a written hypothesis. “System B is better” is too broad. Ask whether a new harness raises repeatable resolution on long-running simulation tasks without increasing invalid artifacts, whether a reasoning setting improves mathematical tasks at an acceptable marginal cost, or whether a tool-policy change reduces environment failures without granting wider authority. Choose the task slice and acceptance measures before looking at outcomes.

Keep everything except the tested factor fixed. Use the same task release, container image, network policy, secrets, trial count, concurrency policy, time limit, grader digest, and aggregation code. Randomize or interleave run order when provider load or shared infrastructure could create time effects. If one system needs a different dependency, adapter, or prompt, record that difference as part of the system rather than describing the comparison as model-only.

For a small slice, publish counts and trial-level results instead of a decimal with false precision. A difference of two resolved tasks may disappear under another three runs. Examine paired outcomes: which exact tasks changed, whether the same failure class moved, and whether the artifact quality improved. When confidence is weak, the correct decision may be “continue testing,” not “pick the higher number.”

Finally, preserve the losing system's artifacts. Regression analysis is often more valuable than the headline win. If a new version gains three tasks but begins inventing unsupported scientific interpretations on two previously safe tasks, the aggregate rate hides an unacceptable trade. Promotion should require no material regression on protected tasks, stable infrastructure, acceptable review burden, and domain-owner sign-off on representative artifacts.

Adapt the protocol to your own research platform

1. Collect ten to twenty recurring workflows from practicing researchers, not model demos.
2. Define the scientific artifact and independent acceptance checks before writing the agent instruction.
3. Package data, dependencies, licenses, network needs, and resource limits into a reproducible environment.
4. Create an oracle and run it repeatedly on the same infrastructure used for agents.
5. Require domain review, technical review, and a final bar-raiser for each task.
6. Run multiple trials with no interactive rescue and preserve trial-level receipts.
7. Report domain slices, cost, latency, partial checks, flakes, and failure classes alongside resolution.
8. Maintain a private holdout and rotate tasks that become contaminated, saturated, or underspecified.
9. Use human scientific review before any benchmark success changes production authority.

A good internal benchmark should change engineering priorities. If most failures are missing packages or stalled subprocesses, fix the runtime. If artifacts are complete but claims exceed the evidence, improve verification and review. If success varies wildly across trials, reduce nondeterminism and add checkpoints. The score is the index; the failure corpus is the roadmap.

Failure modes to record instead of flattening into zero

Decision checklist

• Pin task, grader, harness, model, image, and policy versions.
• Prove the oracle passes repeatedly before evaluating agents.
• Preserve trial-level artifacts, logs, cost, time, and failure class.
• Compare systems within one protocol and release only.
• Inspect domain and task slices before acting on an aggregate.
• Maintain private or rotating tasks to detect public-task overfitting.
• Keep scientific validity and production authorization with qualified humans.

FAQ

What does Terminal-Bench-Science measure?

It measures whether a configured model-and-agent system can complete versioned computational research workflows in a controlled terminal environment and satisfy task-specific artifact graders.

Is the leading score a base-model ranking?

No. It is a system result. The model, harness, reasoning settings, tools, environment, task release, trials, and grader all contribute.

Can a passing task prove scientific correctness?

No. It proves that the released checks passed. Scientific interpretation, construct validity, causal claims, provenance, and safe use still require domain review.

Should a company use only this benchmark to choose an agent?

No. Combine it with internal workflows, private holdouts, other terminal and scientific benchmarks, security tests, cost limits, and human evaluation.