A poisoned benchmark can become the next version of your coding agent

The attack turns evaluation into a write channel

Most teams think of a benchmark as measurement: freeze some tasks, run an agent, score the traces, compare versions. That mental model breaks when the score is fed into an automated loop that rewrites the agent's prompts, tools, skills, or harness. The benchmark no longer observes the system. It helps manufacture the next system.

On September 15, Franziska Roesner and Tadayoshi Kohno published a bounded proof of concept against three research systems: a modified Darwin Gödel Machine, the Self-Improving Coding Agent, and Hyperagents. The attacker controls the benchmark, including its reward function, but does not directly edit the agent or base model. In one reported Hyperagents experiment using Sonnet 4.5, poisoned tasks pushed the system to evolve instructions that disabled HTTPS certificate validation on neutral URL-fetching tasks. The paper also reports that contamination could survive later evolution on clean benchmarks.

This is not evidence that every production coding agent is compromised. The systems are research prototypes, the attack assumes influence over evaluation inputs, and the measured outcomes belong to the paper's configurations. The result matters because its control lesson is broader: if an external task, reward, or grader can influence what becomes the next agent version, that artifact sits on the promotion path and needs the same provenance, isolation, review, and rollback discipline as code.

Adjacent evidence makes the path less hypothetical. August's EvoMal paper studied self-evolving agents that retrieve skills and author new ones. A planted skill could become a template for agent-authored copies; those copies re-entered the library and remained retrievable after the original seed was removed. Separately, public coding-agent benchmarks continue to expose ordinary contamination, broken tasks, leaked answers, reachable hidden tests, fix history, and network retrieval. These are different failures, but they share one root mistake: treating evaluation inputs as inert data.

Separate three kinds of contamination

“Benchmark contamination” now covers several mechanisms. A team cannot choose controls until it names which one it faces.

The third failure is the new boundary. The agent does not need to “escape” a sandbox. It can behave exactly as the improvement loop asks: observe a score, propose a change that scores better, and preserve that change. The attacker wins when the promotion system mistakes benchmark performance for a complete safety case.

Model the assets precisely. The target may be a system prompt, tool description, skill library, retrieval index, source code, dependency lockfile, evaluator policy, or memory carried between generations. The attacker may control a task, expected answer, fixture, reward function, source repository, issue body, dependency, or evaluator-side file. The harmful behavior may trigger immediately or only on a later task family.

The minimum useful security question is therefore not “did the candidate pass?” It is: which signed inputs could influence the candidate, which state changed, who independently evaluated the delta, and what evidence authorized promotion?

Build a promotion system the candidate cannot edit

known-good baseline + signed improvement request
  -> isolated proposer creates a candidate delta
  -> policy gate rejects forbidden files and authority changes
  -> evaluator runs signed tasks in fresh answer-isolated sandboxes
  -> independent grader scores functional + security evidence
  -> clean holdout + security canaries + regression suites
  -> named promotion service verifies quorum and signatures
       | promote immutable candidate
       | quarantine for investigation
       | reject and preserve evidence
  -> monitored rollout with automatic rollback to known-good baseline

The proposer may inspect allowed source, traces, and aggregate feedback, then generate a candidate patch. It must not edit benchmark tasks, graders, policy, signing keys, promotion code, or the immutable baseline. Its filesystem identity and credentials should make those writes impossible, not merely discouraged.

The policy gate inspects the exact delta before expensive evaluation. Reject changes that weaken TLS, disable certificate or signature verification, broaden network access, suppress exceptions, delete assertions, rewrite security canaries, alter approval logic, or add opaque binary artifacts. A model can help explain the diff; deterministic policy decides whether it is admissible.

The evaluator materializes every task from a content-addressed manifest into a fresh environment. Hidden tests and solution artifacts are mounted only into the grader, not the agent. Later fix history is stripped. Network access is absent by default and every exception is explicit, narrow, and logged. Caches are either disabled or included in the digest.

The grader is a separate principal. It verifies tests, policy checks, prohibited-action logs, resource limits, and artifact hashes. An agent-written “PASS” file is evidence from the subject, not a verdict. The same applies to LLM judges: use them for bounded semantic review, preserve their version and prompt, and do not let them replace deterministic assertions.

The promoter requires a complete evidence bundle and a rule that the candidate cannot change. Promotion is a state transition from one immutable version to another, signed by a named service or reviewer. A partial suite, missing manifest, altered task, failed canary, or unexplained score jump routes to quarantine.

Attest every input, not just the container image

A reproducible image is necessary but insufficient. The candidate can receive an answer through an instruction file, wrapper binary, environment variable, cache, proxy, or task-specific scaffold. Bind all of those inputs into one manifest and sign the final evidence bundle.

apiVersion: agent-eval-integrity/v1
run_id: eval-2026-09-22-0042
baseline:
  agent_digest: sha256:<known-good-agent>
  prompt_digest: sha256:<system-prompt>
candidate:
  patch_digest: sha256:<candidate-delta>
  parent: eval-2026-09-18-0017
suite:
  manifest_digest: sha256:<task-index>
  task_source: private-holdout-2026q3
  grader_digest: sha256:<grader>
  reward_digest: sha256:<reward-policy>
environment:
  image_digest: sha256:<rootfs>
  kernel_policy_digest: sha256:<sandbox-policy>
  network: none
  cache: disabled
  hidden_tests_mount: grader-only
promotion:
  policy_digest: sha256:<promotion-rules>
  required_suites: [functional, security-canary, clean-holdout]
  signer: spiffe://eval/promoter
  rollback_target: sha256:<known-good-agent>

The manifest gives reviewers a stable answer to “what ran?” A signature adds integrity, but its claim must stay narrow. A signed bundle proves that a key signed these recorded inputs and results. It does not prove that the task set covers the real deployment, that the signer is trusted unless its key is anchored, or that a grader catches every semantic failure.

Promotion code should fail closed and explain why:

def decide_promotion(bundle, trusted_keys, policy):
    verify_signature(bundle, trusted_keys)
    verify_hashes(bundle.inputs, bundle.artifacts)
    require(bundle.parent == policy.current_baseline)
    require(bundle.suites == policy.required_suites)
    require(bundle.forbidden_access_attempts == 0)
    require(bundle.security_canaries.failed == 0)
    require(bundle.clean_holdout.regressions <= policy.max_regressions)
    require(bundle.delta.touches(policy.immutable_paths) is False)
    require(bundle.review.quorum >= policy.review_quorum)
    return immutable_release(bundle.candidate_digest, bundle.evidence_digest)

Security canaries must test behavior, not keywords

A poisoned agent may look normal on the same tasks that taught it. Use hidden, independent canaries that express invariants the candidate must preserve. Keep them outside the proposer workspace and rotate the concrete cases without changing the policy meaning.

Include positive controls so the system cannot pass by refusing everything. A safe candidate should still complete normal HTTPS requests, legitimate test updates, approved package installs, and explicitly allowed network calls. Record functional utility and security behavior separately; do not compress them into one weighted score that lets a performance gain buy a security regression.

“It passes clean tasks now” is not a decontamination proof

The September paper reports cases where contamination persisted after later clean evolution. EvoMal similarly shows why deleting the original planted artifact may not be enough when agent-authored descendants remain retrievable. Once a candidate has influenced later prompts, tools, skills, or code, the dependency graph matters more than the latest aggregate score.

1. Freeze the lineage: preserve poisoned inputs, traces, candidate deltas, descendants, and promotion evidence.
2. Stop promotion: revoke the candidate's release authority and pause downstream learning from its artifacts.
3. Return to a known-good root: rebuild from an immutable baseline predating the untrusted input rather than editing the latest version in place.
4. Diff behavior and artifacts: inspect prompt, tool, skill, dependency, policy, and source changes across every descendant.
5. Run targeted canaries: test the demonstrated payload class plus nearby variants and dormant triggers.
6. Use a fresh holdout: tasks must not come from the poisoned loop or its authored descendants.
7. Reissue trust deliberately: rotate signing keys or promotion identities if the boundary may have been crossed.

Do not rely only on “unlearning” prompts. EvoMal reports a counter-prompt that materially reduced its tested copying path, which is useful defense-in-depth evidence. It is not a substitute for preventing unreviewed agent-authored skills from entering the trusted retrieval pool. Prompt defenses target copy behavior; admission control can cut retrievability structurally.

Failure modes to test before automation

Run a 30-day promotion-integrity rollout

Week 1: inventory every input to the improvement loop: tasks, repositories, issue text, graders, reward code, prompts, skills, tools, images, caches, model versions, and environment variables. Draw who can write each artifact. Freeze one known-good baseline and one rollback target.

Week 2: isolate agent and grader. Remove hidden tests, later git history, host credentials, process metadata, and default network access from the candidate. Create the run manifest, stable signing key, artifact hashes, and tamper-evident evidence bundle.

Week 3: build functional positive controls and security canaries. Seed answer reads, gold-patch mining, network retrieval, reward-file edits, test weakening, TLS disablement, copied malicious skills, and promotion-policy changes. Confirm each is blocked, detected, or both.

Week 4: shadow the full promotion flow without automatically replacing production. Require a named review quorum for unexplained gains, any security-relevant delta, and every change to prompts, tools, skills, dependencies, sandbox policy, grader, or reward. Rehearse quarantine and rollback from the evidence bundle.

Measure more than pass rate: forbidden-access attempts, canary failures, task-level regressions, unexplained score jumps, evaluator disagreement, promotion rejections, lineage gaps, rollback time, and the fraction of candidate changes that touch high-risk surfaces. The objective is not to stop improvement. It is to make every promoted improvement attributable and reversible.

For persistent workspace and cross-run state controls, use the agent trust-handoff guide. For checking whether tests can actually fail, use the test-suite auditor guide. This page adds the adversarial benchmark-to-promotion boundary.

FAQ

Can containerization stop benchmark poisoning?

Containerization can block answer access, host writes, and unapproved egress. It cannot by itself detect a malicious task or reward that legitimately enters through the benchmark channel, nor can it stop answer keys already embedded in the scaffold. Pair isolation with input provenance and independent promotion authority.

Should self-improving agents be banned from editing prompts and skills?

Not necessarily. They can propose changes. The safety boundary is admission: changes should enter a candidate branch, pass policy and independent evaluation, and be promoted by a different principal. Agent-authored artifacts should not become trusted merely because the agent wrote them.

Can a second model serve as the independent grader?

It can add semantic review, but independence is not created by using another model name. Preserve its version and prompt, keep it outside candidate control, and combine it with deterministic tests, policy checks, and human review for high-impact promotion.

What should trigger an incident?

Unsigned or changed evaluation inputs, a candidate touching immutable paths, forbidden artifact access, security-canary failure, unexplained score jumps, missing lineage, altered reward logic, or a promoted behavior outside its declared delta should all stop promotion and open an investigation.