Make the coding agent prove the tests can fail

A green suite proves only that this implementation satisfied these checks

The dangerous sentence in AI-assisted development is “the tests pass.” It sounds objective, but it compresses several different claims: the requirement was translated correctly, the test reaches the relevant branch, the assertion can fail, the fixture represents legal input, the expected output is right, and the implementation did not exploit a shortcut in the harness. A green bar confirms none of those claims individually.

A paper posted on August 3 turns this weakness into an operating method. In Coding Agents as Test-Suite Auditors, an off-the-shelf coding agent receives a problem statement and one reference solution, then generates adversarial tests without seeing the hidden official suite or the accepted submissions being audited. One agent arm found 589 individually verified accepted-but-buggy submissions among 20,375 audited AtCoder submissions. Extending the same certification process across five agent arms produced a legality- and tolerance-clean union floor of 906.

The result is not “AI writes better tests than people.” On historically rejected logic bugs, each agent suite remained within 1.7 percentage points of the re-judged official suite, complementing rather than replacing it. The useful claim is narrower: coding agents can search for counterexamples that a fixed suite missed, provided that a separate evidence chain verifies the test, the expected output, and the legality of the input.

That distinction matters outside programming contests. A production test suite is an executable model of what a team believes can go wrong. It drifts as architecture changes, fixtures become stale, mocks simplify real behavior, and regression tests accumulate around previously observed failures. Coding agents can cheaply explore boundaries that maintainers did not enumerate. They should be treated as adversarial test authors, not as authorities on whether their own tests are correct.

Separate test generation from judgment

Anthropic's agent-evaluation guidance makes the same separation at a higher level. A task defines inputs and success criteria; a trial is one attempt; graders inspect the trace or outcome; the harness supplies tools and environment; and the outcome is the final state, not the agent's statement about that state. If a booking agent says a flight was booked, the database reservation is the outcome. If a coding agent says it fixed a race, the repeatable failing and passing executions are the outcome.

OpenAI's current third-party evaluation guidance adds a reporting constraint: state the claim the harness was designed to test and show evidence that the result is valid. Tool access, retries, context, resource budgets, and recovery logic can materially change observed performance. The same is true for test-suite audits. A high mutation score under one configuration does not establish correctness for every dependency, platform, workload, or concurrency schedule.

Start by defining an audit target smaller than “test the service.” Good targets are observable and consequential: authorization checks for tenant boundaries, idempotency of payment retries, parser behavior on invalid encodings, ordering guarantees in an event consumer, or a calculation invariant across currencies. The auditor gets the contract and a constrained workspace. It does not get production credentials, release authority, or a blank permission to rewrite the test harness.

A counterexample needs a chain of custody

The new paper's strongest contribution is not the agent prompt. It is the certification chain. Independent accepted solutions vote on the expected output. Brute-force programs resolve disagreements. A per-problem validator written from the statement alone confirms that the input is legal. The official judge is the object being audited, so it cannot also be the final source of truth for a newly discovered miss.

A production version will use different components, but the logic transfers. For a pricing engine, compare the candidate implementation with an intentionally slow reference calculation on bounded inputs. For a serializer, round-trip through an independent implementation and validate the schema. For a permission system, express invariants such as “a tenant identifier can never expand access” and check final resource state. For a database migration, apply it to a copied dataset, reverse it where supported, and compare checksums and constraints.

for candidate in auditor.generate_tests(contract, reference_code):
    if not domain_validator.accepts(candidate.input):
        reject(candidate, "illegal input")
        continue

    expected = differential_oracle(candidate.input)
    if expected.disagrees:
        expected = brute_force_or_human_resolution(candidate.input)

    observed = run_in_clean_environment(target, candidate.input)
    repeated = reproduce(candidate, target, runs=3)

    if observed != expected and repeated.stable:
        queue_for_human_promotion(candidate, expected, observed)
    else:
        quarantine(candidate)

The oracle must be independent enough to fail differently. Running two prompts through the same model, repository context, fixture generator, and flawed helper function creates correlated agreement. A differential oracle is stronger when implementations were developed independently or use different algorithms. A brute-force oracle is useful on reduced input sizes. A property oracle is useful when an exact answer is difficult but an invariant is crisp. Human adjudication remains necessary when the specification itself is disputed.

Minimize every accepted counterexample. A 50-step scenario that fails once is expensive to understand and likely to become flaky. Reduce it to the smallest input and action sequence that preserves the mismatch. Store the original finding, reduction log, validator result, expected-output evidence, environment digest, and final promoted test. This chain lets a later maintainer distinguish a real invariant from an accidental fixture.

Use mutation testing to ask whether assertions have teeth

Coverage answers whether code ran. Mutation testing asks whether a selected defect would be noticed. A mutation tool changes an operator, removes a call, negates a condition, alters a boundary, or modifies a return value, then runs the suite. A killed mutant caused a failure. A surviving mutant shows either an untested behavior, an equivalent change, or a test that observes execution without checking the consequence.

This gives the auditor a mechanical feedback loop. First run mutation analysis on changed or critical code. Give the agent surviving mutants, relevant requirements, and the existing tests. Ask it to write the smallest test that kills a specific mutant without coupling to implementation trivia. Re-run the mutation tool. The agent does not grade its own success; the mutant either survives or it does not.

audit_scope:
  paths: ["src/authorization/**", "src/billing/retry.ts"]
  mutation_budget: 180
  max_agent_tests: 12
  prohibited_changes: ["src/**", "test-helpers/oracle.ts"]
  required_gates:
    - legal_fixture
    - baseline_passes
    - named_mutant_killed
    - three_clean_reproductions
    - no_new_network_or_clock_dependency
    - maintainer_review
  evidence:
    - mutation_report
    - test_diff
    - command_log
    - environment_digest

Mutation score is not a universal quality number. Equivalent mutants cannot be killed because they do not change observable behavior. Some mutations are trivial while the dangerous fault class is absent from the operator set. Full-repository mutation can consume unacceptable compute. Focus on critical boundaries and changed code, cap the budget, and review the surviving-mutant categories rather than rewarding a single percentage.

A July pre-registered study of adversarial test hardening provides another caution. Its critic loop killed many mutants left by the initial suite, but the researchers also found that an earlier apparent model-lineage effect came from a harness artifact: an output cap truncated one model. A later review found another confound in how initial suites were sampled. The lesson is uncomfortable and useful: the evaluator itself must be audited.

Promote evidence, not every generated test

Do not place an unconstrained agent in the blocking CI path. Use a two-lane design. The deterministic lane runs established unit, integration, static, and security checks. The auditor lane runs on scheduled builds, high-risk changes, or a bounded pull-request budget. It creates findings and candidate tests in an isolated branch or artifact store. Only promoted tests become part of the blocking suite.

name: critical-test-audit
on:
  pull_request:
    paths: ["src/authorization/**", "src/billing/**"]

jobs:
  deterministic:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm test
      - run: npm run mutation:changed -- --reporter json

  agent-auditor:
    needs: deterministic
    permissions:
      contents: read
    steps:
      - run: ./scripts/build-audit-bundle.sh
      - run: ./scripts/run-agent-auditor.sh --max-tests 12 --no-network
      - run: ./scripts/validate-and-reproduce.sh --runs 3
      - uses: actions/upload-artifact@v4
        with:
          name: candidate-tests-and-evidence
          path: audit-output/

Keep the agent away from the oracle code, mutation report parser, and promotion decision where practical. Otherwise it can modify the scoreboard instead of the system. NIST documents grader gaming in coding evaluations: an agent may hard-code tested values or bypass a test rather than implement the intended fix. Protect the harness as build infrastructure, use read-only credentials, hash the evaluator image, and review changes to graders like changes to production authorization code.

Record variance across trials. Agent test generation is nondeterministic, and a flaky evaluation may reflect unstable environment data rather than unstable reasoning. Freeze dependencies, clock, locale, seedable randomness, external responses, and service versions. When the real system is nondeterministic, make the tolerated distribution explicit instead of retrying until green.

Use the auditor where an oracle exists or can be built

The worst first target is a broad, subjective product experience with no stable outcome and many live dependencies. The best first target is a consequential module with a clear contract, deterministic sandbox, known failure classes, and a maintainer who can judge counterexamples. Success means better evidence and caught defects, not a large volume of generated test code.

Failure modes that can make the audit weaker than the suite

A two-week pilot should leave a small, trusted suite

1. Select one critical module and write the exact contract, legal input domain, risk tier, owner, and prohibited side effects.
2. Freeze the evaluator image, dependencies, seeds, clocks, fixtures, reference implementation, validator, and baseline suite.
3. Run coverage and mutation analysis to identify weak assertions and surviving fault classes; do not optimize a percentage blindly.
4. Give the auditor bounded read access, the contract, and a fixed test budget. Deny production credentials, network access, source edits, and grader edits.
5. Validate input legality, compute expected outcomes independently, minimize counterexamples, and reproduce each mismatch at least three times.
6. Have a maintainer classify every candidate as promote, narrow, quarantine, reject, specification issue, or product defect.
7. Measure certified defects found, promoted tests, false findings, equivalent mutants, flaky candidates, review minutes, compute cost, and added CI time.
8. Re-run on a held-out set and after a harness change. Stop if the audit cannot distinguish product failure from evaluator failure.

Keep the first promoted set deliberately small. Ten tests with clear contracts and strong oracles are more valuable than a thousand generated cases that nobody can explain. The durable asset is the link between requirement, counterexample, expected outcome, reproduction, and owner.

Frequently asked questions

What is a coding-agent test-suite auditor?

It is a coding agent used as an adversarial test author. It searches for legal inputs and action sequences that distinguish a correct implementation from a plausible but wrong one. Independent validators and oracles determine whether each finding is real.

Is mutation testing enough?

No. It is a useful mechanical signal that assertions can catch selected injected faults. It does not prove the specification is right, the input domain is covered, the oracle is correct, or real production fault classes are represented.

Can one model judge another model's tests?

It can triage or critique, but agreement is weak evidence when the systems share data, prompts, helpers, or assumptions. Prefer executable state checks, independent algorithms, differential consensus, brute-force references, formal properties, and calibrated human judgment.

Should generated tests block a pull request automatically?

Only after promotion. A candidate test should first pass legality, oracle, reproduction, isolation, relevance, and ownership gates. Unverified findings belong in an artifact or review queue, not the blocking suite.