Route coding agents by repeated task evidence, not model rank

The model is not the agent you are buying

Netlify published a compact experiment on August 12: one small web-app prompt, eleven models, three runs per model, all executed through the same agent setup. The outputs differed substantially in design, implementation, and reported credit use. The important result was not a universal winner. It was a reminder that model choice becomes intelligible only when the task and execution environment are visible.

A coding agent is a compound system. The model proposes actions, but the harness decides what repository context enters the prompt, how tools are described, whether commands run in a sandbox, how errors return, which skills are loaded, when the loop stops, and what consumes the token budget. A model can look excellent in a benchmark yet fail inside a weak harness. A smaller model can outperform a larger one after the harness provides a better repository map, a deterministic test tool, and a bounded recovery loop.

This makes “Which model is best?” the wrong procurement question. Ask which versioned combination completes a defined task class under an acceptable quality, security, latency, and cost envelope. The answer may be one model for landing-page generation, another for multi-file repair, and a third for security-sensitive migrations. Routing is an engineering decision, not a fan ranking.

Recent experiments point toward inspectable, scenario-level evaluation

The Netlify experiment used OpenRouter-backed model access and ran the same coffee-shop site prompt three times per candidate with default settings. Netlify reported average credit consumption ranging from 2.4 for its lowest-credit variant to 519 for its highest-credit variant. Those are gateway credits in this experiment, not a portable dollar-price table, and the task was a narrow visual build. Treat the numbers as evidence of dispersion, not as a price-performance verdict for production software.

Netlify also released AXIS, an open-source evaluation harness that describes a scenario with a prompt and rubric, runs agents, captures tool calls, responses, and recovery behavior, and emits an inspectable 0–100 report across four dimensions. Netlify says its internal Agent Runners skills improved average scores by 26 points while reducing time and cost on its scenarios. That is a first-party result that needs independent reproduction, but it illustrates the key variable: changing the skills layer can change observed model performance without changing the model.

The initial Hacker News thread drew more than 200 points and 90 comments during this scan window. The useful community questions concerned defaults, reproducibility, visual judgment, price units, and whether a single design prompt generalizes. Those objections do not invalidate the experiment. They specify the metadata a serious evaluation must publish.

OpenAI's evaluation playbook similarly recommends stating the evaluation claim, controlling the harness, setting a budget, validating scoring, repeating runs, and checking for broken tasks or reward hacking. Google's LLM Comparator supports side-by-side rubric-based comparison, including human evaluation. The shared direction is clear: a benchmark result is the beginning of a claim audit, not the end of model selection.

Freeze the complete evaluation unit

A reproducible record identifies more than a model string. Hash or version every component that can change the outcome:

Do not silently tune only the favored model. Either use a fixed prompt and budget to compare raw fit, or document a separate optimized track in which every candidate receives the same tuning allowance. Report both. A production route may reasonably use model-specific settings, but the selection process should reveal that advantage rather than call it model intelligence.

Separate execution, evidence, and routing

Keep the hidden oracle outside the agent's readable workspace. For a code repair, that can mean public tests the agent may run plus hidden regression, mutation, and security checks executed afterward. For a visual build, use both automated checks and a rubric that defines hierarchy, fidelity, overflow, responsiveness, and accessibility. “Looks good” is not an audit trail.

Capture tool calls and recovery. Final-output scoring misses the difference between a clean solution and a lucky patch reached after destructive commands, repeated failures, or unauthorized network access. The trace is also where teams learn whether the prompt, tool schema, or skill—not the model—needs repair.

Build a small suite from work you actually accept

Start with 20 to 40 tasks across three to five classes. Use recent completed work, but scrub secrets and freeze the starting commits. Include ordinary cases, known exceptions, ambiguous requests, and hostile inputs. A suite made entirely of clean happy paths selects agents that fail when production becomes interesting.

suite: product-engineering-v1
repetitions: 5
task_classes:
  - name: bounded_bug_fix
    fixtures: 12
    oracle: hidden_tests_and_mutation
    risk: moderate
  - name: component_from_spec
    fixtures: 8
    oracle: browser_checks_plus_human_rubric
    risk: low
  - name: dependency_migration
    fixtures: 6
    oracle: build_tests_security_policy
    risk: high
variants:
  - model: model-a
    harness: agent-runner-3.2
    skills: repo-skills@sha256:...
    max_tokens: 60000
    max_minutes: 25
    max_tool_calls: 120
controls:
  network: deny_except_registry_mirror
  secrets: fixture_only
  destructive_host_tools: false

Run each task at least three times for exploration; use five or more when a route will carry consequential work. Agent outcomes are non-deterministic even at low temperature because tool output, environment timing, context selection, and recovery diverge. Show the full distribution. A model that passes 80% with predictable cost may be more useful than one that passes 90% but occasionally consumes ten times the budget or changes unrelated files.

Blind human reviewers to model identity where practical. Calibrate a rubric on known good, marginal, and failing examples. When using an LLM judge, record judge model and prompt, test position and verbosity bias, and compare a sample against qualified humans. Never allow the candidate to be its only judge.

Score accepted work, then price the route that produced it

A composite score is useful only if hard failures remain visible. A route that exposes credentials or edits outside scope must not compensate with beautiful code. Apply policy gates first, then compare quality and efficiency among surviving runs.

hard_fail = unauthorized_write
         or secret_exposure
         or hidden_test_access
         or destructive_command
         or provenance_missing

accepted = !hard_fail
        and required_tests_pass
        and scope_validator_pass
        and reviewer_score >= 4

accepted_cost = total_provider_cost / max(accepted_runs, 1)
route_score = 0.45 * pass_rate
            + 0.20 * reviewer_quality
            + 0.15 * recovery_quality
            + 0.10 * latency_score
            + 0.10 * cost_score

Report pass rate with uncertainty, median and p95 wall time, median and p95 cost, tool calls, retries, changed-file count, reviewer correction minutes, and unsafe-action attempts. “Cost per accepted run” is more useful than cost per token because a cheap failed attempt creates rework. Also separate provider spend from internal compute, reviewer time, test infrastructure, and incident risk.

For Netlify's experiment, the wide credit spread is a reason to normalize units before a purchase decision. Provider credits, listed token prices, cached input, tool costs, and subscription quotas are not interchangeable. Recalculate with the billable units in your route and record the access date because prices and model aliases change.

Publish routing as a versioned policy with a fallback

After the suite passes, map task classes to variants. Keep the policy narrow and expiring. A passing front-end route does not inherit permission to perform database migrations.

routes:
  component_from_spec:
    primary: model-b+harness-3.2+ui-skills-7
    fallback: model-a+harness-3.2+ui-skills-7
    min_repeated_pass_rate: 0.90
    max_p95_cost_usd: 3.50
    max_p95_minutes: 18
    permissions: [read_repo, write_workspace, run_tests]
    human_gate: visual_and_accessibility_review
    expires: 2026-09-15
  dependency_migration:
    primary: model-a+harness-3.2+migration-skills-2
    permissions: [read_repo, write_workspace, registry_mirror]
    human_gate: security_and_owner_approval
    fallback: human_only
rollback_if:
  - unsafe_action_attempts > 0
  - rolling_pass_rate_20 < 0.85
  - model_or_harness_digest_changes

Log why a route was selected and which policy version applied. If classification confidence is low, choose the more restrictive route or a human triage lane. Shadow-run a candidate on replayed or low-risk tasks before replacing a production route. Re-evaluate after model aliases, system prompts, skills, tool schemas, sandbox images, budgets, repository architecture, or acceptance tests change.

Failure modes that model tables hide

Two-week evaluation and release checklist

1. Define the claim: task class, users, consequence, acceptable quality, and excluded uses.
2. Select 20–40 scrubbed fixtures with starting commits, exceptions, adversarial cases, and independent oracles.
3. Freeze model version, harness, prompt, skills, tools, sandbox, permissions, and all budgets.
4. Verify fixture integrity and keep hidden tests, expected patches, and judge exemplars inaccessible to candidates.
5. Run at least three repetitions per exploratory variant; record complete traces, patches, tests, spend, and timing.
6. Apply hard safety and scope gates before calculating a composite quality score.
7. Classify every failure and repair broken tasks, graders, tools, or harnesses before blaming the model.
8. Blind and calibrate human reviewers; validate any LLM judge against known examples and deterministic evidence.
9. Calculate pass rate, p95 latency and cost, correction time, and cost per accepted run for each task class.
10. Shadow the proposed route, publish a primary and fallback, set expiry and rollback triggers, and require the relevant human release gate.

Frequently asked questions

What is the right unit for coding-agent model evaluation?

A versioned task executed by a specific model, harness, prompt and skills bundle, tool environment, permission set, and budget, then checked by an independent oracle. Store the manifest digest with every result.

How many times should each task run?

Three is a useful exploratory minimum; five or more is better for release-critical routes. Increase repetitions when outcomes are variable or the cost of failure is high. Report failures and tails, not only the average.

Can an LLM judge another coding agent?

Yes, as one rubric signal. Calibrate it on known examples, blind candidate identity, test common biases, preserve its evidence, and anchor the release decision in tests, validators, security checks, and qualified human review.

Should the cheapest model that passes once become the default?

No. Require repeated success, safe traces, acceptable tail latency and cost, and a fallback. Optimize cost per accepted outcome rather than cost per attempt or token.

When must the team re-run the evaluation?

After a model alias or version changes, and after material changes to the agent harness, system prompt, skills, tool schemas, runtime image, permissions, budget, repository architecture, or acceptance oracle.