AI coding agents need a patch-size contract, not another leaderboard

Patch size is evidence about change, not a verdict about quality

Most coding-agent leaderboards reward a successful test outcome. Parsimony, released October 13, asks a second question: among attempts that solved the same task, which agents changed less syntax? The project parses patches published by DeepSWE, SWE-bench Verified, SWE-bench Live, and other boards. It counts statements, calls, names, literals, and other syntax elements instead of lines, then exposes the data, analyzer version, task population, and hashes.

The question matters because generation is cheap and review is not. A patch that introduces two abstractions, three helper layers, and a configuration surface for a narrow bug may pass every test while transferring cost to maintainers. A compact patch can be easier to understand, revert, and carry through the next release. The 77-point Hacker News launch discussion and the maintainer's r/codex post focused on this gap rather than claiming that terseness is quality.

But the metric becomes dangerous when compressed into “smaller is better.” Deleting a check lowers net units. Replacing readable branches with a dense expression can shrink syntax while raising cognitive cost. A patch can omit tests, documentation, telemetry, migration logic, error handling, or compatibility work and appear efficient. Parsimony itself states that models which write more code can also solve more tasks and that harness differences, missing patches, and language boundaries matter.

The practical move is to treat patch size as a review-budget and regression-risk signal. It can tell a platform team where to look, which agent-task pairs tend to expand scope, and whether a new harness changes the shape of accepted work. It cannot replace a maintainer or a test suite.

Parsimony turns published patches into a reproducible secondary measure

The analyzer reconstructs source before a patch, applies the diff, parses both versions, and counts language-specific syntax units. Net units equal added units minus removed units. Churn uses both sides. Total measures the resulting amount. Comments, formatting, punctuation, and identifier length do not dominate as they would in a raw line count.

task + frozen repository + published patch
        |
        v
reconstruct changed files
        |
        v
parse before / parse after
        |
        v
added | removed | net | churn | missing
        |
        v
shared solved-task cohort + uncertainty

Comparison must stay within language and task cohort. A Python statement and a TypeScript node are not interchangeable units. One model may have results on easy tasks while another covers the full board. Comparing raw means rewards sample selection. Parsimony compares models on shared tasks and publishes measured counts so missing data remains visible.

The result is closer to a laboratory instrument than a universal score. It can show that Agent A tends to add 0.8 times the syntax of Agent B on the same solved tasks under a specified harness. It cannot show that A's patch is clearer, has stronger validation, respects repository policy, or will reduce maintenance cost. Current studies reinforce that boundary: maintainability research finds differences in validation, error handling, downstream code size, and later task difficulty; repository-policy research finds violations even when patches are functionally correct.

Define a vector gate instead of a blended quality score

A composite score hides tradeoffs. Raise the patch-size weight and agents learn to omit work; raise correctness alone and sprawling patches win. Keep each dimension visible and define which ones block release.

version: 1
population: repo-issues-2026-10
repository_revision: 8b71d2c
harness: agent-runner@2.4.1
limits: {wall_minutes: 30, tool_calls: 120, cost_usd: 8}
gates:
  correctness: {required_tests: pass, hidden_tests: pass}
  task_scope: {unrelated_files: 0, unexplained_dependencies: 0}
  policy: {repository_rules: pass, licenses: pass, secrets: pass}
  structure: {new_high_complexity_functions: 0}
  patch_size: {compare_shared_tasks: true, alert_ratio: 1.5}
  evidence: {commands: required, logs: required, diff_summary: required}
human_release: {owner: maintainer, exact_commit_required: true}

Correctness is blocking. Run deterministic tests, hidden tests where appropriate, type checks, linters, security tests, and task-specific negative cases. A size improvement never compensates for a failed assertion.

Scope is blocking when unexplained. Record which files, dependencies, schemas, and interfaces changed. A larger patch can be valid when a safe migration needs compatibility or when the issue spans generated clients. Require a reason, not an arbitrary maximum.

Policy is blocking. Repository rules, license boundaries, generated-file policies, documentation requirements, approvals, and provenance remain enforceable even when code works. The current SWE-CC study reports that agents still violate applicable project policies, which is why size cannot stand in for compliance.

Structure and size are diagnostic gates. Flag new complex functions, duplication, coupling, excessive file spread, dependency growth, and size ratios. Route the patch to review with the evidence that triggered concern. Never squash these dimensions into a number nobody can explain.

Review cost closes the loop. Record active review minutes, clarification rounds, requested splits, and whether a reviewer could explain the changed behavior without the agent transcript. If a smaller patch takes longer to verify because it is dense or under-documented, the team should not celebrate its unit count.

Downstream change cost is the delayed gate. Revisit accepted patches when the subsystem changes again. Record whether the earlier patch increased files touched, forced compatibility work, caused a revert, or made the next task harder. Sequential benchmarks exist because isolated issue resolution misses this compounding behavior.

Reproduce the measurement before using it for procurement or routing

The Parsimony repository is MIT-licensed and requires Python 3.12 or later. Its committed datasets and offline site build make the boards inspectable. Run the project checks and rebuild against frozen data before trusting a chart.

git clone --depth 1 https://github.com/La5u/Parsimony
cd Parsimony
scripts/check.sh
scripts/rebuild-site.sh --offline

python -m parsimony dataset --output verified.jsonl
python -m parsimony analyze <submission-id>   --dataset verified.jsonl --limit 20 --output results.jsonl

For an internal evaluation, store the repository revision, issue text, environment image, harness, model identifier, inference settings, tools, time and cost budgets, test commands, patch, analyzer version, and missing-measurement reason. Hash the input population and result file. If the agent can inspect git history or public solutions, isolate that path or label the run contaminated.

for each task in frozen_population:
  run candidate and baseline with identical limits
  verify tests and task-specific negative cases
  measure added, removed, net, churn, files, dependencies
  compute structure and policy results
  retain failures and missing measurements
compare only shared, valid attempts
report distributions and outliers, not only means

Do not compare languages in one unit total. Do not compare a ten-task hand-run sample with a complete 500-task submission. Do not discard failed or unparsable patches without the denominator. The honest artifact is a table with coverage, uncertainty, and outliers.

Use size to ask better review questions

At pull-request time, show the metric beside the diff, not above it. A reviewer should see why the patch exists, what changed, what verification ran, which policies apply, and how size compares with relevant baselines. The purpose is to focus attention, not automate taste.

At model-routing time, segment by task. A verbose agent may be better at migrations that require tests and documentation, while a compact agent may suit isolated fixes. Routing by one global mean loses this distinction. Track accepted patch rate, revert rate, review minutes, post-merge defects, and subsequent change cost to learn whether the offline metric predicts anything locally.

Use medians and distributions rather than a winner-takes-all rank. A stable agent with a narrow range may be easier to govern than one that is usually compact but occasionally produces a twenty-file rewrite. Publish outliers with task names and artifacts; they often reveal more about prompt scope, permissions, and harness behavior than the average.

Keep execution cost separate. A compact final patch produced after failed edits, repeated test runs, and abandoned files is not operationally parsimonious. Report tokens, tool calls, wall time, sandbox compute, and reviewer time beside code-change measures.

Failure modes that turn a useful metric into a bad incentive

Public boards are useful for hypothesis formation. Procurement needs representative languages, monorepo shape, build systems, private conventions, security rules, and reviewer behavior. A benchmark win cannot waive local acceptance.

Generated tests deserve their own lane. An agent can expand a test suite while weakening assertions, snapshotting the wrong behavior, or duplicating coverage. Excluding tests entirely hides verification cost; mixing tests and implementation into one number makes comprehensive regression work look wasteful. Report implementation, tests, documentation, configuration, migrations, and generated outputs separately.

Refactors need a different baseline from bug fixes. A safe refactor may delete thousands of units, while a compatibility migration may temporarily increase code before planned removal. Attach the metric to a task contract that names the expected change class, required artifacts, and sunset work.

Roll out the metric without teaching agents to game it

1. Collect four weeks of baseline patches without changing acceptance.
2. Freeze representative tasks and record harness, model, tools, and budgets.
3. Measure net, churn, file spread, dependencies, and structural signals.
4. Compare only shared valid tasks; publish missing and failed denominators.
5. Ask reviewers whether alerts matched real verification cost.
6. Create task-class thresholds as review triggers, not universal rejection lines.
7. Test omitted validation, dense rewrites, dependency displacement, and deletion gaming.
8. Track review time, reverts, defects, and future change cost.
9. Version the contract and name the owner for threshold changes.

Success is not a smaller average patch. It is lower verification and maintenance burden without lower correctness, weaker controls, or hidden scope. If size does not predict those outcomes, keep it as descriptive telemetry.

Run a shadow period before enforcement. Show reviewers the proposed alert but do not change merge authority. Log false positives, missed risky patches, and task classes where the measure is meaningless. Review results with maintainers, security, quality, and developer-experience owners.

Calibrate on real categories rather than one fleet-wide percentile. A dependency bump, schema migration, parser change, and narrow bug fix have different legitimate shapes. For each category, inspect the relationship between the patch vector and review minutes, escaped defects, reverts, and follow-up edits. A threshold earns enforcement only when it predicts an outcome the team actually cares about and reviewers can explain the alert.

When enforcement begins, keep an override path with a reason, owner, and expiry. Repeated exceptions can reveal an unrealistic threshold, missing task class, or harness that expands scope. Treat threshold changes like policy changes: version them, test historical fixtures, communicate the effective date, and retain the old result.

FAQ

Does a smaller AI-generated patch mean better code?

No. It may be easier to review, but it may omit necessary work or compress complexity.

Why count syntax units instead of lines?

Lines change with formatting. Syntax units are more stable, although they still do not measure semantics or readability.

Can teams use the public leaderboard to choose a model?

Use it to form a hypothesis. Choose from an internal frozen evaluation that reflects your repositories and review cost.

Should patch size ever block automatically?

An unexplained scope explosion can block pending review. The number alone should not.

Sources and further reading

• Parsimony benchmark and method — accessed Oct 18, 2026.
• La5u/Parsimony repository — code, data, limits, and license.
• Hacker News launch discussion.
• r/codex maintainer post.
• SWE-bench — benchmark context.
• Is Agent Code Less Maintainable Than Human Code?.
• ChainSWE.
• SWE-CC.
• Coding Agents in the Wild.
• Related: benchmark integrity.