The scarce resource is maintainer judgment
AI coding tools make generating a plausible patch cheap. Understanding whether that patch belongs in a project remains expensive. A useful policy manages that asymmetry: it requires the submitter to prepare enough evidence that a maintainer can review the change without reconstructing the entire agent session.
A new preprint posted August 4 gives the policy debate a larger evidence base. The TRACE study analyzed 29,624 GitHub repositories and identified 385 with AI policies. Its five dimensions are Transparency, Responsibility, Attribution, Constraints, and Enforcement. The authors report that policy adoption is associated with more disclosure, richer review interaction, maintainer engagement, and improved code-quality measures while AI-assisted contribution continues. They also report stronger outcomes for policies that emphasize transparency and responsibility than for restriction alone.
Those results are promising, but they are not a universal causal law. The paper uses repository mining, policy classification, propensity-score matching, and longitudinal difference-in-differences. Unobserved differences can remain, repositories vary sharply, and the preprint has not yet accumulated the scrutiny of a mature standard. Treat the findings as evidence for a design direction, then validate the direction against your own queue, defect rate, contributor experience, and review cost.
A second 2026 study found 118 AI policies among 1,000 popular repositories and documented wide variation in what projects permit, what they ask contributors to disclose, and where humans must intervene. A July benchmark goes one step further: it tests whether coding agents comply with repository rules under different steering conditions. Together, the research shifts the question from “Was AI used?” to “What observable obligations apply, and did this contribution satisfy them?”
Do not make maintainers prove that a contribution used AI. Make contributors prove that the change is reviewable, attributable, licensed, tested, and owned.
Turn TRACE into repository controls
TRACE is useful because its dimensions map to artifacts and gates. A policy that exists only as a paragraph in CONTRIBUTING.md is easy to miss and difficult to audit. Each dimension should produce evidence a bot can check or a reviewer can inspect.
| Dimension | Question | Repository control |
|---|
| Transparency | Was AI assistance used, and where? | Required PR field plus an Assisted-by trailer or equivalent project-approved metadata. |
| Responsibility | Who reviewed and owns the change? | Named human submitter, responsibility attestation, and human-only DCO or sign-off where applicable. |
| Attribution | Which tools and source artifacts shaped the result? | Agent/model details when known, task link, generated-artifact list, third-party snippets, and license notes. |
| Constraints | Which uses, paths, and change classes are allowed? | Protected paths, size limits, prohibited autonomous actions, secrets rules, and risk-tier routing. |
| Enforcement | What happens when evidence or controls are missing? | Failing status checks, auto-labeling, code-owner requirements, merge rules, and documented closure reasons. |
Existing project policies show what these ideas look like in practice. Linux kernel documentation says AI agents must not add Signed-off-by tags because only a human can certify the Developer Certificate of Origin. It recommends an Assisted-by: AGENT_NAME:MODEL_VERSION [TOOLS] trailer. OpenInfra likewise makes the human signatory responsible for the full contribution, including AI-assisted content, and explicitly connects that responsibility to license compatibility.
Ansible requires AI-assisted work to meet the same project, platform, code-of-conduct, and license rules as any other contribution. It also says autonomous maintainer-side uses such as releasing, testing, spam filtering, and AI contribution detection should be manually authorized. Open edX adds another important mechanism: a defined path for proposing a tool for review. That converts ad hoc tool adoption into a project decision.
The common pattern is pragmatic. The policy does not pretend a maintainer can infer provenance from code style. It binds a human to the submission, names evidence expectations, and preserves the project's normal quality bar.
Make the pull request an evidence contract
1. Repository advertises policyCONTRIBUTING.md, agent instruction files, PR templates, and machine-readable policy describe the same obligations.
2. Contributor prepares evidenceThe human identifies assistance, task provenance, affected artifacts, validation, license concerns, and known limits.
3. CI validates completenessStatus checks reject missing fields, forbidden paths, oversized changes, unapproved generated files, and absent tests.
4. Risk router assigns reviewRoutine changes use normal review; sensitive paths require code owners, security, legal, release, or domain review.
5. Human decides exact changeThe maintainer reviews the diff and evidence, asks questions, accepts responsibility boundaries, and merges or closes.
The evidence bundle should be proportional. A typo does not need a transcript dump. A generated parser, cryptographic change, dependency update, workflow edit, or new binary artifact needs more than “used Copilot.” Ask for the information that changes the review.
## AI assistance disclosure
- Assistance level: [none | completion | interactive | agent-generated]
- Human owner: @username
- Task source: fixes #1234
- Agent/model: Codex / model-version (if known)
- Affected artifacts: src/parser.ts, tests/parser.test.ts
- Third-party material: none / list source and license
- Validation performed:
- npm test -- parser
- npm run lint
- Independent checks: new regression fixture from issue report
- Known limitations: malformed UTF-16 input still rejected upstream
I reviewed the complete diff, can explain the change, and accept
responsibility for its correctness, provenance, and license compatibility.
Do not require private prompts or full conversation logs by default. They may contain credentials, customer data, unrelated source code, or personal information, and they are rarely an efficient review artifact. Prefer a structured manifest of inputs, tools, outputs, tests, and human decisions. Request deeper session evidence only when the risk and privacy model justify it.
A useful precedent appeared in a recent PostHog pull request: autonomous implementation PRs were changed to link their source issues so reviewers could trace the work back to the originating signal. That is a small control with high value. Provenance should survive the handoff from issue to research summary to agent task to patch.
Put enforceable rules beside the prose
The following example is intentionally tool-neutral. A repository can implement it with GitHub Actions, another CI system, a merge queue, or a small policy service. The important property is that the same rule drives contributor instructions and enforcement.
version: 1
ai_contributions:
allowed: true
disclosure_required_for: [interactive, agent-generated]
responsibility:
human_owner_required: true
agent_may_sign_dco: false
evidence:
required: [task_source, assistance_level, validation, limitations]
conditional:
dependency_change: [package_source, license, lockfile_diff]
generated_artifact: [generator, source_input, reproduction_command]
security_change: [threat_model, negative_test]
limits:
max_changed_lines_without_design_issue: 800
protected_paths:
- .github/workflows/**
- SECURITY.md
- CODEOWNERS
- package-lock.json
- deploy/**
routing:
protected_path: code-owner
dependency_change: supply-chain-review
security_change: security-review
enforcement:
missing_disclosure: fail
missing_evidence: fail
policy_conflict: close-with-reason
A simple validator can parse the pull-request body, list changed files, and emit one status check. Keep deterministic checks deterministic. Do not ask an LLM to decide whether a DCO field exists, whether .github/workflows/release.yml changed, or whether the patch exceeds a line budget.
Define policy precedence before an agent opens a pull request
Repository rules often arrive from several places: an organization-wide policy, a repository policy, a directory-specific contributor guide, and instructions embedded in the task. Those layers need an explicit order of precedence. A practical hierarchy is law and contractual obligations first, organization security policy second, repository policy third, directory-level rules fourth, and task instructions last. An agent should stop when two rules at the same level conflict; silently choosing the more convenient interpretation turns policy enforcement into guesswork.
The resolved policy should also be pinned to an identifier—ideally a commit hash or versioned policy bundle—in the pull-request manifest. That gives reviewers an answer when a repository changes its disclosure requirements while a long-running branch is still open. CI can evaluate the contribution against the pinned version, flag that a newer policy exists, and require a human to decide whether migration is necessary. This small piece of versioning prevents “the rules changed underneath me” from becoming an untraceable exception.
policy = load(".github/ai-contribution-policy.yml")
pr = github.pull_request()
assert_required_fields(pr.body, policy.evidence.required)
files = github.changed_files(pr.number)
for rule in match_rules(files, pr.body, policy):
require_evidence(rule.conditional_fields)
add_required_reviewers(rule.reviewers)
if changes_generated_artifact(files):
require_reproduction_command()
compare_generated_output_in_clean_checkout()
publish_status(check="ai-contribution-policy", result="pass")
An LLM can assist after this layer: summarize the evidence, flag inconsistent explanations, identify suspiciously unrelated changes, or propose questions. Its findings should be advisory unless independently verified. The policy engine owns blocking facts; maintainers own judgment.
Route by review cost and consequence
Disclosure is not a scarlet letter. It is a routing signal. If every AI-assisted patch receives maximum scrutiny, contributors will hide assistance and reviewers will burn out. If disclosure changes nothing, the field becomes ceremony. Tie it to the parts of the change that actually alter risk.
| Change class | Evidence | Review lane |
|---|
| Docs or small test cleanup | Task link, disclosure, normal tests | Normal maintainer review |
| Bug fix with regression test | Reproduction, failing-before/passing-after evidence, scope explanation | Domain maintainer |
| Generated code or assets | Generator version, source inputs, reproducible command, license | Generated-artifact owner |
| Dependency or lockfile change | Source, license, lifecycle scripts, advisory check, resolved diff | Supply-chain review |
| Auth, crypto, workflow, release, or security policy | Threat model, negative tests, privilege change, rollback | Required specialist/code-owner review |
| Large agent-generated feature | Accepted design issue, change map, staged commits, full verification | Split or reject before line review |
Reviewer backpressure must be explicit. A maintainer should be allowed to close a patch that creates disproportionate verification work, even if it compiles. GitHub's maintainer guidance describes the broader challenge as a loss of reliable signals and a mentorship problem. An August 6 r/opensource policy discussion made the same concern bluntly: submitting a machine-written body and expecting a human to read and debug it transfers work rather than contributing value.
Track median time to first useful review, reopened changes, defects after merge, percentage of disclosed assistance, missing-evidence failures, protected-path exceptions, and review time by change class. The goal is not maximum disclosure volume. It is a queue where disclosed contributions arrive with better evidence and cost no more to verify than comparable human-only changes.
Failure modes that make policy performative
| Failure | Why it fails | Better control |
|---|
| AI detector as gate | False positives punish style and false negatives invite evasion. | Enforce evidence, scope, tests, provenance, and human responsibility. |
| Disclosure with no consequence | The checkbox collects metadata but does not change review. | Route sensitive classes and require proportional evidence. |
| Full transcript requirement | Leaks private context and overwhelms reviewers. | Use a structured evidence manifest; escalate only when needed. |
| “Human reviewed” checkbox | Does not show what was verified or by whom. | Name the owner and record tests, source review, and known limits. |
| Model-name allowlist | Model identity does not guarantee provenance or correctness. | Validate outputs and restrict change classes, paths, and tools. |
| Policy hidden from agents | The coding agent never receives the repository obligations. | Mirror rules in contributor docs, PR templates, and agent-readable instructions. |
| No maintainer-side policy | Release bots and review agents can act without accountable approval. | Apply the same authorization, logging, and manual release rules to project automation. |
| Permanent ban with no exception path | Legitimate accessibility, translation, or analysis uses go underground. | Document allowed uses and a reviewable exception process when compatible with project goals. |
There are projects for which a ban is reasonable: legal uncertainty, safety-critical components, contributor-capacity limits, community values, or a deliberate human-authorship standard can all justify it. The enforcement lesson does not change. State the observable behavior that will be accepted, explain the disposition, and avoid pretending a style classifier can prove tool use.
Ship the policy in two weeks without freezing contribution
Measure the current queueRecord review time, closure reasons, defect escape, patch size, generated artifacts, and where provenance is already missing.
Choose a policy familyDecide whether the project permits assistance, restricts change classes, requires pre-approval, or prohibits it; write the reason plainly.
Define the evidence contractUse the smallest required fields that let maintainers understand task source, assistance, validation, provenance, and ownership.
Protect expensive pathsRoute workflows, dependencies, security, generated artifacts, release files, and large diffs to explicit owners.
Automate objective checksFail missing metadata, reproduction commands, tests, or protected reviewers through one explainable status check.
Expose rules to agentsKeep contributor prose, PR templates, machine-readable policy, and agent instructions consistent and versioned together.
Pilot before auto-closingRun the validator in warning mode, sample false positives, and publish examples before making it blocking.
Review the policy quarterlyCompare disclosed and undisclosed lanes, reviewer cost, contributor feedback, and defects; change rules when evidence changes.
Start with one outcome: no sensitive or substantial AI-assisted contribution reaches line review without a human owner, task provenance, validation evidence, and the right specialist lane. That is achievable without solving authorship detection, preserving every prompt, or forcing maintainers to become AI forensics investigators.
FAQ
Should an open-source project ban AI-generated contributions?
A ban can fit legal, safety, capacity, or community constraints. It remains difficult to verify from text alone. Write the allowed behavior, responsibility, and disposition clearly; enforce observable contribution requirements rather than relying on detection.
Can a repository detect whether code was written by AI?
Not reliably enough to make detection the control boundary. A repository can reliably check whether required metadata exists, sensitive paths changed, tests ran, generated artifacts reproduce, license notes are present, and required humans approved.
What belongs in an AI-assisted pull request?
At minimum: human owner, assistance level, task source, affected artifacts, validation, known limitations, third-party provenance, and responsibility attestation. Add model/tool details when known and relevant.
Does disclosure make a change safe?
No. It makes the change routable and auditable. Safety comes from scope, tests, specialist review, license compatibility, protected merge rules, and human judgment.
