More prompts can produce less security
An approval dialog can transfer a decision to a human. It cannot make the decision understandable, limit the consequences of a mistake, or prove that the action shown in the dialog is the only action that will occur. Treating the click as a hard boundary confuses visible consent with enforced capability.
The most useful current evidence is uncomfortable. Anthropic reports that users approved roughly 93% of Claude Code permission prompts. Its engineering team observed that attention fell as prompt volume increased, and sandboxing reduced prompts by 84%. A separate Scale X permission game, discussed widely on Hacker News in early August, reports that the average player missed one in three simulated threats. The game is a self-selected, non-peer-reviewed exercise, so its headline number is not a universal human-error rate. It is still a concrete demonstration of the failure mode: even people who know a task contains traps begin pattern-matching instead of investigating.
Recent academic work points in the same direction. A review of 59 papers, 21 production agent systems, and 26 security plugins found that runtime approval, scope configuration, and policy specification dominate deployed defenses, while the authors describe a persistent tradeoff between cognitive burden and security guarantees. Another study followed 400 repeat reviewers across 11,429 reviews of agent-generated pull requests. Approval increased while inline comments decreased. The authors cannot prove habituation caused the shift, and they explicitly note possible confounding from improving agent quality. The operational warning remains: a team cannot assume review depth stays constant as volume rises.
The right conclusion is not “remove humans.” Humans remain necessary for intent, accountability, exceptions, and business judgment. The conclusion is narrower and more useful: do not ask a human to compensate for controls a machine can enforce more reliably.
A permission prompt should answer a difficult business question. It should not be the only thing standing between an untrusted document and a production credential.
Three findings change the design
1. Approval quality decays under repetition
A prompt has information value only when it is rare enough to interrupt routine and complete enough to support judgment. “Allow npm run build?” hides the files changed before the command, the scripts referenced by package.json, the environment variables available to child processes, and the network destinations reachable during the build. Approving that line is not reviewing the resulting capability graph.
Prompt count is therefore a security metric, not merely a usability metric. When a normal task produces dozens of prompts, the product has externalized policy design to the user. The likely outcomes are broad “always allow” grants, quick approval streaks, or migration to a bypass flag. Each outcome reduces the review the prompts were supposed to create.
2. The dangerous step may happen before or after the dialog
Anthropic describes project configuration that once loaded before a folder-trust prompt. The visible trust decision arrived after untrusted local state had already influenced execution. At the other end of the sequence, an agent can modify a script without asking, then request approval for a familiar build command that runs the changed script. The approved command is benign only if the reviewer also verifies its transitive inputs.
Prompt injection through GitHub issues, pull-request titles, README files, web pages, and tool output creates the same structural problem. The agent sees attacker-controlled text inside a task that otherwise looks legitimate. Pillar Security documented a path from untrusted issue content into a privileged GitHub workflow and linked the resulting Google security advisory. The connector may be trusted; the data returned through it is not.
3. Containment limits damage even when judgment fails
Anthropic's strongest engineering claim is architectural: constrain what the agent can reach with filesystem boundaries, virtual machines, scoped credentials, and egress controls. Model safeguards and approval classifiers remain probabilistic. Environment controls can make a forbidden path unavailable. If production credentials never enter the agent runtime, a prompt injection cannot exfiltrate them. If the egress proxy denies unknown destinations and account contexts, an approved shell command cannot silently open an arbitrary outbound channel.
NIST's agent-security work and joint guidance from Australia, the United States, the United Kingdom, Canada, New Zealand, and Germany converge on familiar security fundamentals adapted to agents: least privilege, strong identity, continuous monitoring, testing, and explicit human oversight. The novelty is not that agents abolish cybersecurity. It is that natural-language inputs and autonomous tool selection connect untrusted content to traditional capabilities at high speed.
Trace the whole action, not the line in the dialog
1. Untrusted inputAn issue, repository, email, tool result, webpage, or retrieved document enters agent context.
2. Model planThe agent translates mixed trusted and untrusted text into file edits, shell commands, API calls, or sub-agent tasks.
3. Hidden state changeThe agent edits a script, configuration file, workflow, lockfile, memory file, or instruction file inside its allowed workspace.
4. Familiar approvalThe user sees a plausible command or publish action without a complete diff, provenance map, or capability summary.
5. Transitive executionThe command reads credentials, reaches a network, changes infrastructure, or persists instructions beyond the visible task.
This chain explains why command blocklists are weak by themselves. The same effect can be encoded, split across commands, delegated to another tool, hidden in a generated file, or reached through a permitted domain. It also explains why “the user approved it” is an incomplete incident analysis. The question is whether the system presented a reviewable decision and enforced a reasonable maximum consequence if the decision was wrong.
| Task | Weak gate | Stronger control |
|---|
| Install a dependency | Approve the package-manager command | Curated registry, lockfile diff, lifecycle scripts disabled, isolated build, outbound deny |
| Fix a public issue | Trust issue text because it came through GitHub | Mark issue content untrusted, use a read-only triage agent, pass structured facts to the editing agent |
| Run tests | Approve npm test | Inspect changed scripts, run in a disposable workspace, inject no production secrets, capture network and file writes |
| Deploy a patch | Approve one CLI command | Separate deploy identity, signed artifact, policy check, staged environment, rollback, named release owner |
| Use an MCP tool | Approve the server once | Pin local code or continuously review remote service, scope each method, validate returned content, log calls |
Build five boundaries before the human gate
Resource boundary
Give the agent a copied or disposable workspace. Mount only required paths. Prefer read-only mounts for source material and explicit write targets for generated artifacts. Resolve symlinks before validating paths. Keep shell startup files, personal documents, browser profiles, SSH material, cloud credentials, and global tool configuration outside the mount.
Identity boundary
An agent should not inherit a developer's full identity. Issue a short-lived principal for one task, repository, environment, and method set. Separate read, write, merge, and deploy identities. A task that summarizes issues does not need permission to edit them; a task that opens a pull request does not need permission to merge it.
Network boundary
Deny outbound access by default, then grant destinations and operations deliberately. Domain allowlists are insufficient when a permitted service supports attacker-controlled accounts or file uploads. Treat an allowed destination as a bundle of capabilities: which endpoint, method, tenant, token, content type, and response size are permitted?
Change boundary
Agents should produce staged changes that are easy to diff and reverse. Keep generated edits on an isolated branch, worktree, patch set, or transaction. Require independent checks before merge or release. Protect workflow files, agent instructions, package manifests, deployment configuration, and credential-related code with stricter ownership rules than ordinary source files.
Evidence boundary
Capture tool calls, policy decisions, identity used, files changed, network attempts, tests run, artifact digest, reviewer decision, and final outcome. Do not rely on the model's prose recap. Logs should come from the enforcement layer and remain available even if the agent fails, is compromised, or summarizes selectively.
Put the boundary in policy, not in reviewer memory
The following example is illustrative. Its point is separation: deterministic rules decide what the runtime can do; the approval service handles the narrow remainder; the release system decides what can reach production.
agent_task:
id: issue-fix-1842
workspace:
source: repo-snapshot:sha256:8f3...
writable: ["/workspace/src", "/workspace/tests"]
denied: ["**/.github/workflows/**", "**/AGENTS.md", "**/.env*"]
identity:
principal: agent/pr-writer
ttl: 45m
permissions: ["repo:read", "branch:write", "pull_request:create"]
denied: ["pull_request:merge", "secrets:read", "deployment:create"]
network:
default: deny
allow:
- host: registry.npmjs.org
methods: [GET]
integrity_required: true
execution:
container: ephemeral
production_credentials: false
record_file_writes: true
approval_rules:
- when: "new_dependency OR protected_path_change OR external_write"
require:
- capability_summary
- complete_diff
- provenance
- rollback_plan
reviewer: code_owner
release_gate:
require: [tests_passed, policy_passed, artifact_digest, human_owner]
deploy_identity: separate
A reviewer can now answer a bounded question: should this exact diff, created from these inputs, receive this additional capability for this duration? They are not being asked to interpret every shell command the agent invents. If the requested capability contradicts the task, the system can reject it before a person is interrupted.
Spend the approval budget on high-information decisions
Every team has a limited attention budget. Use it for actions with high consequence, ambiguous intent, or policy exceptions. Low-level repeated decisions should become enforced policy. A useful prompt contains the task goal, requested capability, affected resources, complete change summary, provenance of untrusted inputs, predicted consequences, tests, and rollback. It should never use urgency, default the risky choice, or hide “always allow” beside a one-time grant.
| Decision | Machine should enforce | Human should decide |
|---|
| Read outside the workspace | Deny unless path is explicitly mounted | Whether the task genuinely requires a new mount |
| Send data externally | Destination, method, tenant, and data-class policy | Exceptional disclosure purpose and accountable owner |
| Modify protected files | Block or require code-owner lane | Whether the architectural or workflow change is appropriate |
| Merge or deploy | Tests, signatures, policy, artifact digest, environment separation | Business readiness, residual risk, and release timing |
| Override a failed check | Prevent silent bypass; record exception | Time-bounded risk acceptance by an authorized owner |
Measure whether the gate remains healthy. Track prompts per completed task, percentage approved, median decision time, “always allow” use, repeated approval streaks, requests missing a diff or provenance, reviewer concentration, reversals, incidents after approval, and changes caught at the final release gate. A rising approval rate is not automatically bad, but rising approval combined with lower review evidence and more downstream corrections should trigger investigation.
Rotate reviewers for repetitive agent output, sample approved work for secondary inspection, and show reviewers their own approval trajectories. The recent habituation study recommends rotation, streak audits, and trend dashboards. Those controls should be tied to defects and incidents, not used to punish fast reviewers or manufacture activity.
Failure modes that survive a permission dialog
| Failure | Why the dialog misses it | Control |
|---|
| Prompt fatigue | Routine approvals train the reviewer to click | Reduce prompt count; reserve prompts for exceptional capabilities |
| Transitive script execution | The visible command looks familiar | Show dependency and file diffs; run inside a constrained environment |
| Pre-consent configuration | Untrusted files are parsed before trust is established | Defer all project-local execution and config loading until after validation |
| Trusted connector, poisoned data | The integration is approved, but returned content is attacker-controlled | Label provenance, isolate retrieval, and sanitize structured handoff |
| Allowed-domain exfiltration | The destination is approved but the operation or account is not | Method, tenant, token, and content-aware egress proxy |
| Shared developer credential | One mistake inherits broad standing access | Per-task short-lived agent identity with no production secret |
| Review of the wrong version | The artifact changes after approval | Bind approval to a digest and invalidate it on material change |
| Model-authored audit narrative | The agent can omit the evidence of its own failure | Enforcement-layer logs and independent verification |
A practical rollout checklist
Inventory capabilitiesList filesystem, shell, network, APIs, connectors, credentials, memory, and release authority for every agent mode.
Classify inputs by trustTreat issues, PR text, repositories, webpages, messages, retrieved documents, and tool output as untrusted unless proven otherwise.
Remove ambient credentialsUse task-specific short-lived identity and keep production secrets outside the runtime.
Enforce containmentUse a disposable environment, narrow mounts, symlink-safe path checks, and deny-by-default egress.
Protect sensitive filesRequire a stricter lane for workflows, instructions, manifests, deployment configuration, and security policy.
Design the approval payloadShow goal, capability, complete diff, provenance, consequences, verification, expiry, and rollback.
Separate release authorityThe agent that produces a change cannot silently merge or deploy it with the same identity.
Instrument review healthTrack prompt volume, approval streaks, evidence completeness, reversals, incidents, and downstream defects.
Run adversarial drillsSeed poisoned issues, modified scripts, encoded commands, symlinks, approved-domain exfiltration, and persistent instruction files.
Preserve rollbackMake changes versioned, staged, digest-bound, and reversible before increasing autonomy.
Start with one high-volume agent workflow and capture every prompt for a week. Classify each prompt: deterministic policy, missing product context, genuine business judgment, or avoidable noise. Move the first two categories into the control plane. Improve the payload for the third. Delete the fourth. Then repeat after measuring whether task completion, false blocks, incidents, and reviewer effort improved.
FAQ
Are human approvals useless for AI agents?
No. Humans should approve consequential exceptions, business commitments, release readiness, and residual risk. They should not be asked to simulate a filesystem sandbox, inspect obfuscated shell, or remember every allowed network capability.
Is a sandbox enough?
No. A sandbox limits reachable resources, but mounted data can still be damaged or exfiltrated through permitted paths. Combine isolation with scoped identity, egress policy, input provenance, staged changes, and independent logs.
Should a team use automatic command approval?
Only inside a well-defined containment boundary. A probabilistic approval classifier can reduce interruption, but it will miss some risky actions. It should never replace resource isolation or a release gate.
What is the most important first change?
Remove standing production credentials from the agent runtime. That single step reduces the maximum consequence of prompt injection, mistaken intent, malicious repositories, and an incorrect approval.
How does this relate to pull-request review?
The same attention problem appears at a larger unit. Keep agent changes small, require tests and policy checks, rotate repetitive review, audit approval streaks, and bind merge approval to the exact commit.
