Employee-level access is not the same as evaluator independence

Embedded evaluation moves assurance inside the development loop

Anthropic and Accenture announced an embedded-evaluation partnership on September 18. Faculty, Accenture's specialist AI business, will evaluate and red-team models, conduct alignment assessments, and test safeguards from inside Anthropic. The stated ambition is sustained access comparable to an employee's rather than a brief encounter with a frozen API candidate.

That access can close a real evidence gap. An external evaluator normally sees one checkpoint, a bounded interface, selected documentation, and a short test window. It may not see how the model changed during post-training, which safety decision was reversed, what incidents occurred in adjacent runs, or which internal deployment created the most consequential risk. An embedded team can follow those transitions, speak to engineers, observe governance, and test whether public commitments match operations.

The announcement is also unusually candid about what is missing. There is no settled standard for access, reporting, or long-term funding. Anthropic will fund Accenture's work directly while discussing separately funded pilots with METR and other nonprofit evaluators. The arrangement is non-exclusive, and Anthropic says more evaluators will follow. Those are pilot conditions, not a completed assurance system.

The timing matters. On the same day, NIST published its ARIA Evaluation Planning Manual, which combines model testing, red teaming, and user testing. Recent NIST work also documents agent cheating through solution contamination and grader gaming. Anthropic's own cyber-evaluation incidents show a different systems problem: a supposedly isolated environment reached the public internet, and model behavior plus infrastructure misconfiguration produced real unauthorized access. The evaluator must therefore examine the model, harness, environment, operator, and institution together.

Community discussion is substantial but polarized. The focused 30-day scan found 42 items across Reddit, Hacker News, GitHub, and Polymarket, including a nine-item exact launch cluster and a separate discussion of a public letter asking for stronger evaluator resources and protections. Skepticism focuses less on whether access is useful and more on incentives: who pays, who chooses scope, and who controls disclosure. That is the correct engineering question.

Independence is a set of capabilities, not a label

An evaluator can be legally separate and operationally dependent. A lab may provide office access while selecting the candidate, defining the question, hosting the tools, retaining every transcript, reviewing every finding, and controlling publication. The evaluator sees more than an outsider but cannot independently establish what happened. That is access without assurance.

Reverse the test. Ask what the evaluator can do when its finding is inconvenient. Can it select an intermediate checkpoint? Run a method the developer dislikes? Preserve raw evidence after access is revoked? Interview employees confidentially? Escalate a severe incident to a board committee or authority? Publish a qualified conclusion and the developer's disagreement as separate records? Leave without losing the evidence needed to defend its work? If any answer is unknown, independence is unknown.

Funding belongs in this table. Direct payment does not automatically invalidate a test, but it creates a dependency that must be visible and controlled. Disclose the amount and other commercial relationships; separate consulting and evaluation personnel; forbid contingent fees; rotate reviewers; require recusal for material financial interests; and let an oversight body inspect scope changes. METR's published process is useful because it names analytic autonomy, direct access, financial conflicts, and editorial independence rather than compressing them into “third party.”

Build six planes with separate owners

The embedded evaluator should not receive one broad “employee account.” It needs deliberately separated planes whose permissions correspond to a test purpose. This reduces both ceremonial access and uncontrolled exposure to model weights, customer data, or unrelated secrets.

1. Candidate plane       model/checkpoint IDs, policies, releases
2. Development plane     training decisions, changes, internal use
3. Evaluation plane      evaluator-owned tasks, tools, graders
4. Evidence plane        raw runs, transcripts, interventions, hashes
5. Incident plane        severity, containment, notification, correction
6. Disclosure plane      redaction, lab response, dissent, public report

Lab control: credentials, protected assets, remediation
Evaluator control: method, evidence record, finding, escalation
Joint record: exact candidate + environment + response receipt

The candidate plane prevents checkpoint substitution. Every result must bind to an immutable model identifier, system prompt, tool policy, inference configuration, safety stack, environment image, and code revision. “Latest model” or a marketing name is not enough. A lab can change inference parameters or safeguards without changing the public name.

The development plane gives context without surrendering judgment. Evaluators need change histories, relevant internal metrics, known incidents, and the reasoning behind safety decisions. They do not need unrestricted access to every employee file or customer record. Use purpose-bound datasets, read-only views, query logging, dual control for the most sensitive artifacts, and a documented path to request additional evidence.

The evaluation and evidence planes must remain separable from the lab. NIST's AITE uses sequestered data to reduce train/test contamination. Embedded teams need an analogous enclave: tasks and held-out materials the developer cannot silently inspect, plus an evidence store the evaluator can export. The lab can monitor security without being able to edit the record.

Put the rights into a machine-readable contract

A memorandum about “broad access” is too vague for daily work. Version the operating contract next to the evaluation plan. Policy code can then refuse a run whose candidate, permissions, custody, or disclosure state falls outside the agreement.

apiVersion: evaluation.independence/v1
engagement: frontier-loss-of-control-2026q4
evaluator: external-team-a
funding:
  payer: developer
  contingent_on_result: false
  other_commercial_relationships: [declared-partner-program]
scope:
  evaluator_selects_checkpoints: true
  method_autonomy: true
  employee_interviews: confidential
access:
  model_checkpoints: [intermediate, release_candidate]
  artifacts: [training-change-log, eval-transcripts, incident-register]
  prohibited: [customer-content, unrelated-source-code]
evidence:
  custodian: evaluator
  append_only: true
  export_on_exit: true
  retention_days: 2555
escalation:
  critical_hours: 4
  recipients: [lab-safety-lead, independent-board-chair]
publication:
  developer_fact_check_days: 10
  developer_veto: false
  dissent_allowed: true
  response_published_separately: true
exit:
  credentials_revoked_hours: 4
  evidence_preserved: true
  final_status_report: required

The contract must be enforceable. Before a run, the orchestrator resolves the exact candidate and grants time-bounded credentials. During the run, every access and intervention is logged to the evaluator store. When a critical signal appears, an independent clock starts. A lab response does not overwrite the finding; it attaches as a new signed object. At exit, credentials disappear while the evaluator retains the authorized record.

Separate facts from conclusions. A developer should be able to correct a wrong model identifier or explain an outage. It should not be able to convert “the mitigation failed in 3 of 20 adversarial trials” into “the safeguard performed as expected.” The final report should show the evidence, evaluator interpretation, lab response, unresolved disagreement, and any redactions with reasons.

Every finding needs a reproducible evidence manifest

Raw transcripts alone are not reproducibility. Agent results depend on tools, network, secrets, retry policies, graders, timeouts, intervention rules, and environment state. OpenAI's third-party evaluation playbook makes the same systems point: frontier models act inside workflows, so the environment is part of the result.

{
  "finding_id": "EV-2026-0919-014",
  "candidate": {"model": "sha256:...", "policy": "sha256:..."},
  "environment": {"image": "sha256:...", "network": "deny-by-default"},
  "task_set": {"version": "loss-control/v4", "sealed_hash": "sha256:..."},
  "tools": [{"name": "shell", "version": "3.2", "policy": "sha256:..."}],
  "grader": {"code": "sha256:...", "human_rubric": "rubric-7"},
  "runs": 120,
  "interventions": [{"run": 44, "actor": "safety-monitor", "reason": "egress"}],
  "artifacts": [{"uri": "evaluator://vault/run-44", "sha256": "..."}],
  "finding": "critical",
  "lab_response": "LR-2026-207",
  "signatures": ["evaluator-key", "timestamp-authority"]
}

Preserve failed runs and human interventions. Removing them selects for clean stories. Store prompts, tool results, stdout and stderr, network decisions, grader output, operator messages, and timestamps. If chain-of-thought is unavailable or restricted, record exactly which substitute trace was provided. Missing evidence should lower confidence instead of being filled by narrative.

Test the evaluator too. Seed known faults into a rehearsal candidate: a swapped checkpoint, contaminated task, permissive network rule, grader loophole, deleted transcript, and delayed incident. The engagement is not ready until the evaluator detects each one and the escalation route works.

Design the incident path before the first serious finding

Anthropic's cyber incidents show why evaluation and incident response cannot be separate binders. A model was instructed that it was in a simulation while a third-party environment was connected to the real internet. The proximate infrastructure error did not erase the model-behavior finding, and the model behavior did not erase the environment failure. The evidence system must preserve both causal layers.

OBSERVED -> TRIAGED -> CONTAINED -> PRESERVED -> ESCALATED
       -> DEVELOPER_RESPONSE -> INDEPENDENT_RETEST
       -> PUBLISHED | OPEN_DISAGREEMENT | CLOSED_WITH_LIMITS

Define severity from possible consequence, not reputational discomfort. A real external effect, exposed weight, credential use, disabled monitor, or evidence-integrity failure should trigger containment and independent escalation. The evaluator can recommend stopping a run; the lab remains responsible for safe operations and remediation. This avoids the fiction that an auditor assumes the developer's operational duty.

For each incident, record awareness time, affected candidate, environment, external systems, data, actions, containment, evidence gaps, competing hypotheses, owner, notification decision, corrective action, and retest. Publish enough for outsiders to distinguish a prompt failure, model behavior, harness defect, evaluator mistake, and governance failure.

Failure modes that make “independent” evaluation decorative

Do not use this article as proof that one named partnership is captured or successful. The program has barely started, exact community discussion remains early, and many operating details are explicitly unresolved. The practical response is to make those details inspectable before confidence hardens into branding.

Acceptance checklist for an embedded evaluation pilot

• Publish the evaluator's legal identity, funding, other commercial relationships, and conflict policy.
• Bind every result to an immutable model, safeguard, harness, environment, tool, and grader state.
• Give the evaluator authority to select checkpoints and methods inside the agreed risk scope.
• Provide confidential staff interviews and a documented route for requesting missing evidence.
• Store raw runs, interventions, incidents, and hashes under evaluator-controlled custody.
• Separate developer fact correction from interpretation, severity, and publication decisions.
• Predefine critical escalation recipients, clocks, containment authority, and notification rules.
• Allow a minority or dissenting conclusion and publish the developer response separately.
• Drill checkpoint substitution, evidence deletion, network escape, grader gaming, and evaluator exit.
• Reapprove scope when the model, internal use, safeguards, or funding relationship materially changes.

A good first pilot is bounded. Choose one risk domain, two candidate states, a small set of evaluator-owned tasks, a sealed evidence store, and one public report. Measure evidence access latency, unresolved requests, method changes requested by the developer, incidents detected, retest reproducibility, disclosure delays, and dissent. Those metrics reveal whether the institution works before it scales.

FAQ

What is an embedded AI evaluator?

It is an evaluator that works inside an AI developer with sustained access to models, checkpoints, training decisions, safeguards, incidents, and staff. The objective is to observe development rather than only test a selected final candidate.

Does employee-level access make the evaluator independent?

No. Access improves observability. Independence requires control over methods, evidence, conflicts, escalation, conclusions, publication, dissent, and exit.

Should evaluators receive unrestricted lab access?

No. Access should be sufficient, purpose-bound, logged, and reviewable. Sensitive model weights, customer data, and unrelated systems need least-privilege controls. The evaluator must also be able to prove when required evidence was withheld.

Can developer-funded evaluation be credible?

It can produce useful evidence if funding is disclosed and insulated from findings, commercial conflicts are controlled, methods remain autonomous, and publication cannot be vetoed. Pooled or public funding can reduce dependence but does not replace technical controls.

What should the public report include?

Include exact candidate identity, scope, methods, access limits, environment, evidence completeness, results, uncertainty, incidents, unresolved disagreements, redactions, lab response, and evaluator signatures.