Give every consequential agent run an execution receipt

A trace tells a story; a receipt defines an acceptance claim

A current r/AI_Agents discussion asks the production question directly: how do you know an agent did what it says it did? The strongest answer is not a longer natural-language summary. It is a machine-checkable record that lets a different system reject the run when required evidence is absent, inconsistent, or altered.

Distributed tracing is the right foundation. It gives each run a trace identifier, nests work into spans, records timing and errors, and connects an agent operation to model calls, retrieval, tool calls, databases, queues, and downstream services. OpenTelemetry's developing GenAI conventions now name agent creation, agent invocation, workflows, planning, and tool execution. That shared vocabulary reduces vendor-specific parsing and lets existing collectors and observability backends understand agent work.

But an operational trace is optimized for debugging and monitoring. It may be sampled. A framework may omit attributes. A span can say a tool returned success without proving which external state the service observed. Content may be deliberately excluded for privacy. The collector or agent may be able to rewrite stored data. A green span status does not prove the business outcome was correct. None of those limitations makes tracing useless. They define the additional contract a receipt must satisfy.

Agent observability is becoming a shared protocol, not only a dashboard

The OpenTelemetry project has moved GenAI definitions into a dedicated semantic-conventions repository. Its agent document is marked Development, an important boundary for implementers: the concepts are useful, but attribute names and requirements can still change. Pin the semantic-convention version or commit used by each instrumentor and translate it through one adapter instead of scattering raw gen_ai.* strings across an application.

The current conventions distinguish an invoke_agent span from a plan span and direct tool-using implementations to an execute-tool span. Provider-specific systems are also exposing detailed traces. The OpenAI Agents SDK records generations, tool calls, handoffs, guardrails, and custom events, while OpenSearch documents an Agent Traces interface driven by OpenTelemetry fields. Google Cloud's agent instrumentation likewise consumes OTel-formatted GenAI spans and events. Interoperability is becoming plausible.

Community objections show where dashboards stop. In an August 7 r/OpenTelemetry thread about end-to-end tests from traces, a practitioner warned that traces often lack enough information to reproduce concurrency and external behavior. A separate August 15 discussion about verifying agents converged on adversarial review rather than trusting the final claim. Research on execution provenance makes the same distinction more formally: state checkpoints support recovery, traces support observability, and provenance or receipts support an acceptance argument.

Search intent is also specific. Google Autocomplete returned “AI agent observability,” “tools,” “open source,” “OpenTelemetry,” “evaluation,” and “governance.” This is not evidence of mass-market popularity, and the topic did not appear in the current US Daily Search Trends RSS. It is evidence that practitioners are trying to connect runtime visibility with evaluation and control.

Separate telemetry, evidence assembly, and verification

Do not let the agent create its own receipt after the run from memory. Build it from events observed at enforcement boundaries: the gateway that admitted the task, the policy engine that allowed or denied a tool call, the service that received a write, the test system that evaluated the outcome, and the human approval service. The agent can add an explanation, but that explanation is a claim inside the receipt, not the receipt's authority.

Define the acceptance claim before instrumenting. “The agent ran” needs little evidence. “The agent prepared a patch in an isolated workspace and all required tests passed” needs repository and environment digests, write-scope enforcement, patch hash, test identities and results, and release approval. “The agent refunded a customer correctly” additionally needs authorization, request binding, amount and currency, idempotency, service-side response, ledger reconciliation, and an appeal route.

Make the receipt small, versioned, and explicit about gaps

A useful schema separates observed facts from assertions. Store bulky or sensitive content behind controlled references and bind each reference with a digest. Include a completeness section so “not captured” cannot be confused with an empty value.

receipt_version: agent-receipt/v1
claim: "workspace patch passed bounded release checks"
run:
  id: run_01J...
  trace_id: 4bf92f3577b34da6a3ce929d0e0e4736
  started_at: 2026-08-16T02:14:18Z
  ended_at: 2026-08-16T02:18:43Z
subject:
  agent_id: coding-agent
  agent_version: sha256:7b1...
  model: {provider: example, requested: model-x, observed: model-x-202608}
task:
  request_digest: sha256:1f0...
  repo_commit: 92d4...
  workspace_digest: sha256:aa8...
policy:
  bundle: agent-write-policy@sha256:40c...
  decisions: [read_repo: allow, write_workspace: allow, network: deny]
actions:
  - seq: 17
    tool_call_id: call_83a
    tool: run_tests
    request_digest: sha256:61b...
    response_digest: sha256:9cd...
    receiver_receipt: refs/test-service/rcpt_77
outcome:
  patch_digest: sha256:c81...
  checks: {unit: pass, security: pass, scope: pass}
  verifier: release-gate@sha256:0d9...
approval:
  decision: approved
  actor: team:payments-maintainers
integrity:
  previous_receipt: sha256:000...
  merkle_root: sha256:ded...
  signatures: [sigstore:...]
completeness:
  required_fields_missing: []
  content_capture: hashed_reference_only

Use an ordered action sequence even though trace spans form a tree. Sequence numbers make authorization and state transitions easier to audit. Preserve parent span identifiers as well, but do not assume timestamps alone establish causality: clocks drift, asynchronous jobs overlap, and retried requests arrive out of order. Bind a tool response to the exact request identifier and idempotency key observed by the receiving service.

Record the requested and observed model when the provider exposes both. A stable route name can point to a different snapshot tomorrow. Version prompts, tools, policies, retrieval indexes, and environment images separately. The receipt should reveal which component moved without forcing reviewers to hash an entire deployment as one opaque blob.

Use OpenTelemetry as the transport vocabulary

Map the receipt to trace events rather than replacing tracing with a proprietary log. The top-level run becomes an agent or workflow span. Planning remains a separate child or sibling only when the instrumentation can identify planning reliably. Each external action gets an execute-tool span plus the normal HTTP, database, messaging, or RPC spans emitted by the called service.

from opentelemetry import trace

tracer = trace.get_tracer("agent-runtime", "3.4.0")

with tracer.start_as_current_span("invoke_agent release_patch") as run:
    run.set_attribute("gen_ai.operation.name", "invoke_agent")
    run.set_attribute("gen_ai.agent.name", "release_patch")
    run.set_attribute("gen_ai.agent.version", AGENT_DIGEST)
    run.set_attribute("app.receipt.version", "agent-receipt/v1")
    run.set_attribute("app.policy.digest", POLICY_DIGEST)
    run.set_attribute("app.task.digest", TASK_DIGEST)

    with tracer.start_as_current_span("execute_tool run_tests") as tool:
        tool.set_attribute("gen_ai.operation.name", "execute_tool")
        tool.set_attribute("gen_ai.tool.name", "run_tests")
        tool.set_attribute("gen_ai.tool.call.id", call_id)
        tool.set_attribute("app.request.digest", request_digest)
        result = test_service.run(scoped_request)
        tool.set_attribute("app.response.digest", result.digest)
        tool.set_attribute("app.receiver.receipt_id", result.receipt_id)

The app.* fields are an application contract, not proposed OpenTelemetry standard fields. Keep them in a documented namespace and version them. Promote only low-cardinality, queryable values to span attributes. Put large lists, test logs, screenshots, diffs, and sensitive payloads in governed artefact storage, referenced by digest and retention class.

Bind actions at the receiver and keep replay claims narrow

If the same component executes an action and writes the only log of that action, the evidence proves what the component reported. It does not prove what the receiver saw. The Notarized Agents paper proposes inverting that trust boundary: the service receiving an agent call signs a confidential receipt of its observation. You do not need a public transparency log for every internal workflow, but the architectural lesson is sound. Capture high-consequence evidence at the service that enforces the action.

For an email tool, the receiver receipt can bind message digest, recipients, sending identity, policy decision, provider message ID, and acceptance time. For a database migration, bind migration digest, database identity, transaction or change identifier, affected schema version, and rollback evidence. For a browser agent, bind the requested action to screenshots or accessibility-tree hashes before and after the change, plus the site response identifier where available.

A hash only detects change if the verifier trusts the original commitment and canonicalization. Define byte encoding, field order, Unicode normalization, excluded fields, and digest algorithm. Chain receipts or place their hashes in append-only storage controlled separately from the agent runtime. Rotate signing keys through an auditable process and record key identifiers and verification time.

Replay is a debugging tool, not a promise that a stochastic system will produce the same reasoning. Preserve the environment manifest, model snapshot when available, sampling settings, tool definitions, request cassettes, state checkpoints, clocks, random seeds, and allowed network responses. A replay verifier should state whether it reproduced the same requests, the same observed responses, the same artefact digest, or merely an equivalent accepted outcome. Those are different claims.

Capture less content, but make omissions visible

OpenTelemetry warns that input and output messages can contain personal or sensitive information. Tool arguments can include credentials, customer records, source code, or medical and financial data. Full content capture should be an explicit exception, not a default checkbox enabled because debugging is easier.

Use separate telemetry and evidence access. SRE may need latency and failure classes without access to customer content. Security may need tool scope and policy denials. A qualified reviewer may temporarily access a restricted artefact. The receipt should carry a classification and reference, not copy the same sensitive payload into every backend.

Make verification fail closed on missing evidence

def verify(receipt, policy, keyring, artefact_store):
    require_schema(receipt, policy.receipt_version)
    require_no_missing_fields(receipt.completeness, policy.required_fields)
    verify_signatures(receipt.integrity, keyring)
    verify_chain(receipt.integrity.previous_receipt)
    require_digest(receipt.policy.bundle, policy.approved_bundle)
    require_scope(receipt.actions, policy.allowed_tools, policy.allowed_targets)
    for action in receipt.actions:
        verify_receiver_receipt(action.receiver_receipt, action.request_digest)
    verify_artefact(receipt.outcome.patch_digest, artefact_store)
    require_checks(receipt.outcome.checks, policy.required_checks)
    require_approval(receipt.approval, policy.approver_roles)
    require_not_expired(receipt, policy.max_age)
    return "ACCEPT"

Do not return “warning” for a missing field that the acceptance claim requires. Reject and route to a bounded remediation lane. A missing optional latency metric should not block a security claim; a missing policy digest or receiver receipt should. Define requirements per task and consequence instead of building one universal receipt with hundreds of fields.

Test the verifier with adversarial fixtures: remove a critical span, change a tool result after signing, substitute a model alias, reuse an approval on a different task digest, reorder actions, break the chain, provide a valid trace with no outcome check, and include a secret in a supposedly redacted artefact. The receipt format is only as useful as its rejection tests.

Failure modes that make polished traces misleading

Implement one receipt for one consequential action

1. Choose a bounded action with a clear outcome, such as merging a patch, issuing a refund, changing an account, or sending an external message.
2. Write the acceptance claim and list the minimum facts an independent verifier needs. Mark which system can authoritatively observe each fact.
3. Instrument the agent and tools with OpenTelemetry. Pin the GenAI semantic-conventions and instrumentor versions.
4. Create an application receipt schema that records task, agent, model, policy, action, outcome, approval, integrity, and completeness.
5. Replace raw content with hashes and controlled references unless an approved debugging or review case requires redacted content.
6. Add receiver-side evidence for the highest-consequence action and an independent oracle for the claimed outcome.
7. Store receipt commitments outside the agent's write boundary. Define canonicalization, signature, timestamp, retention, and key-rotation rules.
8. Write negative tests that remove, alter, replay, or mismatch required evidence. Make the release gate reject every fixture.
9. Run in shadow mode, compare verifier decisions with expert review, and measure missing-evidence, false-reject, incident-debug, and reviewer-time rates.
10. Expand only after the receipt proves useful. Different action classes should have different required fields and approvers.

Frequently asked questions

What is an AI agent execution receipt?

It is a compact, versioned evidence object that binds a run to the exact task, inputs, policies, versions, actions, observed results, approvals, outcome checks, and integrity commitments needed for a specific acceptance claim.

Is an OpenTelemetry trace an audit trail?

It can be part of one. A normal trace may be sampled, incomplete, mutable, or missing receiver and outcome evidence. Add completeness requirements, policy bindings, independent observations, integrity checks, and a verifier before calling it proof.

Should traces include chain-of-thought?

No. Verify observable inputs, decisions, tool requests, policy results, external responses, artefacts, outcomes, and human approvals. Hidden reasoning is not required and can create privacy, security, and reliability problems.

Does a signed receipt prove the action was correct?

A signature proves that a named key committed to specific bytes. Correctness still depends on who controlled that key, which facts were observed, whether required fields were complete, and whether an independent outcome oracle passed.

Can the same receipt support deterministic replay?

It can preserve a replay envelope, but a live rerun may diverge. State what replay reproduced: requests, responses, artefact digest, or an equivalent accepted outcome. Do not promise deterministic reasoning from a stochastic model.