LLM serving security | Updated October 27, 2026

A cache hit is useful. A cache page owned by the wrong request is a breach.

Cloudflare's GLM 5.3 incident shows why prompt-cache economics and runtime KV isolation cannot share one vague dashboard. A production contract must prove admission, transfer, lifetime, and read ownership - then fail before decode when any proof is missing.

155 affected responsesPrefill/decode isolationSGLang v0.5.22+Two-tenant canariesSources checked Oct 27
A disaggregated LLM serving path with request-owned KV pages and fail-closed checks

The incident was an ownership failure, not a bad answer

Between August 27 and September 9, 2026, a small number of requests to Cloudflare's hosted GLM 5.3 model received text derived from other customers' recent requests. Cloudflare identified 155 affected responses. The common trigger was a prompt longer than the 681,280-token capacity of the prefill KV pool, even though the model advertised a 1,048,576-token context window. The inference engine should have returned an error. Instead, a rejected request crossed the handoff boundary and decode attended over stale pages.

The disclosure is unusually useful because it ties customer impact to a concrete serving path. Cloudflare ran GLM 5.3 on SGLang with disaggregated prefill and decode. Prefill processed the prompt and sent KV state over RDMA to decode nodes. When an over-capacity request was rejected, SGLang replaced it internally with a one-token placeholder. Prefill wrote one page, signalled transfer completion, and decode assumed the full reserved region belonged to the current request. Most pages still held state from earlier requests.

The client received HTTP 200. Reported prompt tokens matched the long input. The answer arrived faster than a full prefill should have taken and could contain prior-user content. Ordinary API success, a coherent response, and plausible token accounting all failed as security indicators. Isolation must be asserted below the response layer.

Cloudflare disabled the model within 40 minutes of declaring the incident, enlarged the pool, patched the rejection path, reproduced the fix with the researcher's script, ran canary-marker tests, and restored service the same day. It found no evidence of deliberate exploitation and reported no other Workers AI model affected. Those scope statements are about Cloudflare's investigation; they are not proof that every SGLang topology or cache design is safe.

Capacity headroom prevented the known trigger. Request ownership is the control that must survive the next trigger.

Separate three cache layers before designing controls

“Prompt cache” often refers to several mechanisms with different operators, evidence, and threat models. Product-level prompt caching reuses an identical prefix and exposes cached-token usage or billing. A distributed prefix cache can persist reusable state across workers. The live KV pool contains attention keys and values that the serving runtime allocates, transfers, reads, and releases during inference. A team can have correct API prefix reuse while the runtime mishandles live pages, or excellent runtime ownership while an application leaks information through unsafe cache keys.

LayerPurposePrimary evidenceCore isolation question
Application/API prompt cacheReuse stable request prefixesPrefix hash, cached-token usage, provider diagnosticsWhich account, organization, user, or tenant may reuse this entry?
Distributed prefix storeMove reusable state across engines or storage tiersCache key, namespace, TTL, encryption, transfer receiptCan another principal infer or retrieve the entry?
Runtime KV poolServe live attention stateReservation, page generation, request ID, transfer count, release stateDoes every page read by decode belong to this request now?

Timing side channels are different from direct content exposure. NDSS research and later cache studies show that shared reuse can reveal information through latency even when the victim's tokens never appear in a response. TensorRT-LLM documents cache salting so reuse can be scoped to a user or tenant. Those controls address cross-principal reuse; they do not replace lifecycle correctness inside a disaggregated engine.

The earlier version of this guide focused on canonical prompt prefixes, cache-miss diagnostics, and cost per accepted task. Those practices remain valid. The October incident adds a stronger invariant: a stable prefix is not safe merely because it is reusable. Every cache layer must have an explicit principal, ownership proof, invalidation path, and fail-closed read rule.

How one rejected prompt crossed the prefill/decode boundary

request R2: 800k tokens
        |
        v
prefill admission reserves 8 pages
        |
  pool fits only 681,280 tokens
        |
        v
R2 rejected, input becomes 1-token placeholder
        |
prefill writes P0, then signals "transfer complete"
        |
        v
decode accepts the 8-page reservation
        |
P0 = R2 placeholder state
P1..P7 = stale state from earlier request R1
        |
        v
attention reads all pages -> HTTP 200 with cross-request content

This chain exposes four independent contracts. Capacity must prove that an accepted context fits the KV pool on every relevant role. Admission must retire a rejected request before bootstrap, allocation, or transfer. Handoff must prove that the sender wrote the exact token/page count the receiver reserved. Lifetime must keep a page unavailable for reuse while late writes or abort work can still arrive.

SGLang's coordinated disclosure describes fixes across those boundaries. Pull request #36988 retires aborted requests when the final prefill result is processed. #38935 rejects intake failures earlier. Router changes pair cancellation across both halves. Deferred KV release prevents decode pages from being reassigned while writes may remain in flight. A transfer checksum verifies the handoff. These are defense-in-depth measures; no single one is a universal ownership proof.

Cloudflare recommends SGLang v0.5.22 or later, or backporting the relevant patches. Version alone is not acceptance. Serving behavior depends on model architecture, quantization, tensor parallelism, context parallelism, CUDA graph capture, memory fraction, transfer backend, router, and proxy timeouts. The release must be tested as an exact serving graph, not as a package name.

Make cache ownership a versioned serving contract

A release manifest should connect the public model contract to the physical resource contract. Record the advertised context window, measured prefill and decode capacities, page size, engine commit, router commit, transfer backend, model/quantization, parallelism, deferred-release state, checksum/canary coverage, and rollback image. Capture the startup log that proves capacity; do not copy a number from staging or a different quantization.

apiVersion: inference.example/v1
kind: CacheIsolationContract
metadata:
  id: glm53-workers-pd-r27
spec:
  model: zai-org/glm-5.3
  engine: sglang@v0.5.22
  topology: {prefill: 8xGPU, decode: 8xGPU, transfer: mooncake-rdma}
  context_tokens: 1048576
  measured_capacity:
    prefill_min_tokens: 2040000
    decode_min_tokens: 2040000
  admission:
    reject_before_bootstrap: true
    token_count_must_fit_both_roles: true
  transfer:
    exact_token_count: true
    checksum_required: true
    abort_ack_preserved: true
  lifetime:
    deferred_release: true
    late_write_quarantine: true
  ownership:
    tenant_namespace: required
    page_generation_tag: required_when_supported
  canary_suite: kv-isolation-v4
  rollback_image: registry.example/sglang:0.5.21-patched-r19

Do not set unsupported properties to true for paperwork. Cloudflare notes that KV checksum and canary coverage do not yet span every model architecture. A release verifier should inspect logs and produce COVERED, PARTIAL, or UNSUPPORTED for each model/backend pair. Unsupported read-time integrity is a risk decision, not a hidden default.

Application prompt-prefix contracts still belong beside this manifest. Canonicalize stable policy and tool schemas; put volatile data after the reusable boundary; version every change; and record actual cached tokens. Namespace reuse by authorized principal, never by an unreviewed global hash. Shared gateway credentials can silently collapse user or tenant boundaries even when the inference engine is correct.

Acceptance must include two tenants and hostile timing

Happy-path generation cannot validate isolation. Build a test harness that uses two independently authorized tenants, unique high-entropy canary strings, over-limit and near-limit prompts, cancellations, disconnects, concurrent transfers, retries, and page reuse pressure. The oracle is simple: tenant B must never observe tenant A's canary in content, logs, timing class, or cache metadata beyond what the approved design intentionally shares.

async def cross_tenant_canary(client_a, client_b, limit):
    secret_a = random_canary("TENANT_A")
    await client_a.generate(prompt=pad(secret_a, tokens=limit - 8))

    for size in [limit - 1, limit, limit + 1, limit + page_size]:
        result = await client_b.generate(prompt=pad("TENANT_B", tokens=size))
        assert secret_a not in result.text
        assert result.request_id == result.kv_owner_request_id
        assert result.transferred_tokens == result.reserved_tokens or result.failed_closed
        if size > measured_capacity:
            assert result.status in {400, 413, 422}
            assert result.decode_started is False
Test classFault injectedRequired result
Capacity edgePrompt at capacity ± one pageAccepted request fits both roles or fails before handoff
Abort raceClient disconnect during RDMABoth halves cancel; pages stay quarantined until writes finish
Short transferSender writes fewer tokens than reservedChecksum/count mismatch; no decode
Page reuseImmediate high-concurrency reallocationGeneration tag/owner mismatch stops the read
Proxy timeoutTimeout before first tokenBounded abort; no orphan transfer or premature release
Tenant namespaceSame prefix under two principalsNo unauthorized reuse or cross-principal hit signal
Rolling upgradeMixed engine/router versionsCompatibility gate or safe drain; no ambiguous ownership

Run the suite after every change that can move capacity or lifecycle behavior: weights, quantization, context length, tensor/context parallelism, CUDA graph sizes, memory fraction, transfer backend, router, proxy timeout, and model upgrade. A weekly test on an unchanged canary environment misses the risky moment, which is the release that changed the physical contract.

Bind approval to a cache-isolation receipt

The release receipt should be machine-readable and human-reviewable. It names the exact artifact, environment, topology, contract hash, measured capacity evidence, patch set, test suite, failures, waivers, approvers, deployment ID, observation window, and rollback trigger. “SGLang patched” is not a receipt. A commit can be patched while the router, proxy, or configuration remains unsafe.

{
  "release": "glm53-pd-r27",
  "engine_image": "sha256:4e9...",
  "router_image": "sha256:07a...",
  "contract_hash": "sha256:be2...",
  "capacity_log": "evidence://run/kv-cap-1842",
  "patches": [36988, 38935, 39461, 39500, 41023, 41612],
  "tests": {"passed": 148, "failed": 0,
            "unsupported_controls": ["glm53-page-canary"]},
  "decision": "BOUNDED_RELEASE",
  "rollback_if": ["owner_mismatch", "short_transfer",
                  "cross_tenant_canary", "capacity_drift"],
  "approvers": ["inference-owner", "security-owner", "privacy-owner"],
  "expires_on_change": true
}

Keep unsupported controls visible. A named security owner may accept a bounded release with compensating measures - smaller context, dedicated pools, restricted tenants, lower concurrency, or a different topology - but the system must not render missing evidence as a pass. Expire approval automatically when any manifest field changes.

Keep optimization metrics downstream of isolation

Prompt reuse can lower prefill latency and cost. Disaggregated serving can improve accelerator utilization. Those benefits remain valuable only after isolation passes. Put security invariants ahead of throughput in the release gate: zero cross-tenant canaries, zero decode starts after rejection, zero short-transfer acceptance, and zero premature page reuse. Then compare time to first token, throughput, cached tokens, retries, and cost per accepted task.

eligible_for_performance_comparison = (
    cross_tenant_canary_failures == 0
    and decode_after_rejection == 0
    and short_transfer_accepts == 0
    and owner_mismatches == 0
)

cost_per_accepted_task = total_inference_cost / accepted_tasks
isolation_adjusted_capacity = throughput if eligible_for_performance_comparison else 0

This ordering prevents a configuration from winning because it raises cache density or throughput while weakening resource separation. Cloudflare's larger pool restored capacity headroom, but the incident report rejects relying on size alone because future model or runtime changes can move the threshold. Optimization is a conditional score, not permission to serve.

Failure modes that a version bump cannot close

FailureMisleading evidenceRelease control
Pool smaller than contextLong prompts returned HTTP 200 in basic testsParse startup capacity on every role and test the boundary
Rejected request reaches decodePrefill internally marked it finishedReject before bootstrap/allocation/transfer; assert no decode start
Late RDMA write corrupts new ownerClient already disconnectedPair aborts and defer release until transfer completion
Checksum flag unsupportedConfiguration accepts the flagVerify coverage in logs for exact model/backend
Global prefix namespaceHigh cache-hit rateSalt or scope reuse to authorized principal
Timing side channelNo content appears in another responseCross-principal latency probes and scoped cache keys
Mixed-version rolloutEvery image is individually healthyCompatibility matrix, drain, and rollback rehearsal
Observability leakLogs help debug ownershipRedact prompt/KV content; restrict and expire evidence

Do not use customer prompts as production canaries. Generate unique synthetic markers and keep the mapping in a restricted test service. If real cross-request content appears, stop affected serving, preserve minimum necessary evidence, follow the incident and privacy process, and avoid broad log exports that create a second exposure.

Polymarket results surfaced by the focused scan were unrelated markets about Cloudflare earnings, future incidents, and a possible GLM release. They are excluded. Community attention and GitHub stars show interest, not safety. The release decision rests on the exact patch set and local tests.

A seven-stage path from patch to defensible release

  1. Inventory: enumerate prompt caches, distributed prefix stores, live KV pools, principals, keys, transfer backends, and every component that can accept, move, read, or release state.
  2. Pin: freeze engine, router, model, quantization, topology, proxy, container, driver, and configuration identities. Record the security fixes included.
  3. Measure: extract actual capacity from every prefill and decode role. Compare it with the configured context limit after graph capture and allocation.
  4. Fail closed: prove over-capacity, rejected, cancelled, and disconnected requests stop before unsafe handoff or decode.
  5. Challenge ownership: run two-tenant canaries, short transfers, abort races, immediate page reuse, proxy timeouts, and mixed-version tests.
  6. Release narrowly: deploy a bounded model/tenant cohort with named monitors, automatic quarantine, a timed rollback, and preserved receipts.
  7. Observe and expire: watch capacity drift, aborts, checksum coverage, owner mismatches, latency anomalies, and canary alerts; invalidate approval on any serving-shape change.

The Cloudflare report shows why this process should include upstream issue review. Another SGLang issue had described a related failure path before the incident. A mature program maps upstream issues to local topology and decides whether they are reachable, rather than waiting for a vendor severity label or a customer-visible symptom.

Frequently asked questions

Was this a normal prompt-cache miss?

No. A rejected long request was allowed into a disaggregated handoff, and decode read stale runtime KV pages. Product-level prompt-cache diagnostics would not prove those pages belonged to the current request.

Does a larger KV pool solve the problem?

It removes the documented capacity mismatch when measured correctly. It does not prove safe rejection, abort pairing, transfer integrity, page lifetime, tenant namespace, or read ownership. Keep it as one independent control.

Should every SGLang operator upgrade to v0.5.22?

Cloudflare recommends v0.5.22 or later or backporting the relevant fixes for disaggregated deployments. Evaluate current release notes and your exact topology; do not infer that an untested upgrade is safer than a verified pinned build.

Can cache salting replace page ownership?

No. Salting can prevent unauthorized prefix reuse across principals. It cannot stop decode from reading stale pages after an unsafe lifecycle transition inside the runtime.

What is the minimum release evidence?

Exact artifact identities, measured capacities, rejection-before-handoff proof, abort and deferred-release tests, transfer count/checksum evidence, two-tenant canaries, unsupported-control disclosure, named approval, monitoring, and rehearsed rollback.

Sources and further reading

Current facts were checked on October 27, 2026. Incident scope statements are attributed to the organizations that investigated them; research papers and community metrics do not prove a particular deployment is affected.

Related guides