AI gateway plugins need a compatibility and failure-isolation contract

The hard part is not loading a plugin. It is preserving meaning.

Respan launched an open-source Rust AI gateway on October 20, 2026. The Hacker News launch drew 45 points and 19 comments, while the repository had 51 stars during the October 21 focused scan. That is a useful current engineering signal, not evidence of broad adoption. The project calls the gateway pre-alpha and invites routing, tool, filtering, observability, and security features as plugins.

The architectural idea is credible. Provider APIs differ in authentication, request bodies, tool-call encodings, streaming event shapes, usage reports, finish reasons, and errors. A gateway can translate those edges into a smaller internal protocol. Plugins can then inspect or transform one representation instead of reimplementing logic for OpenAI, Anthropic, Gemini, and every OpenAI-compatible endpoint.

But normalization changes the risk. A plugin may redact a secret, inject a policy message, select a model, cache a response, record a trace, limit tokens, or rewrite a fault. Those actions are not independent. A cache placed before identity scoping can leak one tenant's answer to another. A redactor placed after tracing can preserve the secret in telemetry. A retry plugin can repeat a non-idempotent tool request. A filter that does not recognize a new streaming event can silently drop it.

Rust helps with memory safety and explicit types. It does not prove semantic compatibility, correct ordering, bounded latency, safe retries, complete redaction, or operational readiness. An open-source license lets a team inspect the implementation; it does not replace representative tests. A small core reduces the amount of privileged code, but it also increases the importance of the extension contract around that core.

Separate the client edge, canonical event stream, and provider attempts

Published Respan crate documentation describes a gateway that runs each request through plugins and streams a response back. Its conversation crates model a response as lifecycle events: start, block start, deltas, block end, usage, finish, and provider-native events. The HTTP layer binds client and provider I/O in opposite directions. Typed faults carry a client message, a fault class, an internal source, and extensions that protocol edges may translate.

client protocol edge
  authenticate + decode + request ID
             |
             v
canonical request and event lifecycle
  policy -> redaction -> routing -> cache
             |
        attempt budget
        /           \
 provider A         provider B
        \           /
       typed fault or event stream
             |
 trace + encode + commit to client

The canonical stream is the strongest part of the design because streaming and non-streaming providers can share one plugin surface. It is also the most dangerous part to underspecify. A plugin needs to know whether an event may be delayed, duplicated, removed, replaced, or reordered. The gateway needs to define whether a usage event can arrive before finish, whether provider-native events must be preserved, how cancellation propagates, and what happens when a plugin rejects an event after bytes have reached the client.

Define three commitment points. Request accepted means authentication, schema validation, and budgets passed, but no provider attempt is guaranteed. Provider committed means an upstream accepted work or produced side effects. Client committed means response headers or stream bytes escaped the gateway. Before client commitment, a gateway may be able to switch providers and return a clean fault. After client commitment, it may only terminate or annotate the stream; pretending a second provider's answer is a continuation corrupts the protocol.

Good plugin boundaries make forbidden behavior obvious. A metrics plugin should not mutate content. A model router should not read raw tool results unless policy requires it. A cache should not own retries. A redactor may replace sensitive text but must publish what class was removed and whether downstream evaluation remains meaningful. Capabilities should be declared and enforced, not inferred from code review alone.

Ship a manifest for the complete plugin set

Production teams should pin the gateway core and every plugin as one artifact. Cargo's SemVer guidance warns that pre-1.0 packages treat the left-most non-zero component as the compatibility boundary and that many runtime behavior changes remain judgment calls even when code compiles. “Compatible dependency update” is therefore not sufficient release evidence for a pre-alpha request path.

gateway_release: gw-2026-10-21.3
core:
  package: respan-gateway-core
  version: 0.1.0
  source_digest: sha256:7c1e...45aa
canonical_protocol: conversation-events/v1
plugins:
  - id: tenant-context
    version: 2.4.1
    hooks: [before_request]
    capabilities: [read_headers, add_context]
    budget_ms: 4
  - id: pii-redact
    version: 1.8.0
    hooks: [before_trace, before_provider]
    capabilities: [read_content, replace_content]
    fail_mode: closed
  - id: model-router
    version: 3.2.0
    hooks: [route]
    capabilities: [read_metadata, select_provider]
    max_attempts: 2
compatibility:
  fixtures: fixtures/gateway-v7.jsonl
  expected_trace_schema: otel-genai-profile-2026-10
rollback: gw-2026-10-07.5

The manifest makes order reviewable. Identity and data classification usually precede caching, routing, and traces. Redaction must happen before any sink that cannot receive raw content. Policy evaluation must name whether it sees the original or redacted request. Response validation must occur before cache write. Metering must distinguish provider attempts from one client request. There is no universally correct order, but hidden order is indefensible.

Record capability scope. A plugin that only counts tokens does not need prompt text. A router may need model, latency target, context length, and budget, but not employee names inside the prompt. A security classifier may need content and tool definitions but should not receive provider credentials. When the framework cannot technically restrict access, the manifest still creates a testable review claim and exposes the gap.

Version configuration and external dependencies too. A plugin can change behavior because a policy file, classifier model, remote list, certificate, or feature flag changed while its package version stayed constant. The release digest should cover resolved configuration, schemas, model identifiers, dependency lock, and enabled features. If the runtime fetches policy dynamically, log the policy revision on every request.

One fault taxonomy must own retries, fallback, and client truth

Provider errors arrive as HTTP status codes, JSON bodies, stream resets, timeouts, malformed events, quota messages, safety stops, or connection failures. A gateway should translate them into a closed internal taxonomy without erasing the provider evidence. Suggested classes include authentication, authorization, quota, rate limit, timeout, transport, protocol, provider overload, content policy, invalid request, unsupported capability, plugin rejection, cancellation, and internal defect.

{
  "request_id": "gw_01JAZ...",
  "fault_class": "provider_overload",
  "client_status": 503,
  "safe_message": "Model temporarily unavailable",
  "provider": "provider-a",
  "provider_status": 529,
  "attempt": 1,
  "retryable": true,
  "client_committed": false,
  "source_digest": "sha256:...",
  "trace_id": "4bf92f..."
}

The client message is intentionally separate from the internal source. Returning a raw provider error may expose credentials, tenant IDs, prompt fragments, or infrastructure detail. Logging only the safe message destroys the evidence SREs need. Keep the source in a protected diagnostic record with retention and access controls.

Retry ownership must be singular. Provider SDKs, the gateway core, routing plugins, service meshes, and application clients can each retry. If every layer tries twice, one user request can become sixteen upstream attempts. Envoy's circuit-breaking guidance recommends retry budgets because unbounded retries amplify overload. The gateway should assign one shared deadline, one maximum-attempt count, and an attempt ledger visible to downstream observability.

for candidate in route.fallbacks:
    if budget.deadline_expired() or budget.attempts_exhausted():
        return final_fault("attempt_budget_exhausted")
    if stream.client_committed():
        return terminate_stream(last_fault)
    result = attempt(candidate, idempotency_key=request.id)
    budget.record(result)
    if result.ok: return result
    if not result.fault.retryable: return result.fault
return final_fault("no_eligible_provider")

Idempotency is not an HTTP detail. A chat completion that only generates text may be repeatable enough for fallback. An agent request may trigger a remote tool, send email, open a ticket, or mutate a repository before the model response fails. The gateway cannot safely retry unless the application supplies an idempotency key and the tool plane provides receipts or deduplication. Routing the second attempt to another model does not undo the first model's action.

Trace decisions without turning telemetry into a second data breach

Respan's broader repository and docs emphasize OpenTelemetry-style tracing. OpenTelemetry's current GenAI conventions name operations, models, usage, tool calls, inputs, and outputs, while warning that system instructions, tool arguments, and results may contain sensitive information. A gateway trace is valuable precisely because it crosses providers and plugins. That makes it a concentrated copy of prompts, identities, decisions, tools, costs, and failures.

Split operational fields from content. Export request ID, tenant pseudonym, gateway release, plugin set digest, operation, provider, model, route reason, attempt number, latency, token counts, fault class, cache decision, policy result, and stream completion by default. Make prompt, completion, tool arguments, and tool results opt-in, filtered, encrypted, access-controlled, and short-lived. Do not assume that “observability” is an unlimited processing purpose.

Instrument every hook with start, finish, status, latency, mutation digest, and fault. The mutation digest can prove that content changed without storing the content. For regulated or highly sensitive workloads, retain a local evidence receipt while exporting only aggregates. If a plugin changes a route, blocks content, writes cache, or redacts fields, record the decision and policy version.

Trace schemas are contracts too. OpenTelemetry semantic conventions evolve, and the GenAI fields remain under active development. Pin the profile expected by dashboards and alerts. Test that an upgrade neither drops fields nor changes cardinality enough to break cost, SLO, or security monitoring.

Upgrade with replay, fault injection, shadowing, and a real rollback

Begin with a representative corpus: short and long prompts, structured output, tool definitions, multimodal input where supported, multiple tenants, streamed and non-streamed responses, cancellation, policy rejection, cache hit and miss, rate limit, provider timeout, malformed event, and partial stream. Store normalized inputs, permitted golden properties, and expected faults. Do not require exact prose equality when the model is stochastic; require schema, policy, tool, routing, metering, and lifecycle properties.

Replay the corpus against the current and candidate plugin sets. Compare canonical event sequences, mutation digests, route decisions, fault classes, attempt counts, latency, token accounting, trace shape, and redaction. A new plugin may improve content filtering while adding 300 ms to every stream. It may reduce cost while changing tool-call arguments. The review must show the tradeoff, not hide it behind a single success rate.

Inject failures between every stage. Crash or time out a plugin. Return an unknown provider event. Reset a stream after headers. Make telemetry unavailable. Exhaust the cache. Rotate credentials. Force provider overload. Cancel the client during fallback. Verify that the gateway fails in the declared direction, does not leak raw content, and does not exceed the attempt budget.

Shadow traffic only after redaction and legal review. The candidate path must not call side-effectful tools or double-charge an external action. Compare decision receipts asynchronously. Then canary a small, bounded traffic slice by tenant and workload, with automatic rollback on error, latency, trace completeness, policy divergence, or cost thresholds.

Rollback must restore core, plugins, configuration, policy models, schemas, and cache compatibility. Keeping yesterday's binary is not enough if the candidate wrote cache entries or state the old release cannot read. Test backward and forward compatibility or isolate state by release. Preserve both release receipts so an incident reviewer can identify exactly what ran.

Failure modes that a happy-path gateway demo will miss

OWASP's LLM supply-chain guidance is relevant because the gateway sits below many applications. A compromised or overprivileged plugin can observe every prompt and credential, rewrite policy, or redirect requests. Verify source and license, pin dependency locks, produce an SBOM, scan releases, review maintainers and publishing keys, and make optional plugins opt-in. Do not load extensions dynamically from an untrusted registry in production.

The most deceptive failure is plausibility. A gateway can return a fluent answer while usage is wrong, policy was skipped, fallback selected an unauthorized model, or a tool call vanished. Acceptance tests must inspect the decision path, not just the final text.

A practical adoption checklist

1. Inventory client protocols, providers, models, tools, tenants, data classes, and current retry layers.
2. Pin the gateway core, plugin set, dependency lock, configuration, policies, schemas, and model identifiers.
3. Document hook order, mutation rights, capabilities, deadlines, fault behavior, and fail-open or fail-closed mode.
4. Define a canonical request and event lifecycle, including cancellation and client commitment.
5. Create one typed fault taxonomy and one attempt ledger across every provider and retrying layer.
6. Build a replay corpus with stream, tool, tenant, cache, policy, and failure fixtures.
7. Test redaction before every telemetry, cache, and diagnostic sink.
8. Run plugin timeout, crash, unknown-event, partial-stream, provider-overload, and telemetry-outage injections.
9. Shadow only safe requests, then canary by bounded workload with automatic rollback thresholds.
10. Retain a release receipt containing manifests, test results, approvals, rollout metrics, and rollback proof.

Do not promote a plugin because the package compiled or one provider answered correctly. Promote the complete gateway release when it preserves declared semantics under normal traffic and failures. That standard is slower than copying a quickstart, but much faster than debugging a cross-provider incident with no attempt ledger, no trace contract, and no reproducible plugin set.

FAQ

Does a Rust AI gateway make plugins safe?

No. Rust removes important classes of memory bugs. The system can still misroute requests, leak data, retry side effects, corrupt streams, or change behavior across upgrades.

What belongs in the plugin contract?

Versions, hooks, ordering, schemas, mutation rights, capabilities, time and resource budgets, fault classes, retry ownership, telemetry, sensitive-data policy, compatibility fixtures, rollout criteria, and rollback.

Should plugins be allowed to fail open?

Only when the consequence is explicitly acceptable. Metrics may fail open with a gap alert; authentication, tenant isolation, secret redaction, and mandatory policy checks normally fail closed.

Can a gateway retry every provider error?

No. It needs an idempotency decision, remaining deadline and attempt budget, an eligible fallback, and proof that output or side effects have not already committed.

How much community evidence supports Respan?

The October 21 focused scan found one exact Hacker News launch discussion and the first-party repository. That supports a current case study, not a popularity, reliability, or production-adoption claim.

Sources and further reading

• Launch HN: Respan open-source Rust AI gateway - launch claims, pre-alpha status, and plugin direction; accessed Oct 21, 2026.
• respanai/respan - Apache-2.0 source, packages, plugin structure, and issues; accessed Oct 21, 2026.
• Respan documentation overview - first-party gateway, tracing, evaluation, and security positioning.
• Respan October 9 changelog - load-balancing and fallback behavior changes.
• respan-gateway-core crate documentation - gateway, plugin, call, and fault types.
• respan-gateway-conversation crate documentation - canonical streaming event lifecycle.
• respan-gateway-http crate documentation - client/provider edge bindings.
• OpenTelemetry GenAI semantic attributes - operation, model, usage, tool, and sensitive-content guidance.
• Envoy circuit breaking - request and retry budgets for upstream protection.
• Envoy life of a request - stream lifecycle and early termination reference.
• Cargo SemVer compatibility guide - source and runtime compatibility limits, especially before 1.0.
• OWASP Top 10 for LLM Applications 2025 - supply-chain risk reference.
• Related: benchmark the full agent inference loop.
• Related: build execution receipts around agent actions.