Private MCP is becoming a deployment choice, not a workaround
Two current releases make the architecture concrete. Google announced MCP support in Apigee hybrid 1.17 on October 14, placing the MCP data plane inside a customer's Kubernetes cluster. OpenAI documents Secure MCP Tunnel as an outbound-only path from a customer-run client to an OpenAI-hosted tunnel endpoint. Both reduce the need to publish a private tool server directly. Neither turns connectivity into permission.
Apigee hybrid converts operations from an OpenAPI 3.0.x document into MCP tools. The control plane generates configuration, the cluster fetches that configuration, and agent requests reach an MCP data plane that runs beside the private APIs. Google states that MCP request traffic stays inside the cluster. Its production guidance still requires ingress policy so only Apigee Message Processor pods can reach the MCP service on TCP 443.
OpenAI's tunnel pattern moves the initiation point in the other direction. A customer-run tunnel-client authenticates to the OpenAI control plane, long-polls for queued work, forwards JSON-RPC to a private HTTP or stdio MCP server, and posts the response back. The private server needs no public listener, but the client must reach OpenAI over outbound HTTPS and must remain able to reach the private server. If the client disconnects, tunnel calls fail until it reconnects.
The official openai/tunnel-client repository showed 567 stars and 24 open issues when checked on October 15. Exact community discussion of these two releases was thin. That is a reason to keep the claims narrow: the public artifacts prove architecture and documented behavior, not broad adoption or field reliability.
A private route answers “how can the request get there?” Production review must still answer “who may ask, what may happen, how do we prove it, and what happens twice?”
Two patterns move different boundaries
Pattern A: outbound tunnel
hosted AI product -> hosted tunnel endpoint
^
| outbound HTTPS poll / response
customer network: tunnel-client -> private MCP server -> internal API
Pattern B: in-cluster gateway
agent client -> Apigee Message Processor -> in-cluster MCP data plane
-> private API
control plane ---------------- config bundle --------^The tunnel pattern is useful when a hosted AI product must call a server on a laptop, VM, private service, or on-premises network. It avoids a new inbound firewall rule and provides health, readiness, metrics, and a loopback admin UI in the client. It also introduces a durable customer-side daemon, a hosted queue and endpoint, runtime credentials, and a polling path that operators must own.
The in-cluster gateway pattern is useful when APIs already sit behind Apigee hybrid and must stay within a regulated or multi-cloud runtime boundary. It reuses OpenAPI, gateway policies, analytics, and Kubernetes controls. It also depends on control-plane configuration delivery, watcher identity, Message Processor routing, cluster policy, and the correctness of the generated tool surface.
| Question | Outbound tunnel | In-cluster gateway |
|---|
| Inbound public listener | Not required for the private MCP server | Not required outside the cluster path |
| Request initiation | Customer client polls hosted control plane | Agent traffic enters existing gateway path |
| Primary local component | Long-running tunnel client | MCP data-plane service and Message Processor |
| Config dependency | Tunnel, profile, channel, proxy, server binding | OpenAPI, Discovery Proxy, fetched config bundle |
| Natural failure | Disconnected client or stale queue work | Config, watcher, route, or network-policy failure |
| Best fit | Hosted product reaching private or local tools | Governed APIs already running in Apigee hybrid |
Do not choose from the table alone. An organization may use both: an in-cluster gateway exposes governed APIs as tools, while an outbound tunnel gives a hosted agent a private route to that gateway. Composition creates another end-to-end system. It must preserve identity, deadlines, cancellation, error classes, and evidence across every hop.
Review seven boundaries separately
- Placement. Identify where tool definitions, request bodies, responses, credentials, logs, queues, and configuration exist. “Data plane is local” does not mean every control-plane artifact is local.
- Transport identity. Prove which tunnel client, pod, workload, organization, and workspace may use the path. Rotate and scope runtime credentials independently from user credentials.
- Acting principal. Preserve the human or service principal on whose behalf the agent acts. A shared gateway account cannot explain which user requested a payroll export.
- Tool authorization. Limit discovery and calls by principal, task, environment, tool, resource, method, and consequence. A visible tool is not automatically an allowed tool.
- Downstream authority. Decide whether the MCP server uses delegated user tokens, a workload credential, a per-tool service identity, or a brokered token exchange. The downstream API is the enforcement point that matters.
- Evidence. Correlate product invocation, tunnel or gateway transport, MCP request, policy decision, downstream effect, response, and approval. Transport logs alone do not prove business outcome.
- Recovery. Define duplicates, retries, cancellation, stale work, partial effects, reconnects, rollbacks, and operator escalation. A private path can still repeat a destructive action.
The MCP 2026-07-28 authorization specification requires OAuth 2.1 protections, protected-resource metadata, authorization-server discovery, token audience validation, and scope minimization. Those rules secure an MCP authorization flow. They do not decide whether a particular user may invoke refund_invoice for a particular account, or whether the server may reuse a system credential to bypass that user's business permissions.
Enterprise Managed Authorization adds a useful identity assertion pattern when the client and server share an enterprise identity provider. It still needs a policy that maps identity to resources and actions. Treat protocol conformance as a prerequisite, not a business-authorization verdict.
Write the connection as a versioned contract
A connectivity contract gives security, platform, API, and application owners one review object. Pin the protocol version, component versions, tool catalog, identities, destinations, timeouts, evidence fields, retry rules, and rollback. Keep secrets out of the file; reference secret-manager identities.
apiVersion: private-mcp.connectivity/v1
connection: finance-readonly-prod
transport:
pattern: outbound-tunnel
client_release: pinned-sha256
control_plane: api.openai.com:443
mcp_target: unix:///run/finance-mcp.sock
identity:
workload: spiffe://corp/agent-connectors/finance
acting_principal: required
downstream_token: delegated-and-audience-bound
authorization:
discover: [ledger.lookup, invoice.read]
deny: [invoice.pay, vendor.write, journal.post]
constraints:
environment: production
max_rows: 500
purpose: approved_finance_analysis
reliability:
deadline_seconds: 30
max_attempts: 2
idempotency_key: required_for_effects
disconnected_behavior: fail_closed
evidence:
correlation_id: required
policy_decision: required
downstream_receipt: required
payload_logging: redacted
rollback:
owner: platform-oncall
action: disable-tunnel-and-revoke-runtime-key
For Apigee hybrid, the same contract would name the 1.17+ runtime, Discovery Proxy, OpenAPI document hash, watcher identity, MCP service, allowed Message Processor selectors, Kubernetes policy object, backend credential model, and configuration-reconciliation evidence. A generated tool catalog should not change merely because an API specification added an administrative operation.
Diff the contract in review. Additions to tools, scopes, destinations, identities, or retry counts are security changes even when the network route stays the same. Re-run acceptance tests after a protocol, client, gateway, OpenAPI, authorization-server, backend, model, or policy update.
Keep workload, user, and downstream identity distinct
The tunnel or gateway needs a workload identity to join the system. The agent request needs an acting principal to express whose task is running. The downstream API needs a credential and audience appropriate to the requested action. These may be related, but they are not interchangeable.
Use delegated OAuth when the downstream action must remain inside the user's authority. Use a service identity only for genuinely service-owned work, and constrain it with resource and action policy. Never accept “the agent was approved” as a substitute for deciding which principal owns the side effect.
OpenAI documents that OAuth discovery may travel through the tunnel while the authorization server itself is not automatically tunneled. That creates a precise failure to test: the MCP server is reachable, but browser-facing authorization cannot complete. The correct result is an explicit authentication failure, not a fallback to an overprivileged server token.
def authorize(call, transport, principal, policy):
assert transport.client_id in policy.allowed_clients
assert principal is not None
assert call.tool in policy.tools_for(principal)
assert call.resource in policy.resources_for(principal)
assert call.purpose in policy.approved_purposes
token = broker.issue(
subject=principal,
audience=call.downstream_audience,
scopes=policy.min_scopes(call)
)
return token
Authorization-dependent discovery is preferable when the tool catalog itself reveals sensitive systems or operations. If the platform cannot filter discovery, return stable denial responses at call time and keep prohibited tools out of model context where possible. A denial needs a machine-readable reason, not a fabricated empty result.
Operate the path as a service, not a cable
OpenAI's client exposes /healthz, /readyz, /metrics, /health/mcp, and a loopback admin UI. Use them to prove process health, control-plane connectivity, MCP discovery, queue behavior, and channel availability. Do not expose the admin UI remotely without an explicit operator network and authentication decision.
Apigee hybrid needs its own evidence: configuration bundle age, watcher success, MCP service endpoints, Message Processor reachability, policy reconciliation, backend latency, tool-list version, and denied connections from non-authorized pods. A green Kubernetes pod is weaker than a synthetic call that proves the intended principal can list one allowed tool and cannot list or call a denied tool.
# Illustrative acceptance probes
curl --fail --silent http://127.0.0.1:PORT/readyz
curl --fail --silent http://127.0.0.1:PORT/health/mcp?details=true
# Then run signed end-to-end fixtures:
# 1. allowed principal + allowed read tool -> exact result + receipt
# 2. allowed principal + denied write tool -> structured denial
# 3. wrong audience token -> 401, no downstream request
# 4. duplicate effect request -> one effect, same receipt
# 5. stopped client / blocked pod route -> explicit unavailable state
Correlate evidence with one run ID across the hosted product, transport, MCP server, authorization service, and downstream API. OpenAI notes that tunnel transport requests are not themselves emitted as ChatGPT Compliance Platform app events, while normal app-level invocation and auth lifecycle logging still applies. That boundary belongs in the evidence map. Do not assume one compliance export contains every hop.
Measure availability by consequence. A read-only documentation tool can retry more freely than a payment or change-management tool. For effects, require idempotency at the downstream system, persist an outcome receipt, and distinguish “request accepted,” “action performed,” and “result confirmed.” Cancellation after the downstream effect is not a rollback.
Failure modes private placement does not remove
| Failure mode | What it looks like | Required response |
|---|
| Tunnel or gateway unavailable | Tool list works from cache but calls hang or fail | Expire catalog, return unavailable, alert owner |
| Identity collapse | Every call reaches the backend as one admin service account | Require acting principal and least-privilege token broker |
| Stale generated tools | OpenAPI changed but discovery or policy did not | Pin hashes, reconcile, block unmatched versions |
| OAuth path split | Private MCP is reachable; authorization server is not | Fail closed and route to operator, never substitute credentials |
| Duplicate effect | Reconnect or retry repeats a write | Downstream idempotency key and effect receipt |
| Partial evidence | Transport success exists without downstream outcome | Correlate policy, call, effect, response, and confirmation |
| Fail-open fallback | Private route failure silently switches to public or local file path | Explicit fallback policy; default deny for sensitive tools |
| Overbroad discovery | Model sees tools the principal cannot use | Filter catalog or return stable authorization denial |
| Control-plane drift | Local runtime uses old policy or routing material | Maximum config age, reconciliation alert, safe stop |
One current r/mcp practitioner comment captured a small but important reliability rule: a server that cannot reach its intended application should fail instead of silently writing to a local fallback. Apply the same principle to enterprise connectivity. A fallback that changes destination, credential, evidence, or effect semantics is a new route and requires its own approval.
Use a four-stage rollout
- Map. Draw every hop, identity, credential, tool catalog, log, queue, and control plane. Mark which data leaves the runtime boundary and which metadata the hosted platform stores.
- Constrain. Start with read-only tools, a non-production backend, a single workspace, explicit scopes, low row limits, and no automatic fallback.
- Challenge. Test denied principals, token audience, stale config, network partitions, stopped clients, duplicate delivery, slow downstreams, cancellation, oversized responses, and audit export completeness.
- Release. Require platform, API, security, data, and application owners to approve the exact contract. Publish the operator runbook, escalation route, revocation action, recovery target, and review date.
Production gate
- Exact component, protocol, config, OpenAPI, and policy versions are recorded.
- Workload identity, acting principal, and downstream credential are distinguishable in evidence.
- Allowed and denied discovery and call fixtures pass.
- Network policy limits the local MCP service to intended callers.
- Retries cannot duplicate consequential actions.
- Health signals cover route readiness, not only process liveness.
- Logs cover policy decision and downstream outcome without leaking secrets.
- Operators can revoke the route, rotate credentials, reconcile config, and recover service.
The decision is not “tunnel versus gateway.” It is whether the composed route produces a narrower, observable, and recoverable authority path than the integration it replaces.
FAQ
Does a private MCP tunnel make a tool safe?
No. It removes a public inbound path. It does not establish user authority, downstream least privilege, safe retries, complete evidence, or recovery.
Is Apigee hybrid safer than an outbound tunnel?
Not categorically. It keeps the runtime data plane in the customer cluster and can reuse gateway and Kubernetes controls. The tunnel keeps the private server off the public internet and fits hosted products. Evaluate the exact topology and consequence.
Should every MCP call use the user's token?
No. Some work is service-owned. The rule is to name the acting principal and use the narrowest credential appropriate to the action. Shared administrative credentials need an exceptional, documented reason.
What is the minimum useful test?
Run one allowed read, one denied write, one wrong-audience token, one duplicate effect, and one disconnected-route test. Verify policy, downstream effect, evidence, and operator recovery for each.
Sources and further reading
Current facts and public artifacts were checked on October 15, 2026. Product behavior and availability can change; validate the current documentation and release before deployment.
