Per-Gateway Isolation¶

By default every Gateway of the class shares one chart-deployed proxy pool and one Cloudflare Tunnel. This guide covers the opt-in hard-isolation mode: a DEDICATED data plane — its own proxy Deployment and its own tunnel — rendered and reconciled by the controller for a single Gateway.

Use it when admission-level scoping (see Multi-Tenancy) is not enough: tenants that must not share a process, a tunnel identity, or a failure domain.

What you get¶

• A dedicated proxy Deployment in the Gateway's namespace, running the same proxy image as the shared plane (chart-wired via --proxy-image, overridable per Gateway).
• A dedicated Cloudflare Tunnel: the Gateway's routes are written to ITS tunnel's ingress document and pushed to ITS proxy pods only. Routes of other Gateways never reach this data plane — and this Gateway's routes never reach theirs.
• Independent lifecycle: rendered resources are controller-owned (deleted with the Gateway, healed on drift), connector draining on shutdown, optional autoscaling on the proxy's in-flight gauge.
• A per-Gateway config-push credential: the controller authenticates config pushes to this plane with the Gateway's own token, never the shared plane's.

Opting in¶

1. Create a Cloudflare Tunnel for the Gateway (one tunnel per isolated Gateway) and store its connector token in the Gateway's namespace:

   kubectl --context <ctx> --namespace tenant-a create secret generic edge-tunnel-token \
     --from-literal=tunnel-token=<connector-token>
   

2. Create a GatewayConfig next to the Gateway:

   apiVersion: cf.k8s.lex.la/v1alpha1
   kind: GatewayConfig
   metadata:
     name: edge-config
     namespace: tenant-a
   spec:
     tunnelTokenSecretRef:
       name: edge-tunnel-token
     replicas: 2
   

3. Reference it from the Gateway:

   apiVersion: gateway.networking.k8s.io/v1
   kind: Gateway
   metadata:
     name: edge
     namespace: tenant-a
   spec:
     gatewayClassName: cloudflare-tunnel
     infrastructure:
       parametersRef:
         group: cf.k8s.lex.la
         kind: GatewayConfig
         name: edge-config
     listeners:
       - name: https
         port: 443
         protocol: HTTPS
   

The controller renders cf-proxy-edge (Deployment) and cf-proxy-edge-config (headless Service) in tenant-a, parses the tunnel identity from the connector token (there is deliberately no separate tunnelID field — it cannot drift from the token), and starts syncing the Gateway's routes to that tunnel. The Gateway's status address becomes <tunnel-id>.cfargotunnel.com, and Programmed turns True only once the rendered Deployment has ready replicas — that is, registered tunnel connectors.

A Gateway without infrastructure.parametersRef keeps the shared data plane, unchanged. Removing the ref later deletes the rendered resources (only when actually owned by the Gateway) and returns the Gateway to the shared plane.

GatewayConfig reference¶

All Secret references are namespace-local by construction, so a Gateway cannot point at another tenant's credentials. That is narrower than it sounds: a tenant does not need another tenant's Secret to reach their tunnel, because the tunnel identity is parsed from the connector token and a token is just base64 JSON. Writing someone else's tunnel UUID into your own token is the same claim without touching their Secret, which is why the controller arbitrates tunnel ownership separately (see below).

Gateway.spec.infrastructure.labels and .annotations propagate to the rendered resources and the pod template; changing them rolls the pods.

Autoscaling¶

spec:
  autoscaling:
    minReplicas: 2
    maxReplicas: 10
    targetInflightPerPod: 50

The rendered autoscaling/v2 HPA scales the proxy Deployment on cftunnel_proxy_requests_in_flight as a Pods-type custom metric — concurrency is the saturation signal for an I/O-bound L7 hop, not CPU. Serving Pods metrics to the HPA requires a metrics adapter (prometheus-adapter or KEDA) exposing the gauge through the custom-metrics API; without one the HPA reports FailedGetPodsMetric and holds minReplicas — visible degradation, never silent. See Metrics & Alerting for adapter examples.

Do not pair a VerticalPodAutoscaler in apply mode with these Deployments: applying VPA recommendations restarts pods, which drops tunnel connectors. Recommendation mode is fine.

Sharing a tunnel¶

A Cloudflare Tunnel belongs to the Gateway already serving it, and the GatewayClass tunnel belongs to the operator. A Gateway whose connector token names a tunnel that another namespace's Gateway — or the GatewayClass itself — already serves is refused: it reports Accepted=False with reason InvalidParameters, emits a TunnelClaimRejected Warning Event, none of its routes are programmed anywhere, and its data plane is not rendered at all. Its routes say so too, so the tenant can see the cause without asking the operator.

The plane is removed rather than merely left unconfigured. That matters when both parties hold the SAME token — an operator handing one token to two Gateways — because both connectors then register and Cloudflare load-balances across them, so a configless survivor would answer a share of the incumbent's requests with 404s.

A forged claim is a different story, and the difference is worth knowing: connector registration authenticates on the account tag and the tunnel secret inside the token, not on the tunnel UUID. A tenant who writes a neighbour's UUID into a token they minted themselves cannot connect to that tunnel at all, and never serves a byte of its traffic. What the claim did buy them, before this rule, was the controller pushing that tunnel's merged configuration — hostnames, backends, and any backend-mTLS client keys — straight to their own proxy pods over the config API, which needs no working connector. That disclosure is what the refusal closes.

Possession, not age, decides who holds a contested tunnel: a Gateway already advertising that tunnel in its status addresses keeps it against any newcomer, however old the newcomer is. Otherwise a tenant whose Gateway happens to predate the victim's could retarget its own token and evict the legitimate holder. Age (then UID) decides only between claims with equal standing, such as two first-time claims.

The refusal exists because a connector token proves nothing. It is base64-encoded JSON, and any tenant who can create a GatewayConfig can write any tunnel UUID into it. Tunnel UUIDs are not secret either: the controller publishes them in Gateway status as <id>.cfargotunnel.com so external-dns can consume them. Without arbitration, naming a neighbour's tunnel is enough to join their partition — and because Cloudflare load-balances a tunnel's requests across every connector, the controller then merges both parties' routes into one ingress document and pushes the union to both parties' proxies. The claimant receives the incumbent's hostnames, backend services, filter bodies, and any backend-mTLS client certificates their routes carry.

Two Gateways in the SAME namespace may share a tunnel: that is a tenant sharing with itself, and no boundary is crossed.

If every party on a tunnel is trusted to see the others' routes — a single-tenant cluster, or a migration from the shared plane to dedicated ones — an operator can set allowSharedTunnels: true on the cluster-scoped GatewayClassConfig (chart value gatewayClassConfig.allowSharedTunnels). That restores the merge behaviour for the whole class. The field is deliberately not on GatewayConfig: a tenant must not be able to grant it to themselves.

When two DEDICATED Gateways end up on one tunnel, every affected route carries a cf.k8s.lex.la/TunnelShared=True condition and a mirrored TunnelShared Warning Event naming the other Gateways, so the collapsed isolation stays visible rather than silent. That covers both ways it can happen — the operator enabling allowSharedTunnels, and two Gateways in one namespace, which needs no opt-in. The condition's reason reads TunnelSharedAcrossNamespaces in the same-namespace case too, which is inaccurate; tracked in issue #680.

A dedicated Gateway sharing the CLASS tunnel is not flagged: the collision detector counts dedicated partitions only, and the shared plane is not one. Watch the controller log for that case.

What this does not settle¶

Ownership here is first-claim-wins, and the claim is still unproven. The controller reads the tunnel UUID out of the connector token and never asks Cloudflare whether the claimant can actually use that tunnel. So a tenant who can create a Secret, a GatewayConfig and a Gateway in their own namespace can name any UUID — including one they have no access to — and become its holder, with a token that will never connect.

The practical consequence is a denial of service, not a disclosure: the rightful owner of that tunnel is then refused, permanently, and no amount of waiting fixes it. The rule protects a tunnel that is already being served; it cannot tell a legitimate first claim from a squatted one.

That denial of service is not confined to this cluster. Arbitration can only contest a tunnel some Gateway represents, so a UUID naming any OTHER tunnel in the same Cloudflare account — another cluster's, another product's, a bare cloudflared — is uncontested, and a GatewayConfig without its own cloudflareCredentialsSecretRef writes with the credentials resolved from the GatewayClass. The controller then overwrites that tunnel's ingress document with the claiming tenant's routes, using the operator's API token, and the outage lands on a service that has nothing to do with this cluster. The account is the blast radius, not the cluster; scope the API token to the tunnels this controller should manage, and give tenants who need isolation their own account rather than their own tunnel.

A claim that has not yet advertised anything is not defended while its configuration is unreadable. An established holder survives a token rotation because its address carries the claim, but a Gateway still bootstrapping — no address written yet — drops out of the arbitration entirely if its GatewayConfig, its token Secret, or the class credentials Secret it falls back to cannot be read at that moment, and a competing claim on the same tunnel is then accepted. The alternative, refusing to arbitrate at all on any unreadable claim, would let any tenant freeze the rule cluster-wide by deleting their own Secret.

Removing a refused Gateway's data plane relies on the controller ownerReference it stamped on the proxy Deployment: the cleanup deliberately leaves alone any object whose reference has been stripped, so it never deletes something another controller now owns. A tenant with write access in their own namespace can therefore strip that reference and keep the pod, and so the connector, running. It gets no route config, so it serves nothing of its own — but on a SHARED token it stays a registered connector on the contested tunnel.

Possession is read from Gateway.status.addresses, so anyone able to write gateways/status in their own namespace can forge it and evict a real holder immediately — possession beats age, so that is faster than the squat above. The Gateway API CRDs carry no RBAC aggregation labels, so the built-in edit and admin roles do not grant it; if you grant status write to tenants, this rule does not hold for them.

Watch for it in the TunnelClaimRejected Warning Events: a Gateway you believe owns a tunnel reporting Accepted=False against a claimant you do not recognise is this case. The remedy is operator-side: delete the squatting Gateway, or remove its spec.infrastructure.parametersRef. The rightful owner's status recovers on its own reconcile, within about half a minute. Nothing wakes it sooner: tunnel squatting is cross-namespace by definition, and the sibling-Gateway watch that makes a data-plane cap recover promptly lists only the event object's own namespace. From that reconcile on, no route change is needed — re-rendering the plane runs a full route sync immediately, and the config reaches the new proxy pods when they become Ready and the EndpointSlice change fires the partition resync. Deleting only the GatewayConfig does NOT release the claim — a Gateway that already advertises a tunnel keeps holding it even when its configuration no longer resolves, which is what stops a token rotation from surrendering a tunnel. Verifying claims against the Cloudflare API would close it properly; that is tracked in issue #679.

Capping data planes per namespace¶

Every opted-in Gateway renders a proxy Deployment, a headless Service, a NetworkPolicy and an optional HPA into its namespace, and registers a connector on its tunnel. Nothing about the opt-in bounds how many a tenant may ask for: a namespace with create on Gateway and GatewayConfig can multiply that as far as its object quota allows.

maxDataPlanesPerNamespace on the cluster-scoped GatewayClassConfig (chart value gatewayClassConfig.maxDataPlanesPerNamespace) caps it. Leave it unset for no cap, so nothing changes until an operator sets one. 0 is rejected by the apiserver rather than accepted as another spelling of unlimited: it is what an operator writes for "no dedicated planes at all", and a field that granted the opposite would fail open. Like allowSharedTunnels, the field is deliberately not on GatewayConfig — a tenant must not be able to raise their own cap. On a cluster upgraded from a release before the field existed, re-apply the GatewayClassConfig CRD first: Helm never upgrades CRDs, and the apiserver prunes a field the installed schema does not declare, so the cap would read back unset and enforce nothing. See CRD upgrades.

apiVersion: cf.k8s.lex.la/v1alpha1
kind: GatewayClassConfig
metadata:
  name: cloudflare-tunnel-config
spec:
  tunnelID: "550e8400-e29b-41d4-a716-446655440000"
  cloudflareCredentialsSecretRef:
    name: cloudflare-credentials
  maxDataPlanesPerNamespace: 3

Past the cap, a Gateway reports Accepted=False with reason DataPlaneQuotaExceeded, emits a DataPlaneQuotaExceeded Warning Event, gets no data plane rendered, and has none of its routes programmed — its routes say the cap was the reason, so the tenant can see the cause without asking the operator. DataPlaneQuotaExceeded is an implementation-specific reason; the Gateway API allows controllers to raise Accepted with reasons outside its own list, and none of the listed ones describes a Gateway refused for capacity rather than for something wrong with its spec.

Which Gateways keep their planes is decided oldest-first by creation timestamp, breaking equal timestamps by UID the same way tunnel arbitration does. A Gateway created now is the newest in its namespace, so it takes a free slot or is refused, and never evicts one already serving. An OLDER Gateway that opts in later does evict, because the order is by creation timestamp and not by when the plane was asked for: the displaced Gateway is refused and its plane removed on the reconcile its sibling's opt-in triggers. Beyond that the verdict does not move between reconciles. Lowering the cap below what a namespace already holds does evict: the newest planes above the new limit are torn down on the next reconcile, so lower it deliberately.

Freeing a slot is not instant, but nothing has to be poked to start it. Deleting a sibling Gateway wakes both controllers on that event, so the refused Gateway's status flips and its plane is rendered without waiting for the refusal's half-minute requeue, and rendering a plane runs a full route sync immediately. What remains is pod startup: that sync resolves the headless Service before the new proxy pods are Ready, so its config reaches them when the EndpointSlice change fires the partition resync. Expect the gap between Accepted=True and traffic flowing to be roughly how long the pods take to start.

A refused Gateway keeps the tunnel CNAME already published in its status.addresses, deliberately: an address is how this controller records possession of a tunnel, and clearing it on a refusal would surrender the tunnel to any other namespace claiming it. The practical consequence after a cap lowering is that external-dns keeps the DNS record and the hostname answers HTTP 530 (no connector) rather than NXDOMAIN. Delete the Gateway to retire the record.

Only Gateways carrying spec.infrastructure.parametersRef count. The shared data plane serves any number of Gateways and is unaffected, which makes dropping the parametersRef the tenant-side remedy when a namespace has more Gateways than dedicated planes it can have.

A Gateway refused for claiming a tunnel it does not own also holds its slot, even though it gets no plane. The two rules are deliberately independent, so neither verdict moves when the other changes; the wasted slot is the refused tenant's own.

A Gateway being deleted holds its slot until it is gone. The controller adds no finalizer of its own, so the window is normally milliseconds — a Gateway held back by someone else's finalizer keeps its slot for as long as that lasts. The alternative under-counts planes whose pods are still running.

A Gateway counts even when its configuration does not currently resolve — a missing GatewayConfig, an unreadable connector-token Secret. It is asking for a plane either way, and counting only the ones that resolve would let a tenant free a slot by making one of their tokens unreadable.

The cap bounds how many dedicated planes one namespace may HOLD, not what they consume and not how many namespaces exist. A namespace capped at one plane still sets that plane's replicas or autoscaling.maxReplicas itself, and a tenant who can create namespaces can multiply planes by spreading them. Pair the cap with a ResourceQuota and restricted namespace creation where either matters.

Securing a tenant data plane¶

A tenant data plane is locked down by default on two axes:

• The config API is authenticated. When a GatewayConfig declares no authTokenSecretRef, the controller generates a random bearer-token Secret (cf-proxy-<gateway>-auth, controller-owned, never rotated) and wires the proxy to it, so a tenant plane is never exposed unauthenticated. authTokenSecretRef is a bring-your-own-token OVERRIDE for operators who want to manage the token themselves (external secret stores, scheduled rotation). The proxy reads the token from a SecretKeyRef env var at start, so the pod template hashes the resolved token: rotating the bring-your-own Secret rolls the proxy pods automatically (same as tunnelTokenSecretRef), and the controller re-authenticates its config pushes with the new token.
• The config API is network-restricted. The controller renders a NetworkPolicy per data plane that admits the config API port (8081) only from the controller pod (its labels AND'd with its namespace, not the tenant's), so neither the config API nor the /metrics it co-serves is reachable from arbitrary pods — including other pods in the controller's own namespace, and including any in the Gateway's. The proxy's data port (8080) takes no in-cluster ingress at all — traffic arrives through the outbound tunnel. Because /metrics shares the locked port, a policy that admits only the controller silently breaks Prometheus scraping and therefore the rendered HPA (it reports FailedGetPodsMetric and holds minReplicas); set proxy.networkPolicy.monitoringNamespaceSelector in the chart to additionally admit your monitoring namespace. Where the CNI does not enforce NetworkPolicy this layer is a no-op (defense in depth, not a guarantee).

Both axes bound reach, not confidentiality: the controller's push is plain HTTP carrying the bearer token and, where the Gateway configures backend mTLS, the client certificate's private key, so on a CNI without pod-to-pod encryption an on-path party inside the cluster reads them — see Config API Authentication.

Also note the RBAC equivalence: create on GatewayConfig (plus a Gateway referencing it) lets a user run an arbitrary image via spec.image under the namespace's default ServiceAccount — see the security reference.

Operational notes¶

• Events: the controller emits ProxyProvisioned (Normal) on the Gateway when the data plane is rendered, and RenderFailed (Warning) when rendering cannot proceed (apply failures) — kubectl describe gateway shows both.
• No proxy image configured: if neither GatewayConfig.spec.image nor the controller's --proxy-image default is set, the data plane cannot be rendered. The Gateway surfaces Accepted=False with reason InvalidParameters and a message naming the missing image (a persistent condition, not just a transient Event) — set one of the two and the Gateway recovers on the next reconcile.
• Namespace at its data-plane cap: the Gateway surfaces Accepted=False with reason DataPlaneQuotaExceeded and a DataPlaneQuotaExceeded Warning Event, nothing is rendered for it, and its routes are programmed nowhere. Delete one of the namespace's other dedicated Gateways, drop this one's spec.infrastructure.parametersRef to serve it from the shared plane, or ask the operator to raise maxDataPlanesPerNamespace. See Capping data planes per namespace.
• Drain: on pod shutdown the proxy unregisters its connectors from the edge and gives in-flight requests a grace period before exiting; the rendered terminationGracePeriodSeconds covers the window.
• RBAC: rendering requires cluster-wide write on Deployments/Services/HPAs (Gateways live in arbitrary namespaces); see the security reference for the exact rules and ownership guards.
• Failure containment: a tunnel-sync failure for one Gateway's tunnel marks only THAT Gateway's routes Pending; other tenants' route statuses are untouched.
• Post-render breakage: if the GatewayConfig stops resolving AFTER a healthy render (token Secret deleted, ref broken), new route changes fail closed — they are not programmed anywhere — but the already-running data plane keeps serving its LAST pushed config until the configuration resolves again or the Gateway is deleted. The Gateway surfaces InvalidParameters, and a route bound only to the broken Gateway reports Accepted=False on that parent (it is served nowhere) — a route accepted on a healthy parent too keeps Accepted=True there.
• Not rendered: per-Gateway PodDisruptionBudget and ServiceMonitor. The shared plane's ServiceMonitor does not select rendered pods; scrape them with a PodMonitor on the cf.k8s.lex.la/gateway label if needed (and admit the monitoring namespace via proxy.networkPolicy.monitoringNamespaceSelector, since the rendered NetworkPolicy locks the metrics port).
• Cloudflare-side residue on teardown: deleting the Gateway (or removing infrastructure.parametersRef) garbage-collects the in-cluster resources, but the LAST ingress document written to the dedicated Cloudflare Tunnel is left in place. It is inert — the connectors are gone, so nothing serves it — but it is not erased from the Cloudflare API. Delete or repurpose the tunnel on the Cloudflare side if you need the account state clean.
• Sustained config-push failure is surfaced: if the controller cannot push a route's config to its data plane for a sustained run of attempts (the proxy pods are unhealthy or the config-API NetworkPolicy blocks the push), the affected routes carry a cf.k8s.lex.la/ProxyConfigPushed=False condition and a ProxyConfigPushFailed Warning Event — the route spec is valid and the tunnel document was written, but the in-cluster proxy never received the config, so matching requests 502. Accepted/Programmed are unchanged (the failure does not flap them on a one-off blip), and the condition clears automatically on the first successful push.
• API-credential rotation is watched: rotating the cloudflareCredentialsSecretRef override (the controller-side API token, distinct from the connector token) re-syncs the Gateway's routes promptly — the controller watches that Secret. Connector-token (tunnelTokenSecretRef) rotation also re-renders the proxy Deployment.
• Well-known labels: every rendered resource (Deployment, headless Service, HPA, NetworkPolicy) carries the GEP-1762 well-known labels gateway.networking.k8s.io/gateway-name and gateway.networking.k8s.io/gateway-class-name on its metadata (never the immutable selector), so ecosystem tooling that understands the Gateway API convention can discover a Gateway's dedicated data plane without a controller-specific label. Both values go through the same truncation as the cf.k8s.lex.la/gateway selector label: Gateway and GatewayClass names are DNS-1123 subdomains up to 253 characters, but Kubernetes label values cap at 63, so tooling matching on the exact Gateway or GatewayClass name will miss for names longer than that.