Upgrading from v3.4 to v3.5¶
v3.5 closes five ways a request, a tenant, an unauthenticated caller, or the controller's own egress could reach something that was not theirs, and stops one request's bug from taking the whole data plane down. There is no CR migration, but the GatewayClassConfig CRD gains fields and must be re-applied by hand, and each change below can stop something that works today: a Gateway, a deployment's request routing, a hand-written proxy Deployment's startup, a scraper that shares the controller's namespace, or the render itself. Several of them need a values edit to keep working, and each section that needs one names it. All are described below with what to do about them.
Do this first: re-apply the GatewayClassConfig CRD¶
v3.5 adds two fields to GatewayClassConfig: allowSharedTunnels and maxDataPlanesPerNamespace. Helm installs the chart's crds/ files only on the first helm install and never touches them on upgrade, so an upgraded cluster keeps the v3.4 schema.
The apiserver PRUNES a field its CRD does not declare, with no error and no Event. Until this is applied, an allowSharedTunnels: true never takes effect and a maxDataPlanesPerNamespace you set reads back as unset — which means no cap is enforced, silently. Apply it once:
kubectl apply --filename https://raw.githubusercontent.com/lexfrei/cloudflare-tunnel-gateway-controller/master/charts/cloudflare-tunnel-gateway-controller/crds/cf.k8s.lex.la_gatewayclassconfigs.yaml
Change that can break: a dedicated data plane sharing a tunnel is now refused¶
Until v3.5 a Gateway with its own data plane could point its connector token at a tunnel another namespace already served, or at the GatewayClass tunnel — documented as a migration path. Both parties' routes were merged into one ingress document and the union was pushed to both parties' proxies.
That merge is now refused by default. A Gateway whose token names a tunnel it does not own:
- reports
Accepted=Falsewith reasonInvalidParametersand a message naming the tunnel, - emits a
TunnelClaimRejectedWarning Event, - has none of its routes programmed, on any plane,
- has its proxy Deployment deleted. That matters when both parties hold the same token, since both connectors register and the edge load-balances across them; a forged token cannot register on the tunnel at all, so there the removal is hygiene.
The reason for the change: a connector token is base64-encoded JSON, so any tenant who can create a GatewayConfig can write any tunnel UUID into it, and tunnel UUIDs are published in Gateway status for external-dns. Claiming a neighbour's tunnel was therefore enough to receive their routes, including any backend-mTLS client certificates those routes carry.
Who is affected¶
Only clusters that use per-Gateway data planes (Gateway.spec.infrastructure.parametersRef → GatewayConfig) AND have two of them on one tunnel, or one of them on the GatewayClass tunnel. Everything else — the shared plane, single-tenant installs, dedicated planes on distinct tunnels — is unaffected.
Two Gateways in the SAME namespace may still share a tunnel: that is a tenant sharing with itself.
Before upgrading¶
List your dedicated planes and the tunnel each one currently advertises (possession, not the token's claim, is what decides ownership — the two diverge exactly when a token has been retargeted):
kubectl get gateways --all-namespaces \
-o jsonpath='{range .items[?(@.spec.infrastructure.parametersRef)]}{.metadata.namespace}/{.metadata.name}{"\t"}{.status.addresses[0].value}{"\n"}{end}'
Any tunnel appearing twice across different namespaces is now contested: the Gateway already advertising it keeps it and the other is refused. A dedicated plane whose token matches your GatewayClassConfig.spec.tunnelID is always refused, since the class tunnel belongs to the operator.
A Gateway that has just been opted in still shows the class tunnel here, left over from the shared plane. That is not a claim on it and is not refused — only the tunnel its own token names counts against the class tunnel.
Note what the rule does not settle: the claim is never verified against Cloudflare, so first-claim-wins is exactly what it says. A tenant can name a UUID they have no access to and hold it, denying that tunnel to its real owner until an operator deletes the squatting Gateway. TunnelClaimRejected Warning Events are the signal.
If you need the old behaviour¶
Give each isolated Gateway its own Cloudflare Tunnel — that is what the feature is for. If the parties genuinely trust each other to see one another's routes (a single-tenant cluster, or a migration in progress), set the escape hatch on the cluster-scoped GatewayClassConfig:
It is deliberately not settable on the namespaced GatewayConfig, so a tenant cannot grant it to themselves.
Side effect on external-dns¶
A Gateway with its own data plane now keeps its <tunnel-id>.cfargotunnel.com status address across a configuration error, where it used to be cleared. That address is what decides tunnel ownership, and clearing it would surrender the tunnel during a token rotation.
The visible consequence: external-dns no longer withdraws the DNS record when such a Gateway's configuration breaks. The record persists until the Gateway is deleted or opts out of its dedicated plane. Gateways on the shared plane are unchanged and still clear their address on a config error.
If you ran a shared tunnel across namespaces¶
Treat anything those routes carried as having been visible to every party on that tunnel. In particular, rotate any backend-mTLS client certificates referenced by routes on the shared tunnel: the merged config that reached every party's proxy included the private keys.
Change that can break: X-Original-Host is no longer trusted¶
The proxy used to honour a client-supplied X-Original-Host header as the routing key and as the Host sent to the backend. It exists so the Gateway API conformance suite can drive domains that are not registered on the Cloudflare account, and the docs always called it a test-only pattern — but the code honoured it everywhere.
Because the Cloudflare edge forwards arbitrary X-* headers from any client, that let a client reaching one hostname be served by a different hostname's backend on the same data plane, with the intended hostname's edge policy evaluated against the wrong name.
The header is now stripped on entry unless the deployment opts in:
Leave it off in production. Turn it on only in a throwaway conformance or e2e deployment — hack/conformance-setup.sh does. If you were relying on the header outside such a deployment, routing will fall back to the real Host after the upgrade.
Two smaller changes ride along, on every deployment regardless of the flag: WebSocket backends no longer receive X-Original-Host or the proxy's internal X-Proxy-Host-Rewritten marker, and request-mirror backends no longer receive either. The plain HTTP path already stripped both, so this makes the three paths agree.
Change that can break: tunnel mode refuses to start without a config-API token¶
The proxy used to start in either mode with an unauthenticated config API when PROXY_AUTH_TOKEN was never wired, logging a warning and carrying on. In tunnel mode that now refuses to start: the config API listens on every interface and one successful push replaces the whole routing table, so no deployment should arrive there by omitting a variable.
Chart installs are unaffected — the chart wires the variable unconditionally, generating a token when you do not supply one. Standalone mode is unchanged, since it is the development mode with no edge in front of it. Note that standalone binds every interface as well, so if you run it inside a cluster its config API is reachable from anything that can route to the pod; the network boundary is yours to provide there.
Affected: a hand-written proxy Deployment (the raw deploy/ path ships none, so this is a manifest you wrote) that never set PROXY_AUTH_TOKEN. Either wire a token, or set PROXY_ALLOW_UNAUTHENTICATED_CONFIG_API=1 to keep running open. A present-but-empty PROXY_AUTH_TOKEN is still a broken configuration in both modes, and the acknowledgement does not cover it.
A TUNNEL_TOKEN that is set but empty is now refused too. It used to select standalone mode silently, so a tunnel deployment whose token Secret resolved to nothing came up as a local proxy instead of failing. Leaving the variable unset is still how standalone mode is requested.
Behaviour change: a panicking request no longer takes the proxy down¶
A panic in a request handler used to escape into cloudflared's per-stream goroutine. On QUIC nothing recovers there, and protocol: auto dials QUIC first, so one request's bug killed the pod and every other tenant's connection with it. Panics are now contained: the request gets a 500, or its stream is reset when the response had already started, and the proxy keeps running. The same guard covers the two goroutines a WebSocket session spawns, where a panic was fatal on every transport.
Nothing needs configuring. The visible consequence is log volume, and only on HTTP/2.
httputil.ReverseProxy raises http.ErrAbortHandler whenever a response copy fails, which a browser navigating away mid-download does routinely. That panic used to unwind into the HTTP/2 machinery, which discards the sentinel silently. Now it is contained, and the connector logs one failed to serve incoming request line per aborted request at error level. The proxy adds no line of its own for these, and the request outcome is unchanged.
If you alert on the proxy's error-log rate, expect a step up proportional to how often clients abandon downloads. On QUIC those same aborts previously took the whole process down, so there the change is strictly an improvement.
A genuine panic logs panic in request handler; request failed, proxy kept running with a stack. See Troubleshooting.
Change that can break: the config API admits the controller pod, not its namespace¶
Both NetworkPolicies guarding a proxy's config API used to admit the whole controller namespace. Pushing config there needs the bearer token, so the reach was not a free routing-table rewrite — but /metrics is served on that same port with no auth, so every workload sharing the namespace could read the hostname series of every tenant. The ingress peer now names the controller pod, with its labels AND'd with the namespace. The token is a reach control either way, not a confidential channel: the push is plain HTTP, so an on-path party on an unencrypted CNI reads it and any backend client key it carries — see Config API Authentication.
Affected: anything else in the release namespace that reaches the config-API port. Prometheus is the usual one, because /metrics is served on that same port. It is admitted today and denied after the upgrade, and the symptom is a scrape that stops rather than an error anywhere.
Add it back explicitly:
proxy:
networkPolicy:
ingress:
from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
Name the namespace your scraper actually runs in. If it shares the namespace with the controller, that is the release namespace, not monitoring.
For per-Gateway data planes the same job is done by proxy.networkPolicy.monitoringNamespaceSelector, which was already the documented way to admit a scraper there.
Three smaller changes ride along. The controller's own policy gained an egress rule for the proxies' config API: without it, enabling networkPolicy.enabled left data planes without a routing table. Gateway status stayed clean throughout, since the push is non-blocking; the complaint surfaced on route status after three consecutive failures, alongside an error log per push and a proxy_push sync-error metric.
The Kubernetes API egress rule gained a destination list, networkPolicy.kubernetesApiIpBlocks, empty by default. Empty renders the rule exactly as before, with no destination restriction at all, and while it stays that way the Cloudflare ranges beside it have no effect, because egress rules are OR'd. Narrowing it is now possible and is the point of the field. Check your CNI first — the egress requirements cover what a CIDR peer does and does not match.
The shared proxy no longer mounts a service account token. It never called the Kubernetes API, and its per-Gateway equivalent already refused one.
Change that can break: an emptied cloudflareIpRanges stops the render¶
networkPolicy.cloudflareIpRanges supplies the to: peers of the controller policy's Cloudflare-egress rule. Emptying both lists rendered that rule with no to: at all, and a rule with no peers permits its ports to every destination — so clearing the list to tighten egress opened 443 instead, with nothing in the output saying so. The proxy policy has refused that shape since v3.2; the controller policy now does too.
The refusal is a template fail, so it aborts the whole render rather than dropping the one rule: with networkPolicy.enabled: true and both lists empty, helm install, helm upgrade and helm template produce no manifests at all and exit with the guard's message.
Affected: a values file that empties networkPolicy.cloudflareIpRanges, either as both ipv4 and ipv6 set to empty lists or as the whole key written null. Writing it {} is not the same thing and still renders: Helm merges that with the chart's own map, so the shipped ranges survive. The chart ships both populated, so an install that never touched the field is unaffected, and one address family is enough — an IPv4-only or IPv6-only allowlist is a populated to: and still renders.
Two ways out. Either restore the ranges, keeping them current from Cloudflare's published list, or drop the policy:
Turning it off drops the whole policy — both its ingress restrictions and all of its egress rules. Note that the guard fires even where the restriction is currently inert: while networkPolicy.kubernetesApiIpBlocks is empty the rule above already permits 443 everywhere, and emptying the Cloudflare ranges to match that is exactly the change being refused.
New dependency: the GatewayClassConfig must resolve¶
Arbitration reads spec.tunnelID, spec.allowSharedTunnels and spec.maxDataPlanesPerNamespace off the GatewayClassConfig, so a Gateway with its own data plane now needs that object to exist and be referenced by a usable parametersRef. The cap is unset by default and changes nothing on upgrade; see the per-Gateway isolation guide to turn it on. Before v3.5 the per-Gateway status path never touched the class chain, and a dedicated plane got status and a rendered Deployment regardless.
A GatewayClass whose parametersRef is missing or points at the wrong kind is a permanent misconfiguration: dedicated planes on it report Accepted=False/InvalidParameters, and rendering and drift-healing stop for them until it is fixed, with one Warning Event per Gateway as the signal. Existing planes are left running rather than torn down.
If the GatewayClassConfig object itself is missing, per-Gateway reconciles return an error and retry rather than writing status or rendering — deliberately, since nothing may advertise a tunnel whose ownership cannot be decided. The dependency is satisfied by any install that already has a working GatewayClassConfig, whether the chart rendered it (gatewayClassConfig.create: true) or you reference your own object from the GatewayClass. Every functioning deployment has one, since route sync already requires it — but note the chart does NOT create one by default, so a GatewayClass whose parametersRef was never wired now reports Accepted=False on its dedicated planes instead of only failing to sync.
Reading the policy does not touch the Cloudflare credentials Secret, so rotating it does not stall arbitration. Building the claim set is a separate step that can: a GatewayConfig without its own credential override resolves through the class chain, so during a rotation a Gateway that has not yet advertised a tunnel drops out of the arbitration. An established holder is unaffected, since its address carries the claim.
What does not change¶
- No CR migration. Existing
GatewayClassConfigandGatewayConfigobjects keep working — but theGatewayClassConfigCRD itself needs the one-time re-apply above. - The shared data plane, the tunnel document format, and route status semantics are unchanged for every deployment not covered above.