Troubleshooting¶

This guide helps diagnose and resolve common issues with the Cloudflare Tunnel Gateway Controller.

Quick Diagnostics¶

# Check controller status
kubectl get pods --namespace cloudflare-tunnel-system

# View controller logs
kubectl logs --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller

# Check Gateway status
kubectl get gateway --all-namespaces

# Check HTTPRoute status
kubectl get httproute --all-namespaces

Installation Issues¶

Schema Validation Errors¶

Problem: values don't meet the specifications of the schema

Common causes:

1. Empty tunnelID:

   Error: execution error at (...gatewayclassconfig.yaml:20:16): gatewayClassConfig.tunnelID is required
   

   Solution: Provide a valid Tunnel ID from Cloudflare Zero Trust Dashboard

2. Invalid characters in tunnelID:

   Error: at '/gatewayClassConfig/tunnelID': pattern mismatch
   

   Solution: Use a valid UUID from the Cloudflare Zero Trust Dashboard (format: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx with lowercase hex digits a-f and 0-9 only)

3. Missing API credentials:

   Error: execution error at (...gatewayclassconfig.yaml:10:12): gatewayClassConfig.cloudflareCredentialsSecretRef.name is required
   

   Solution: Set gatewayClassConfig.cloudflareCredentialsSecretRef.name to the name of a Secret containing an api-token key

4. Missing proxy tunnel token:

   Error: execution error at (...deployment-proxy.yaml:41:18): proxy.tunnelTokenSecretRef.name is required
   

   Solution: Set proxy.tunnelTokenSecretRef.name to the name of a Secret containing a tunnel-token key (see Getting Started / Prerequisites). This field is required on every install, before any gatewayClassConfig.* value.

Verification:

# Validate values before installation
helm lint charts/cloudflare-tunnel-gateway-controller --values my-values.yaml

# Dry-run installation
helm install --dry-run --debug my-release \
  oci://ghcr.io/lexfrei/charts/cloudflare-tunnel-gateway-controller \
  --values my-values.yaml

Pod Startup Problems¶

CrashLoopBackOff¶

Diagnosis:

kubectl get pods --namespace cloudflare-tunnel-system

kubectl logs --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller

kubectl describe pod --namespace cloudflare-tunnel-system \
  --selector app.kubernetes.io/name=cloudflare-tunnel-gateway-controller

Common causes:

Config API Auth Secret Missing or Broken¶

Symptoms:

• Controller pod in CrashLoopBackOff immediately after startup, before it reconciles anything
• Every Gateway managed by this controller loses reconciliation, not just one
• If a proxy pod is also (re)starting against the same broken Secret, it crash-loops too instead of serving anything unauthenticated

Diagnosis:

kubectl logs --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller

error: failed to run controller: resolving shared-proxy auth secret: cloudflare-tunnel-system/cloudflare-tunnel-gateway-controller-proxy-auth-token key "auth-token": shared-proxy auth secret exists but has no usable value at the expected key

A proxy pod hitting the same broken Secret on its own startup logs (JSON, via its structured logger) instead:

kubectl logs --namespace cloudflare-tunnel-system \
  --selector app.kubernetes.io/component=proxy

{"time":"...","level":"ERROR","msg":"refusing to start with a broken config-API auth configuration","error":"PROXY_AUTH_TOKEN is set but empty"}

Cause: a Secret already exists at the name the controller expects for the shared proxy's config-API bearer token, but its auth-token key is missing or empty. Most often this is a hand-created Secret with a typo'd key name. The controller resolves this token once at startup, before wiring any reconciler, and fails closed rather than start with an unusable token: the config API must never be left unauthenticated. Unlike a broken per-Gateway auth Secret, which only stops that one Gateway's data plane, this stops the controller process entirely, because the resolution happens before any Gateway is reconciled.

The controller cannot repair this itself. Its RBAC on Secrets grants create but not update.

Solution: delete the broken Secret and restart both the controller and the proxy, so neither is left running with an empty or stale token.

kubectl delete secret cloudflare-tunnel-gateway-controller-proxy-auth-token \
  --namespace cloudflare-tunnel-system

kubectl rollout restart deployment/cloudflare-tunnel-gateway-controller \
  --namespace cloudflare-tunnel-system

kubectl rollout restart deployment/cloudflare-tunnel-gateway-controller-proxy \
  --namespace cloudflare-tunnel-system

If proxy.authTokenSecretRef.name is set to a Secret you manage yourself, the controller never creates or repairs it; fix the key in that Secret instead, then restart the proxy the same way so it picks up the corrected value.

Proxy Refuses to Start Without a Config-API Token¶

The proxy exits immediately in tunnel mode with:

{"time":"...","level":"ERROR","msg":"refusing to start with a broken config-API auth configuration","error":"PROXY_AUTH_TOKEN is not set: the config API would accept an unauthenticated routing table. Set a token, or set PROXY_ALLOW_UNAUTHENTICATED_CONFIG_API=1 (or =true) to run without one"}

The config API listens on every interface and one successful PUT /config replaces the whole routing table, so tunnel mode will not start without a Bearer token. Chart installs never see this: the chart wires PROXY_AUTH_TOKEN unconditionally and the controller generates a token when you do not supply one. It reaches you on a hand-written proxy Deployment.

Wire a token, or set PROXY_ALLOW_UNAUTHENTICATED_CONFIG_API=1 if the config API is reachable only over a path you already control. The acknowledgement is read as 1 or true; any other spelling is treated as unset, and the message above is what you get.

A PROXY_AUTH_TOKEN that is set but empty is a different problem — see Config API Auth Secret Missing or Broken. The acknowledgement does not cover it, in either mode.

Proxy Pod Stuck NotReady After a Restart¶

Symptoms:

• A proxy pod is Running but 0/1 Ready for longer than expected after a restart or node reboot, with a low or zero RESTARTS count in kubectl get pods — this is the tell that distinguishes the new behavior from the old crash loop an operator might be expecting
• Logs repeat tunnel bootstrap dial failed, retrying, one line per attempt, or tunnel bootstrap: the edge rejected this connector's registration, retrying; check the tunnel token and that the tunnel still exists when the Cloudflare edge answered the registration with a refusal

Cause: this is expected, not a bug. The proxy retries the tunnel's bootstrap dial instead of exiting on a transient failure — cluster DNS not yet reachable while the CNI is still wiring pods up after a reboot, or the Cloudflare edge briefly unreachable. Each retry waits a random duration between 0 and an exponentially growing cap (2s up to 30s, jittered so that replicas which failed at the same instant do not all retry in lockstep), so the logged backoff value moves around rather than climbing in a straight line. /readyz stays false for the whole retry window; the pod does not crash-loop.

Diagnosis: a proxy that is retrying, not wedged, keeps emitting a new retry log line on every attempt — an attempt count that keeps climbing is the pod actively working the problem, not stuck. Follow the logs to watch it happen:

kubectl logs --namespace cloudflare-tunnel-system \
  --selector app.kubernetes.io/component=proxy --follow

Solution: if the retry log line keeps repeating well past the time the underlying network issue should have cleared (DNS, egress to the Cloudflare edge), check the error text on each attempt:

• A DNS/dial/edge-unreachable error clears on its own once connectivity is restored — no action needed.
• the edge rejected this connector's registration means the edge answered the registration with a refusal, which is what a revoked token, a deleted tunnel or a wrong secret produces. A registration that timed out or lost its connection logs the generic line instead. The pod keeps retrying rather than exiting, because cloudflared treats an Unauthorized rejection as possibly transient: a tunnel created moments ago can be refused until it propagates across the edge, and that case clears on its own. If the line keeps repeating, check that the tunnel still exists in the Cloudflare Zero Trust Dashboard, then regenerate the connector token and update proxy.tunnelTokenSecretRef.
• A malformed token (fails to decode) makes the pod exit immediately instead of retrying.

Proxy Pod Restarting Without a Config¶

Symptoms:

• A proxy pod is 0/1 Ready and its RESTARTS count keeps climbing, one restart per two-minute wait plus kubelet's back-off between restarts
• Its logs show waiting for the first config before dialing the edge, then tunnel error with not registering with the edge: waited 2m0s: no config received from the controller

Cause: in tunnel mode the proxy registers with the Cloudflare edge only after the controller has pushed it a config, because the edge sends traffic to every registered connector and one without a routing table would answer every request with a 404. When no push arrives within two minutes the proxy exits rather than wait forever, and kubelet restarts it. Something is stopping the controller's push: the controller is not running or has not been elected leader, or its pushes to this pod fail. For a Service that sets publishNotReadyAddresses: true, as the chart's does, a pod that cluster DNS does not return yet is usually not a cause on its own: the controller compares the addresses its replay reached with the proxy EndpointSlice and retries every ten seconds until DNS returns the pod. The check covers every EndpointSlice of the Service in the same address family, so a Service split over several slices is checked as a whole. A --proxy-endpoints Service without that setting is a cause: DNS never returns a pod that is not Ready yet, and a joining proxy is not Ready until it has its first config, so no push ever reaches it.

Diagnosis: check the controller's logs for failed to push config to endpoint and failed to resync config to endpoint lines naming this pod's IP; the error text says whether the push was refused (auth token, config-API TLS) or never connected (for example, a NetworkPolicy blocks it). A repeated proxy config replay missed pods DNS did not return yet line naming this pod's IP means cluster DNS still does not return the pod. A no successfully pushed config to resync warning names a data plane that no replica has accepted a config from, which a replay skips. It shows when the controller replays to every plane because a proxy EndpointSlice was deleted or could not be attributed to one plane, or for a plane whose Service's EndpointSlices list no pod to configure. A repeated no proxy pod of the partition has taken its config yet warning means no pod of that data plane has taken its first config yet; the failed to push config to endpoint lines next to it say why. A Service without publishNotReadyAddresses: true produces neither a push error nor a missed pods line for the pod: the sign is a Ready=false address for it in the proxy EndpointSlice and no controller log line naming its IP at all.

kubectl logs --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller | grep --extended-regexp 'failed to (push|resync) config|replay missed pods'

Solution: fix what the controller's push error names. When no push or resync line carries the pod's IP and --proxy-endpoints names a Service the chart did not render, give that Service publishNotReadyAddresses: true, so DNS returns pods that are not Ready yet; see Proxy Endpoints. Once a push lands, the waiting pod dials the edge and turns Ready without a restart.

ImagePullBackOff¶

Diagnosis:

kubectl describe pod --namespace cloudflare-tunnel-system POD_NAME

Solutions:

image:
  repository: ghcr.io/lexfrei/cloudflare-tunnel-gateway-controller
  tag: ""  # Uses appVersion from Chart.yaml
  pullPolicy: IfNotPresent

# For private registries
imagePullSecrets:
  - name: ghcr-credentials

Authentication and API Issues¶

Invalid Cloudflare API Token¶

Symptoms:

• TunnelDocumentWriteFailed Warning Events on Gateways
• cftunnel_sync_errors_total keeps rising
• Logs show 401 Unauthorized or 403 Forbidden

Routes keep serving and their status does not change: the tunnel's ingress document, which this token writes, only feeds the Cloudflare dashboard. A dedicated Gateway's tunnel claim is also checked with the token, see Limitations.

Diagnosis:

# Test API token manually
export CF_API_TOKEN="your-token"
curl --header "Authorization: Bearer $CF_API_TOKEN" \
  https://api.cloudflare.com/client/v4/user/tokens/verify

Solution:

1. Create new API token with required scopes:

2. Account > Cloudflare Tunnel > Edit

3. Update secret:

   kubectl create secret generic cloudflare-credentials \
     --from-literal=api-token="NEW_TOKEN" \
     --namespace cloudflare-tunnel-system \
     --dry-run=client --output yaml | kubectl apply --filename -
   
   kubectl rollout restart deployment/cloudflare-tunnel-gateway-controller \
     --namespace cloudflare-tunnel-system
   

Network Connectivity¶

NetworkPolicy Blocking Traffic¶

Symptoms:

• Metrics not accessible from Prometheus
• Health checks failing
• Cannot communicate with Cloudflare API

Diagnosis:

# Check NetworkPolicy rules
kubectl get networkpolicy --namespace cloudflare-tunnel-system

# Test connectivity from debug pod
kubectl run debug --rm -it --image=nicolaka/netshoot \
  --namespace cloudflare-tunnel-system -- bash

# Inside debug pod:
curl http://cloudflare-tunnel-gateway-controller:8080/metrics
curl http://cloudflare-tunnel-gateway-controller:8081/healthz
curl --head https://api.cloudflare.com

With networkPolicy.enabled: true the controller admits only the sources listed in networkPolicy.ingress.from to its metrics and health ports, so the first two requests from a debug pod fail unless the pod matches one of them. An empty list admits no in-cluster source.

DNS Resolution Issues¶

Symptoms:

• Cannot resolve Cloudflare API endpoints
• Errors: no such host or dial tcp: lookup failed

Diagnosis:

kubectl exec --namespace cloudflare-tunnel-system POD_NAME -- \
  cat /etc/resolv.conf

kubectl exec --namespace cloudflare-tunnel-system POD_NAME -- \
  nslookup api.cloudflare.com

Solution: Configure custom DNS

dnsPolicy: "None"
dnsConfig:
  nameservers:
    - 1.1.1.1
    - 8.8.8.8
  searches:
    - cloudflare-tunnel-system.svc.cluster.local
    - svc.cluster.local
    - cluster.local

Gateway API Resources¶

Gateway Not Ready¶

Diagnosis:

kubectl get gateway --all-namespaces
kubectl describe gateway my-gateway --namespace my-namespace

kubectl logs --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller | grep -i gateway

Common causes:

HTTPRoute Not Attached¶

Diagnosis:

kubectl describe httproute my-route --namespace my-namespace

Common causes:

• Namespace mismatch (use ReferenceGrant for cross-namespace)
• Invalid hostname patterns
• Backend service not found

Status Not Updating¶

Problem: Gateway/HTTPRoute status conditions not updating

Diagnosis:

kubectl auth can-i update gateways/status \
  --as=system:serviceaccount:cloudflare-tunnel-system:cloudflare-tunnel-gateway-controller

Solution: Ensure ClusterRole has status subresource permissions

Runtime Errors¶

Panic Logged in the Proxy¶

{"time":"...","level":"ERROR","msg":"panic in request handler; request failed, proxy kept running","panic":"...","host":"app.example.com","path":"/api","stack":"..."}

One request's handler panicked and the proxy contained it: that request got a 500, or its stream was reset when the response had already started, and every other connection kept running. The entry carries the host, the path and a stack trace.

This is a bug worth reporting with the stack.

A client that closes mid-download is a different thing and never produces this line: httputil.ReverseProxy raises http.ErrAbortHandler there, which the proxy treats as routine and does not log. What you do see for those is one line per aborted request from the embedded connector, failed to serve incoming request on HTTP/2 or Request failed on QUIC, with no stack. That is expected traffic noise on a busy plane, not a fault: the stream is reset deliberately, because a truncated body reported as complete would be worse.

A panic inside a WebSocket session logs websocket: panic while copying, closing the session, and only that session ends. A panic during the upgrade itself — the backend dial, the handshake — happens before the copy starts and logs the ordinary line above.

Each panic that writes one of these lines is also counted in cftunnel_proxy_handler_panics_total, labelled by site, so an alert on its increase finds these without anyone reading the logs. See Metrics.

Performance Issues¶

High Memory Usage¶

Symptoms:

• Pods OOMKilled
• Memory usage growing over time

Diagnosis:

kubectl top pod --namespace cloudflare-tunnel-system

Solution:

resources:
  limits:
    memory: 512Mi  # Increase from default
  requests:
    memory: 256Mi

Slow Reconciliation¶

Problem: Changes to Gateway/HTTPRoute take long to apply

Diagnosis:

kubectl logs --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller | grep -i reconcil

Solutions:

1. Check Cloudflare API rate limits in logs
2. Verify network latency to Cloudflare API
3. Ensure sufficient resources (CPU not throttled)

Debug Logging¶

Enable debug logging for detailed diagnostics:

controller:
  logLevel: "debug"
  logFormat: "json"

helm upgrade cloudflare-tunnel-gateway-controller \
  oci://ghcr.io/lexfrei/charts/cloudflare-tunnel-gateway-controller \
  --values values.yaml \
  --namespace cloudflare-tunnel-system

kubectl logs --follow --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller

Collecting Diagnostic Information¶

# Pod status and events
kubectl get pods --namespace cloudflare-tunnel-system --output wide
kubectl describe pod --namespace cloudflare-tunnel-system POD_NAME

# Recent logs
kubectl logs --tail=100 --namespace cloudflare-tunnel-system \
  deployment/cloudflare-tunnel-gateway-controller

# Resource usage
kubectl top pod --namespace cloudflare-tunnel-system

# Gateway API resources
kubectl get gatewayclasses,gateways,httproutes --all-namespaces

Reporting Issues¶

When reporting issues, include:

1. Helm chart version: helm list --namespace cloudflare-tunnel-system
2. Kubernetes version: kubectl version
3. Cloud provider and CNI plugin
4. Relevant pod logs (sanitize secrets!)
5. Gateway/HTTPRoute manifests (sanitize sensitive data)

Report issues at: https://github.com/lexfrei/cloudflare-tunnel-gateway-controller/issues