Skip to content

Metrics & Alerting

The controller exposes Prometheus metrics for monitoring and alerting.

Endpoints

Endpoint Port Description
/metrics 8080 Prometheus metrics
/healthz 8081 Liveness probe
/readyz 8081 Readiness probe

Available Metrics

Route Synchronization Metrics

These metrics track the core synchronization of routes to Cloudflare Tunnel.

Metric Type Labels Description
cftunnel_sync_duration_seconds Histogram status Duration of route sync operations
cftunnel_synced_routes Gauge type Number of routes synced (http/grpc)
cftunnel_ingress_rules Gauge - Total ingress rules in tunnel config
cftunnel_failed_backend_refs Gauge type Failed backend references by route type
cftunnel_sync_errors_total Counter error_type Sync errors by type

A sustained run of error_type="proxy_push" errors for one data plane also surfaces on the affected routes as a cf.k8s.lex.la/ProxyConfigPushed=False condition (plus a Warning Event), so a proxy that stops receiving config is visible on route status, not only on this counter. The condition clears on the first successful push.

Cloudflare API Metrics

Track Cloudflare API interactions for performance and reliability monitoring.

Metric Type Labels Description
cftunnel_cloudflare_api_duration_seconds Histogram method, resource API call latency
cftunnel_cloudflare_api_calls_total Counter method, resource, status API calls count
cftunnel_cloudflare_api_errors_total Counter method, error_type API errors by type

Label values:

  • method: get, list, update
  • resource: tunnel_config, account
  • status: success, error
  • error_type: auth, rate_limit, timeout, server_error, network

Ingress Builder Metrics

Track the conversion of Gateway API routes to Cloudflare ingress rules.

Metric Type Labels Description
cftunnel_ingress_build_duration_seconds Histogram type Rule building duration
cftunnel_backend_ref_validation_total Counter type, result, reason Backend ref validation results

Controller Runtime Metrics

Built-in metrics from controller-runtime.

Metric Type Description
controller_runtime_reconcile_total Counter Total reconciliations per controller
controller_runtime_reconcile_errors_total Counter Total reconciliation errors
controller_runtime_reconcile_time_seconds Histogram Reconciliation duration
controller_runtime_max_concurrent_reconciles Gauge Max concurrent reconciles
controller_runtime_active_workers Gauge Current active workers

Workqueue Metrics

Metric Type Description
workqueue_adds_total Counter Items added to queue
workqueue_depth Gauge Current queue depth
workqueue_queue_duration_seconds Histogram Time in queue
workqueue_work_duration_seconds Histogram Processing time
workqueue_retries_total Counter Item retries
workqueue_longest_running_processor_seconds Gauge Longest running item

Go Runtime Metrics

Metric Type Description
go_goroutines Gauge Number of goroutines
go_gc_duration_seconds Summary GC pause duration
go_memstats_alloc_bytes Gauge Allocated memory
go_memstats_heap_inuse_bytes Gauge Heap in use

Process Metrics

Metric Type Description
process_cpu_seconds_total Counter CPU time used
process_resident_memory_bytes Gauge Resident memory
process_open_fds Gauge Open file descriptors

Proxy (Data-Plane) Metrics

The in-process L7 proxy serves its own exposition at /metrics on the config API port (8081, no auth — the endpoint carries no secrets; the Bearer token protects config writes). The chart's proxy ServiceMonitor scrapes it when serviceMonitor.enabled is set (and proxy.metrics.enabled is on — the template is gated on both); metrics are on by default and can be disabled with proxy.metrics.enabled: false. The same endpoint also surfaces the embedded cloudflared connector metrics (cloudflared_tunnel_*).

On a multi-tenant shared plane the exposition reveals per-tenant hostname series to anything that can reach the pod, so the chart ships this NetworkPolicy on by default (proxy.networkPolicy.enabled: true) — it admits the config API port (and therefore /metrics) only from the controller's own namespace. To keep Prometheus scraping working, admit your monitoring namespace too: add it to proxy.networkPolicy.ingress.from for the shared proxy, or set proxy.networkPolicy.monitoringNamespaceSelector for the per-Gateway data planes the controller renders. Where the CNI does not enforce NetworkPolicy this is a no-op and scraping is unaffected.

Strict CNIs and kubelet health probes

The proxy's liveness/readiness/startup probes also target the config API port, and they originate from the node's kubelet (host network), not from a pod — so a namespaceSelector ingress rule does not match them. Most CNIs (Calico, Cilium) permit node→pod health-check traffic regardless of NetworkPolicy, so probes work out of the box. A CNI configured to enforce policy on host→pod traffic will block them and CrashLoop the proxy; on such clusters add an ingress rule admitting the kubelet/node source for the config API port, or run with proxy.networkPolicy.enabled: false.

Series count scales with the number of CONFIGURED hostnames — a tenant can legitimately mint series by creating many hostnames under its allowed suffix, so budget Prometheus accordingly on hostname-heavy shared planes.

The hostname label always carries the MATCHED route hostname pattern (exact host, *.suffix wildcard pattern, or empty for default-bucket and unmatched requests) — never the raw client Host — so series cardinality is bounded by the pushed config.

Metric Type Labels Description
cftunnel_proxy_requests_in_flight Gauge — Requests currently being served (excluding hijacked WebSocket sessions). The saturation signal for horizontal scaling.
cftunnel_proxy_websocket_active_sessions Gauge — Live post-upgrade WebSocket sessions.
cftunnel_proxy_request_duration_seconds Histogram hostname Wall time from arrival to response completion; WebSocket upgrades observe time-to-upgrade, not session lifetime.
cftunnel_proxy_requests_total Counter hostname, status_class Completed exchanges by status class: 1xx..5xx, aborted (no response written — e.g. client canceled first), other (handler wrote a status outside 100-599). A WebSocket upgrade counts as 1xx at hijack time.
cftunnel_proxy_backend_errors_total Counter hostname, reason Backend dial/connect failures (dial, timeout, tls, canceled, ws_dial, ws_handshake, other).
cftunnel_proxy_response_bytes_total Counter hostname Response body bytes written (post-hijack WebSocket bytes excluded).
cftunnel_proxy_request_bytes_total Counter hostname Request body bytes read.

Scaling the proxy on concurrency

The proxy is an I/O-bound L7 hop behind an outbound tunnel: CPU is a poor saturation proxy and there is no Service load balancer producing ingress signals. Scale on the in-flight gauge instead. Two ways to feed it to an HPA:

KEDA (Prometheus scaler):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: proxy-inflight
spec:
  scaleTargetRef:
    name: <proxy-deployment>
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(cftunnel_proxy_requests_in_flight{job="<release>-proxy"})
        threshold: "50"

prometheus-adapter (Pods metric for a native autoscaling/v2 HPA):

# prometheus-adapter values: expose the gauge through the custom-metrics API
rules:
  custom:
    - seriesQuery: 'cftunnel_proxy_requests_in_flight{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)

Per-Gateway data planes render this HPA automatically when GatewayConfig.spec.autoscaling is set — see Per-Gateway Isolation.

Prometheus Configuration

ServiceMonitor (Prometheus Operator)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cloudflare-tunnel-gateway-controller
  namespace: cloudflare-tunnel-system
  labels:
    app.kubernetes.io/name: cloudflare-tunnel-gateway-controller
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: cloudflare-tunnel-gateway-controller
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

Scrape Config (Prometheus)

scrape_configs:
  - job_name: cloudflare-tunnel-gateway-controller
    kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
            - cloudflare-tunnel-system
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        action: keep
        regex: cloudflare-tunnel-gateway-controller
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        action: keep
        regex: metrics

PromQL Queries

Route Sync Performance

# Sync duration P95
histogram_quantile(0.95,
  sum(rate(cftunnel_sync_duration_seconds_bucket[5m])) by (le)
)

# Sync error rate
sum(rate(cftunnel_sync_errors_total[5m])) by (error_type)

# Routes synced by type
sum(cftunnel_synced_routes) by (type)

# Total ingress rules
cftunnel_ingress_rules

# Failed backend references
sum(cftunnel_failed_backend_refs) by (type)

Cloudflare API Health

# API success rate
sum(rate(cftunnel_cloudflare_api_calls_total{status="success"}[5m]))
/
sum(rate(cftunnel_cloudflare_api_calls_total[5m]))

# API latency P99
histogram_quantile(0.99,
  sum(rate(cftunnel_cloudflare_api_duration_seconds_bucket[5m])) by (le, method)
)

# API errors by type
sum(rate(cftunnel_cloudflare_api_errors_total[5m])) by (method, error_type)

Reconciliation Rate

# Reconciliations per second by controller
sum(rate(controller_runtime_reconcile_total[5m])) by (controller)

# Error rate
sum(rate(controller_runtime_reconcile_errors_total[5m])) by (controller)

# Error percentage
sum(rate(controller_runtime_reconcile_errors_total[5m])) by (controller)
/
sum(rate(controller_runtime_reconcile_total[5m])) by (controller)
* 100

Reconciliation Latency

# P50 latency
histogram_quantile(0.50,
  sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller)
)

# P95 latency
histogram_quantile(0.95,
  sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller)
)

# P99 latency
histogram_quantile(0.99,
  sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller)
)

Queue Health

# Queue depth (should be low)
workqueue_depth{name=~".*gateway.*|.*httproute.*"}

# Average time in queue
sum(workqueue_queue_duration_seconds_sum) by (name)
/
sum(workqueue_queue_duration_seconds_count) by (name)

# Processing time
sum(workqueue_work_duration_seconds_sum) by (name)
/
sum(workqueue_work_duration_seconds_count) by (name)

Resource Usage

# Memory usage
process_resident_memory_bytes{job="cloudflare-tunnel-gateway-controller"}

# CPU usage
rate(process_cpu_seconds_total{job="cloudflare-tunnel-gateway-controller"}[5m])

# Goroutines
go_goroutines{job="cloudflare-tunnel-gateway-controller"}

Alerting Rules

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: cloudflare-tunnel-gateway-controller
  namespace: cloudflare-tunnel-system
spec:
  groups:
    - name: cloudflare-tunnel-sync
      rules:
        - alert: CloudflareTunnelSyncErrors
          expr: |
            sum(rate(cftunnel_sync_errors_total[5m])) > 0.1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High sync error rate"
            description: "Controller experiencing {{ $value | humanize }} sync errors/sec"

        - alert: CloudflareTunnelSyncSlow
          expr: |
            histogram_quantile(0.95,
              sum(rate(cftunnel_sync_duration_seconds_bucket[5m])) by (le)
            ) > 10
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Slow route synchronization"
            description: "P95 sync duration is {{ $value | humanizeDuration }}"

        - alert: CloudflareTunnelFailedBackendRefs
          expr: |
            sum(cftunnel_failed_backend_refs) > 0
          for: 15m
          labels:
            severity: warning
          annotations:
            summary: "Failed backend references"
            description: "{{ $value }} backend references are failing validation"

    - name: cloudflare-tunnel-api
      rules:
        - alert: CloudflareTunnelAPIErrors
          expr: |
            sum(rate(cftunnel_cloudflare_api_errors_total[5m])) by (error_type) > 0
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Cloudflare API errors"
            description: "{{ $labels.error_type }} errors: {{ $value | humanize }}/sec"

        - alert: CloudflareTunnelAPISlow
          expr: |
            histogram_quantile(0.99,
              sum(rate(cftunnel_cloudflare_api_duration_seconds_bucket[5m])) by (le)
            ) > 10
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Cloudflare API latency high"
            description: "P99 API latency is {{ $value | humanizeDuration }}"

        - alert: CloudflareTunnelAPIRateLimited
          expr: |
            sum(rate(cftunnel_cloudflare_api_errors_total{error_type="rate_limit"}[5m])) > 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Cloudflare API rate limited"
            description: "Controller is being rate limited by Cloudflare API"

    - name: cloudflare-tunnel-controller
      rules:
        - alert: CloudflareTunnelControllerHighErrorRate
          expr: |
            sum(rate(controller_runtime_reconcile_errors_total[5m])) by (controller)
            /
            sum(rate(controller_runtime_reconcile_total[5m])) by (controller)
            > 0.1
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "High reconciliation error rate"
            description: "Controller {{ $labels.controller }} has error rate {{ $value | humanizePercentage }}"

        - alert: CloudflareTunnelControllerSlowReconciliation
          expr: |
            histogram_quantile(0.99,
              sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller)
            ) > 30
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Slow reconciliation"
            description: "Controller {{ $labels.controller }} P99 latency is {{ $value | humanizeDuration }}"

        - alert: CloudflareTunnelControllerQueueBacklog
          expr: workqueue_depth{name=~".*gateway.*|.*httproute.*"} > 100
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Workqueue backlog"
            description: "Queue {{ $labels.name }} has {{ $value }} items pending"

        - alert: CloudflareTunnelControllerDown
          expr: up{job="cloudflare-tunnel-gateway-controller"} == 0
          for: 2m
          labels:
            severity: critical
          annotations:
            summary: "Controller is down"
            description: "Cloudflare Tunnel Gateway Controller is not responding"

        - alert: CloudflareTunnelControllerHighMemory
          expr: |
            process_resident_memory_bytes{job="cloudflare-tunnel-gateway-controller"}
            > 512 * 1024 * 1024
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "High memory usage"
            description: "Controller using {{ $value | humanize1024 }}B memory"

Grafana Dashboard

Example dashboard panels:

{
  "title": "Cloudflare Tunnel Gateway Controller",
  "panels": [
    {
      "title": "Routes Synced",
      "type": "stat",
      "gridPos": { "x": 0, "y": 0, "w": 6, "h": 4 },
      "targets": [
        {
          "expr": "sum(cftunnel_synced_routes) by (type)",
          "legendFormat": "{{ type }}"
        }
      ]
    },
    {
      "title": "Ingress Rules",
      "type": "stat",
      "gridPos": { "x": 6, "y": 0, "w": 6, "h": 4 },
      "targets": [
        {
          "expr": "cftunnel_ingress_rules"
        }
      ]
    },
    {
      "title": "Failed Backend Refs",
      "type": "stat",
      "gridPos": { "x": 12, "y": 0, "w": 6, "h": 4 },
      "targets": [
        {
          "expr": "sum(cftunnel_failed_backend_refs)",
          "legendFormat": "failed"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "steps": [
              { "value": 0, "color": "green" },
              { "value": 1, "color": "red" }
            ]
          }
        }
      }
    },
    {
      "title": "Sync Duration (P95)",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 4, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(cftunnel_sync_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "p95"
        },
        {
          "expr": "histogram_quantile(0.50, sum(rate(cftunnel_sync_duration_seconds_bucket[5m])) by (le))",
          "legendFormat": "p50"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "s" }
      }
    },
    {
      "title": "Cloudflare API Latency",
      "type": "timeseries",
      "gridPos": { "x": 12, "y": 4, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(cftunnel_cloudflare_api_duration_seconds_bucket[5m])) by (le, method))",
          "legendFormat": "p99 {{ method }}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "s" }
      }
    },
    {
      "title": "API Calls/sec",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 12, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "sum(rate(cftunnel_cloudflare_api_calls_total[5m])) by (method, status)",
          "legendFormat": "{{ method }} ({{ status }})"
        }
      ]
    },
    {
      "title": "Sync Errors/sec",
      "type": "timeseries",
      "gridPos": { "x": 12, "y": 12, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "sum(rate(cftunnel_sync_errors_total[5m])) by (error_type)",
          "legendFormat": "{{ error_type }}"
        }
      ]
    },
    {
      "title": "Reconciliations/sec",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 20, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "sum(rate(controller_runtime_reconcile_total[5m])) by (controller)",
          "legendFormat": "{{ controller }}"
        }
      ]
    },
    {
      "title": "Reconciliation Latency",
      "type": "timeseries",
      "gridPos": { "x": 12, "y": 20, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "histogram_quantile(0.95, sum(rate(controller_runtime_reconcile_time_seconds_bucket[5m])) by (le, controller))",
          "legendFormat": "p95 {{ controller }}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "s" }
      }
    },
    {
      "title": "Queue Depth",
      "type": "timeseries",
      "gridPos": { "x": 0, "y": 28, "w": 12, "h": 8 },
      "targets": [
        {
          "expr": "workqueue_depth",
          "legendFormat": "{{ name }}"
        }
      ]
    },
    {
      "title": "Memory Usage",
      "type": "stat",
      "gridPos": { "x": 12, "y": 28, "w": 6, "h": 4 },
      "targets": [
        {
          "expr": "process_resident_memory_bytes{job=\"cloudflare-tunnel-gateway-controller\"}"
        }
      ],
      "fieldConfig": {
        "defaults": { "unit": "bytes" }
      }
    },
    {
      "title": "Goroutines",
      "type": "stat",
      "gridPos": { "x": 18, "y": 28, "w": 6, "h": 4 },
      "targets": [
        {
          "expr": "go_goroutines{job=\"cloudflare-tunnel-gateway-controller\"}"
        }
      ]
    }
  ]
}

Health Checks

Liveness Probe

curl http://localhost:8081/healthz
# Returns: ok

Readiness Probe

curl http://localhost:8081/readyz
# Returns: ok

Kubernetes Configuration

livenessProbe:
  httpGet:
    path: /healthz
    port: 8081
  initialDelaySeconds: 15
  periodSeconds: 20

readinessProbe:
  httpGet:
    path: /readyz
    port: 8081
  initialDelaySeconds: 5
  periodSeconds: 10