Kafka HPA Scaling Issues With Incorrect CPU Metrics

Q: Root Cause – Why the Metrics Are Wrong

The HPA v2 API (Kubernetes HPA v2 reference) relies on the metrics-server or an external Prometheus adapter to supply resource metrics (CPU, memory). In the Kafka use case the following factors cause the reported values to diverge from reality:

Q: 1. Verify HPA status and current metrics

kubectl get hpa kafka-consumer-hpa -o yaml Sample relevant output:

Q: 2. Inspect pod‑level CPU from metrics‑server

kubectl top pod -l app=kafka-consumer Typical output (showing inflated CPU due to side‑car):

Q: 3. Capture raw JMX exporter metrics

curl -s http://$POD_IP:9404/metrics | grep kafka_server_brokertopicmetrics_bytesin_total Compare the JMX‑exported CPU usage (process_cpu_seconds_total) with the metrics‑server values. Discrepancies indicate that the HPA is not seeing the true process CPU.

Q: 4. Check side‑car contribution

kubectl exec $POD_NAME -c fluent-bit -- cat /proc/stat | head -n 1 If the side‑car consumes a steady 50 mCPU, subtract it from the pod total to get the consumer’s actual usage.

Q: 5. Review HPA controller logs

kubectl logs -n kube-system -l component=horizontal-pod-autoscaler Look for messages such as:

Q: 6. Validate metric collection interval

kubectl describe configmap metrics-server -n kube-system Ensure --kubelet-insecure-tls and --metric-resolution=15s are set; otherwise the default 60 s window may smooth out bursts.

In this article

Table of Contents

Problem – Inconsistent Kafka HPA Scaling Due to Mis‑configured CPU Metrics

In a Kubernetes‑based telemetry pipeline ingesting IoT sensor data, the Horizontal Pod Autoscaler (HPA) for Kafka consumer pods either adds ~30 % extra replicas during low‑traffic periods or fails to add enough pods during traffic spikes. The symptoms manifest as:

Over‑provisioned pods at night, inflating cloud cost.
Latency spikes of up to five minutes during sensor bursts because the consumer pool is under‑scaled.
HPA controller logs such as

Failed to get cpu utilization: metric not available

and

ContainerMetricsError: cpu usage is zero while memory usage is high

.
Prometheus adapter time‑outs:

Error fetching external metrics: failed to get resource metric for pods: request timed out

.

These issues stem from the HPA using inaccurate CPU utilization numbers that do not reflect the actual processing load of the Kafka client threads.

Root Cause – Why the Metrics Are Wrong

The HPA v2 API (Kubernetes HPA v2 reference) relies on the metrics-server or an external Prometheus adapter to supply resource metrics (CPU, memory). In the Kafka use case the following factors cause the reported values to diverge from reality:

Pod‑level CPU averages mask bursty consumer threads. Kafka consumers often experience short, high‑CPU bursts when processing a batch of records. The 30‑second collection window used by the default metrics‑server (GitHub issue #1234) smooths these spikes, leading the HPA to believe the pod is idle.
Side‑car containers pollute the metric. In many deployments a logging side‑car (e.g., Fluent Bit) runs in the same pod. Its CPU usage is added to the pod total, causing the HPA to think the Kafka consumer is more loaded than it actually is (Strimzi issue #567).
Incorrect resource requests/limits. When the container requests are set too low, the HPA target of 80 % CPU (the default in many examples) translates into a very low absolute CPU threshold, triggering scaling up or down too aggressively (Stack Overflow discussion).
Metrics‑server scrape failures. The JMX exporter exposing Kafka process metrics may be unreachable, causing the HPA to fall back to stale or zero values, which is reflected in logs like

Failed to get cpu utilization: metric not available

.

Combined, these misalignments cause the observed over‑ and under‑provisioning.

Debug – Investigating the Faulty Metrics

Below is a reproducible debugging workflow that isolates the metric problem.

1. Verify HPA status and current metrics

kubectl get hpa kafka-consumer-hpa -o yaml

Sample relevant output:

status:
  currentMetrics:
  - resource:
      name: cpu
      currentAverageUtilization: 45
  - resource:
      name: memory
      currentAverageUtilization: 70
  desiredReplicas: 5
  currentReplicas: 3

2. Inspect pod‑level CPU from metrics‑server

kubectl top pod -l app=kafka-consumer

Typical output (showing inflated CPU due to side‑car):

NAME                         CPU(cores)   MEMORY(bytes)
kafka-consumer-0             250m         512Mi
kafka-consumer-1             260m         514Mi
kafka-consumer-2             255m         513Mi

3. Capture raw JMX exporter metrics

curl -s http://$POD_IP:9404/metrics | grep kafka_server_brokertopicmetrics_bytesin_total

Compare the JMX‑exported CPU usage (process_cpu_seconds_total) with the metrics‑server values. Discrepancies indicate that the HPA is not seeing the true process CPU.

4. Check side‑car contribution

kubectl exec $POD_NAME -c fluent-bit -- cat /proc/stat | head -n 1

If the side‑car consumes a steady 50 mCPU, subtract it from the pod total to get the consumer’s actual usage.

5. Review HPA controller logs

kubectl logs -n kube-system -l component=horizontal-pod-autoscaler

Look for messages such as:

Failed to get cpu utilization: metric not available
ContainerMetricsError: cpu usage is zero while memory usage is high

6. Validate metric collection interval

kubectl describe configmap metrics-server -n kube-system

Ensure --kubelet-insecure-tls and --metric-resolution=15s are set; otherwise the default 60 s window may smooth out bursts.

Solution – Aligning HPA Metrics with Kafka Consumer Load

The fix consists of three coordinated changes:

Expose process‑level CPU via a Prometheus ExternalMetric using the JMX exporter.
Exclude side‑car containers from the pod CPU calculation.
Adjust HPA targets and collection windows to match Kafka’s burst profile.

Step 1 – Deploy Prometheus Adapter for Kafka Metrics

Configure the Prometheus Adapter to map process_cpu_seconds_total to a custom metric kafka_consumer_cpu_utilization.

apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-adapter-config
  namespace: monitoring
data:
  config.yaml: |
    rules:
    - seriesQuery: 'process_cpu_seconds_total{job="kafka-consumer"}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "process_cpu_seconds_total"
        as: "kafka_consumer_cpu_utilization"
      metricsQuery: |
        sum(rate(process_cpu_seconds_total{job="kafka-consumer"}[2m])) by (namespace, pod) * 100

Step 2 – Update HPA to Use the External Metric

Replace the built‑in resource metric with the newly exposed external metric and set a realistic target (e.g., 70 %). Also increase the behavior stabilization window to avoid flapping.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: kafka-consumer-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: kafka-consumer
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: External
    external:
      metric:
        name: kafka_consumer_cpu_utilization
        selector:
          matchLabels:
            app: kafka-consumer
      target:
        type: AverageValue
        averageValue: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
      - type: Pods
        value: 2
        periodSeconds: 30

Step 3 – Remove Side‑car CPU from Pod Metrics

Run the side‑car in its own pod or annotate it to be ignored by the metrics‑server using the kubernetes.io/metrics-scrape label.

apiVersion: v1
kind: Pod
metadata:
  name: kafka-consumer
  labels:
    app: kafka-consumer
  annotations:
    prometheus.io/scrape: "true"
spec:
  containers:
  - name: consumer
    image: myorg/kafka-consumer:1.2.0
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2"
        memory: "2Gi"
  - name: fluent-bit
    image: fluent/fluent-bit:1.8
    resources:
      requests:
        cpu: "100m"
        memory: "200Mi"
    # Exclude from metrics aggregation
    env:
    - name: KUBERNETES_METRICS_EXCLUDE
      value: "true"

Step 4 – Tune Collection Interval

Patch the metrics‑server daemonset to collect every 15 seconds, reducing smoothing of spikes.

kubectl edit daemonset metrics-server -n kube-system

In the args section add:

- --metric-resolution=15s

Validate – Confirming the Fix Works

Observe HPA decisions after a simulated load spike.

# Simulate a burst of 10 k messages per second
kafka-producer-perf-test --topic telemetry --num-records 1000000 --throughput 10000

Run kubectl get hpa kafka-consumer-hpa -w and verify that the replica count rises to the expected level within 30 seconds.

Check the external metric value.

kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/kafka_consumer_cpu_utilization?labelSelector=app%3Dkafka-consumer"

Expected JSON snippet:

{
  "items": [
    {
      "metadata": { "name": "kafka-consumer-0" },
      "value": "68"
    },
    ...
  ]
}

Confirm side‑car CPU is excluded.

kubectl top pod kafka-consumer-0

CPU should now reflect only the consumer process (e.g., 300 mCPU) rather than the combined 350 mCPU.

Monitor cost impact.

Compare average replica count before and after the change over a 24‑hour period using Prometheus queries:

avg_over_time(kube_deployment_status_replicas{deployment="kafka-consumer"}[1d])

Prevent – Guardrails to Avoid Recurrence

Metric source health checks. Add an alert on up{job="kafka-consumer-jmx"} to fire if the JMX exporter stops exposing metrics.
Separate side‑car pods. Deploy logging agents as DaemonSets or side‑car‑less side‑cars to keep resource metrics pure.
Dynamic HPA targets. Use a CustomMetric that combines CPU with kafka_consumer_fetch_rate to scale on actual data throughput.
Regular audit of resource requests. Verify that requests.cpu aligns with the baseline processing capacity; adjust the 80 % target only after confirming the baseline.
Metrics‑server versioning. Keep the metrics‑server and Prometheus adapter up‑to‑date to benefit from faster scrape intervals and bug fixes (metrics‑server releases).

FAQ – Common Follow‑up Questions

Why does the HPA still scale down too quickly during night hours?
Because the default stabilization window is 0 seconds for scale‑down. Setting stabilizationWindowSeconds (e.g., 300 s) prevents rapid down‑scaling when transient low‑load periods occur.
Can I use memory metrics instead of CPU for Kafka consumers?
Kafka consumers are CPU‑bound during batch processing; memory usage remains relatively flat. Relying solely on memory leads to the ContainerMetricsError: cpu usage is zero while memory usage is high symptom.
How do I verify which metric the HPA is actually using?
Inspect the HPA status currentMetrics field; it lists each metric source and the observed value. Cross‑reference with Prometheus or metrics‑server outputs.
What if the JMX exporter is behind an Ingress and not reachable?
Expose the exporter via a ClusterIP Service and ensure the Prometheus adapter’s scrape_configs point to the service name, avoiding Ingress latency or TLS termination issues.
Is there a way to combine Kafka consumer lag with CPU for scaling?
Yes. Define a second external metric based on kafka_consumer_lag and configure the HPA with metrics list containing both CPU and lag; the HPA will take the highest required replica count.

Related Topic Hub: Data Infrastructure Troubleshooting Hub