Kafka Container Crash Loop On Premises With Low CPU

Q: Problem

In an on‑premises data‑pipeline node that hosts a Kafka broker container, the pod repeatedly enters a CrashLoopBackOff state. The container exits after a few seconds, the orchestrator restarts it, and the cycle continues. The environment is a bare‑metal server with 4 vCPU and 8 GiB RAM, shared with other AI‑training services. The symptom manifests as:

Q: 1. Inspect pod events and container exit code

kubectl describe pod kafka-0 ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning BackOff 5m kubelet Back-off restarting failed container Normal Killing 5m kubelet Killing container with id docker://kafka... Normal Pulled 5m kubelet Container image "confluentinc/cp-kafka:7.2.0" already present on machine Normal Created 5m kubelet Created container kafka Normal Started 5m kubelet Started container kafka ... 2. Capture the broker logs just before termination kubectl logs kafka-0 -c kafka --tail=200 ... [2024-06-03 14:12:45,123] INFO [ReplicaFetcherThread-0-0], fetching data from broker 2 [2024-06-03 14:12:45,124] WARN [ReplicaFetcherThread-0-0], Error while fetching data from broker 2 (org.apache.kafka.common.errors.UnknownServerException) java.lang.OutOfMemoryError: Java heap space at java.base/java.util.Arrays.copyOf(Arrays.java:3512) at java.base/java.util.ArrayList.grow(ArrayList.java:237) ... [2024-06-03 14:12:45,130] INFO [KafkaServer id=0] Shutting down because of fatal error 3. Verify cgroup limits inside the container kubectl exec -it kafka-0 -c kafka -- bash # cat /sys/fs/cgroup/cpu.max 200000 200000 # cat /sys/fs/cgroup/memory.max 2147483648 The CPU quota is 200 ms per 200 ms (2 cores), and the memory limit is 2 GiB – exactly matching the JVM heap size, leaving no headroom for native buffers.

Q: 4. Check host‑level memory pressure

ssh root@kafka-node # free -m total used free shared buff/cache available Mem: 7986 6321 210 124 1454 1023 # vmstat 1 5 procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 2 0 0 21568 14848 932480 0 0 0 0 1 3 85 10 5 0 0 Available memory is < 1 GiB, confirming that the OS is under pressure.

Q: 5. Correlate CPU throttling metrics

# cat /sys/fs/cgroup/cpu.stat usage_usec 1234567890 user_usec 987654321 system_usec 23456789 throttled_usec 45678901 throttled_periods 1234 Throttled time > 4 % of total CPU time, indicating the JVM is being limited.

Q: 1. Reduce JVM heap to fit within the container limit

Adjust the environment variable KAFKA_HEAP_OPTS to a value that leaves ~20 % headroom for off‑heap buffers.

Q: 2. Lower log segment size and retention to reduce page‑cache pressure

# kafka-configs.sh --alter --entity-type brokers --entity-name 0 \ --add-config log.segment.bytes=536870912,log.retention.hours=72 Segment size reduced from 1 GiB to 512 MiB and retention shortened from 168 h to 72 h.

Problem

In an on‑premises data‑pipeline node that hosts a Kafka broker container, the pod repeatedly enters a CrashLoopBackOff state. The container exits after a few seconds, the orchestrator restarts it, and the cycle continues. The environment is a bare‑metal server with 4 vCPU and 8 GiB RAM, shared with other AI‑training services. The symptom manifests as:

Kafka logs ending with java.lang.OutOfMemoryError: Java heap space or java.lang.IllegalStateException: Failed to allocate memory for the page cache.
Kubernetes events: Back-off restarting failed container.
CPU usage spikes to 100 % for the java process just before termination.

Root Cause

The broker crashes because the JVM inside the container is unable to obtain the memory and CPU resources it expects. Two intertwined factors are typical in this scenario:

Inadequate container resource limits: The pod spec defines resources.requests.cpu: 1 and resources.limits.cpu: 2, while the Kafka broker’s default heap is 2 GiB (KAFKA_HEAP_OPTS="-Xmx2G -Xms2G"). On a 4‑core host, the cgroup CPU quota (2 cores) is insufficient for the broker’s internal I/O threads, replication threads, and the Java garbage collector, causing the process to be throttled, leading to long GC pauses and eventual OOM.
Page‑cache pressure from large log segments: The broker’s log.segment.bytes is set to 1 GiB and log.retention.hours to 168. With limited host memory, the OS page cache fills quickly. When the JVM tries to allocate direct buffers for network I/O, the kernel denies the request, and Kafka aborts with the “Failed to allocate memory for the page cache” exception.

Both conditions are amplified when the same node runs AI‑training workloads that consume the remaining CPU cycles and memory, leaving Kafka with an even smaller slice of resources than the pod limits indicate.

Debug

Below is a step‑by‑step investigation that reproduces the typical debugging workflow.

1. Inspect pod events and container exit code

kubectl describe pod kafka-0
...
Events:
  Type    Reason                 Age   From               Message
  ----    ------                 ----  ----               -------
  Warning BackOff                5m    kubelet            Back-off restarting failed container
  Normal  Killing                5m    kubelet            Killing container with id docker://kafka...
  Normal  Pulled                 5m    kubelet            Container image "confluentinc/cp-kafka:7.2.0" already present on machine
  Normal  Created                5m    kubelet            Created container kafka
  Normal  Started                5m    kubelet            Started container kafka
...

2. Capture the broker logs just before termination

kubectl logs kafka-0 -c kafka --tail=200
...
[2024-06-03 14:12:45,123] INFO  [ReplicaFetcherThread-0-0], fetching data from broker 2
[2024-06-03 14:12:45,124] WARN  [ReplicaFetcherThread-0-0], Error while fetching data from broker 2 (org.apache.kafka.common.errors.UnknownServerException)
java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3512)
    at java.base/java.util.ArrayList.grow(ArrayList.java:237)
    ...
[2024-06-03 14:12:45,130] INFO  [KafkaServer id=0] Shutting down because of fatal error

3. Verify cgroup limits inside the container

kubectl exec -it kafka-0 -c kafka -- bash
# cat /sys/fs/cgroup/cpu.max
200000 200000
# cat /sys/fs/cgroup/memory.max
2147483648

The CPU quota is 200 ms per 200 ms (2 cores), and the memory limit is 2 GiB – exactly matching the JVM heap size, leaving no headroom for native buffers.

4. Check host‑level memory pressure

ssh root@kafka-node
# free -m
              total        used        free      shared  buff/cache   available
Mem:           7986        6321         210         124        1454        1023
# vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 2  0      0  21568  14848  932480    0    0     0     0    1    3 85 10  5  0  0

Available memory is < 1 GiB, confirming that the OS is under pressure.

5. Correlate CPU throttling metrics

# cat /sys/fs/cgroup/cpu.stat
usage_usec 1234567890
user_usec  987654321
system_usec 23456789
throttled_usec 45678901
throttled_periods 1234

Throttled time > 4 % of total CPU time, indicating the JVM is being limited.

Solution

Address both the JVM sizing and the host resource allocation. The following changes resolved the crash loop in production.

1. Reduce JVM heap to fit within the container limit

Adjust the environment variable KAFKA_HEAP_OPTS to a value that leaves ~20 % headroom for off‑heap buffers.

Before (pod spec excerpt):

<env>
  <name>KAFKA_HEAP_OPTS</name>
  <value>-Xmx2G -Xms2G</value>
</env>

After (pod spec excerpt):

<env>
  <name>KAFKA_HEAP_OPTS</name>
  <value>-Xmx1G -Xms1G</value>
</env>

2. Lower log segment size and retention to reduce page‑cache pressure

# kafka-configs.sh --alter --entity-type brokers --entity-name 0 \
  --add-config log.segment.bytes=536870912,log.retention.hours=72

Segment size reduced from 1 GiB to 512 MiB and retention shortened from 168 h to 72 h.

3. Increase container CPU limit and request to match host capacity

Given the 4‑core host, allocate 3 cores to Kafka and reserve 1 core for the AI workloads.

Before:

resources:
  requests:
    cpu: "1"
    memory: "2Gi"
  limits:
    cpu: "2"
    memory: "2Gi"

After:

resources:
  requests:
    cpu: "2"
    memory: "2Gi"
  limits:
    cpu: "3"
    memory: "2Gi"

4. Enable swap on the host (optional, with caution)

If reducing the heap is not sufficient, enable a small swap file (e.g., 2 GiB) to provide a safety net for page‑cache spikes.

# fallocate -l 2G /swapfile
# chmod 600 /swapfile
# mkswap /swapfile
# swapon /swapfile

5. Restart the broker with the new configuration

kubectl rollout restart statefulset kafka

Verify

Confirm that the broker stays healthy and that resource metrics are within expected bounds.

1. Pod status

kubectl get pod kafka-0 -o wide
NAME      READY   STATUS    RESTARTS   AGE
kafka-0   1/1     Running   0          3m

2. Log inspection for absence of OOM

kubectl logs kafka-0 -c kafka | grep -i "OutOfMemoryError"
# (no output)

3. Metrics check (Prometheus query example)

kafka_server_brokertopicmetrics_bytesin_total{instance="kafka-node:9092"}
kube_pod_container_resource_requests_cpu_cores{pod="kafka-0"}
kube_pod_container_resource_limits_cpu_cores{pod="kafka-0"}

CPU usage should hover around 30‑50 % under normal load, and memory RSS < 1.5 GiB.

4. End‑to‑end data flow test

# Produce test messages
kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic <<EOF
msg1
msg2
msg3
EOF

# Consume
kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning --max-messages 3

All messages should be received without consumer errors.

Prevent

Implement guardrails to avoid recurrence:

Resource‑aware pod templates: Use a LimitRange that caps JVM heap to 0.5 × memory.limit and enforces a minimum CPU request of 2 cores for Kafka workloads.
Monitoring alerts:
- CPU throttling > 5 % for > 2 min → alert.
- JVM heap usage > 80 % → alert.
- Host free memory < 2 GiB → alert.
Log‑segment policy: Keep log.segment.bytes ≤ 512 MiB on low‑memory nodes and tune log.retention.ms based on downstream consumption rates.
Capacity planning: Reserve a dedicated node for Kafka when the AI training pipeline exceeds 70 % of CPU or memory.
Automated health checks: Add a readiness probe that checks /metrics for kafka_server_brokertopicmetrics_bytesin_total and fails if JVM heap usage exceeds a threshold.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does reducing the heap solve the OOM even though the host still has free RAM?
Because the container’s memory limit is enforced by cgroups; the JVM cannot allocate beyond that limit, regardless of host availability. Off‑heap buffers also consume the same limit, so a smaller heap leaves room for native memory.
Can I keep the original 2 GiB heap and just increase the container memory limit?
Yes, but the host only has 8 GiB shared with other services. Raising the limit to >3 GiB would starve the AI workloads and risk swapping, which degrades Kafka latency.
Is it safe to enable swap on a production Kafka node?
Swap can prevent hard OOM kills but introduces latency for page‑cached data. Use a small, low‑priority swap file and monitor swap usage; prefer scaling resources over relying on swap.
How do I know if the crash loop is caused by CPU throttling versus memory pressure?
Check /sys/fs/cgroup/cpu.stat for throttled_usec and throttled_periods. For memory, inspect /sys/fs/cgroup/memory.events for oom events and the broker logs for OutOfMemoryError or Failed to allocate memory for the page cache.
Do I need to adjust Zookeeper resources as well?
If Zookeeper runs on the same node, ensure its JVM heap is also sized relative to its container limit. However, the crash loop described is isolated to the Kafka broker; Zookeeper typically consumes far less CPU.