Anthropic Claude container crashing during large dataset evaluation

Problem – Claude Container CrashLoop During Large‑Dataset Evaluation

In a Kubernetes‑based evaluation pipeline the anthropic/claude container repeatedly enters CrashLoopBackOff while processing a benchmark dataset that exceeds 10 k prompts (≈5 GB JSON). The pod never reaches the Running state long enough to complete the evaluation workflow.

Typical symptom log excerpt:

kubectl logs claude-eval-7f9c9d8c5b-abcde -c claude
...
2024-07-14T10:22:31.412Z ERROR OutOfMemoryError: Java heap space
...
2024-07-14T10:22:35.001Z INFO Shutting down inference engine
...

Pod status (after three restarts):

kubectl get pod claude-eval-7f9c9d8c5b-abcde -o wide
NAME                                 READY   STATUS          RESTARTS   AGE
claude-eval-7f9c9d8c5b-abcde          0/1     CrashLoopBackOff   3      6m

Common error messages observed:

• “Container terminated with exit code 137” – kernel OOM kill.
• “Readiness probe failed: HTTP probe failed with statuscode: 500” – inference latency > probe timeout.
• “java.lang.OutOfMemoryError: Java heap space”.
• “torch.cuda.OutOfMemoryError: CUDA out of memory” on GPU nodes.

Root Cause Analysis

The crash loop is usually the result of one or more of the following interacting factors:

1. Unbounded input size. The Claude API enforces request size limits (see Claude API Reference). When the container receives a batch payload larger than the default --max-input-bytes, the internal tokenizer allocates memory proportional to the token count. Large JSON files (5 GB) exceed the default limit, causing the Java wrapper to allocate more heap than the container permits.
2. Insufficient Kubernetes memory requests/limits. The Anthropic container guide (Docker Container Guide) recommends at least 8 Gi of RAM for batch evaluation. Pods with resources.limits.memory set to 4 Gi quickly hit the cgroup limit, leading to OOM kill (exit code 137).
3. Readiness probe timeout. The default probe expects a 2‑second response. Large inputs cause inference latency >300 s (see real incident “Readiness probe failed”); the probe fails, Kubernetes restarts the container.
4. Concurrent request explosion. The flag --max-concurrent-requests defaults to 5. When the evaluation script streams thousands of prompts without throttling, the container spawns many inference threads, inflating both CPU and memory pressure.
5. GPU memory leaks. On GPU‑enabled nodes, missing torch.cuda.empty_cache() after each batch can accumulate GPU memory, eventually triggering torch.cuda.OutOfMemoryError (see GitHub issue OutOfMemoryError in Claude container on GPU nodes).

Investigation and Debugging Steps

1. Inspect pod events and OOM metrics

kubectl describe pod claude-eval-7f9c9d8c5b-abcde

Look for lines such as:

Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Warning  OOMKilled       2m    kubelet, node-1    Container claude was killed due to out of memory

2. Check container logs for Java or Python OOM traces

kubectl logs claude-eval-7f9c9d8c5b-abcde -c claude | grep -i outofmemory

3. Verify resource usage while a small batch runs

kubectl top pod claude-eval-7f9c9d8c5b-abcde

Compare memory usage against the resources.limits defined in the pod spec.

4. Capture the failing request size

# In the evaluation script, log the byte size of each batch payload
payload_bytes=$(jq -c . input_batch.json | wc -c)
echo "Batch payload size: ${payload_bytes} bytes"

5. Test readiness probe latency manually

curl -v http://:8080/healthz
# Measure response time; if >2s, increase probe timeout.

6. Reproduce OOM on a node with higher memory

kubectl run debug --image=busybox --restart=Never --rm -it -- \
  sh -c "apk add --no-cache curl && curl http:///evaluate -d @large_batch.json"

Solution – Stabilizing Claude Evaluation Pods

1. Adjust Kubernetes resource requests/limits

Before (insufficient memory):

apiVersion: v1
kind: Pod
metadata:
  name: claude-eval
spec:
  containers:
  - name: claude
    image: anthropic/claude:latest
    resources:
      requests:
        memory: "4Gi"
        cpu: "2"
      limits:
        memory: "4Gi"
        cpu: "4"

After (recommended for large batches):

apiVersion: v1
kind: Pod
metadata:
  name: claude-eval
spec:
  containers:
  - name: claude
    image: anthropic/claude:latest
    env:
    - name: JAVA_OPTS
      value: "-Xmx6g -Xms6g"
    resources:
      requests:
        memory: "12Gi"
        cpu: "4"
      limits:
        memory: "12Gi"
        cpu: "8"
    # Enable swap on nodes that allow it (see Anthropic container guide)
    securityContext:
      privileged: true

2. Enforce input size limits

Add the flag --max-input-bytes=10485760 (10 MiB) to the container entrypoint, matching the API recommendation (API Reference).

# Dockerfile snippet
ENTRYPOINT ["java", "-jar", "/app/claude.jar", "--max-input-bytes=10485760"]

3. Enable request throttling

Set --max-concurrent-requests=2 to limit parallel inference threads.

# In the Helm values or pod spec
args:
  - "--max-concurrent-requests=2"

4. Increase readiness/liveness probe timeouts

readinessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 30   # increased from 2
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 20
  timeoutSeconds: 30

5. Shard the dataset

Instead of a single 5 GB JSON file, split into ≤ 1 GB chunks using jq or a custom script, then invoke the container sequentially.

jq -c '.[]' large_dataset.json | split -l 10000 - dataset_part_
# Each part now contains 10k prompts, well within memory limits.

6. GPU memory cleanup (if using GPU nodes)

Insert torch.cuda.empty_cache() after each batch in the evaluation script.

import torch

def evaluate_batch(batch):
    # ... inference logic ...
    torch.cuda.empty_cache()

Verification – Confirming the Fix

1. Redeploy the pod with the updated manifest.
2. Run a controlled batch (e.g., 5 k prompts) and watch the pod stay Running for the full duration.
3. Check logs for absence of OOM traces:

kubectl logs claude-eval-xxxx -c claude | grep -i outofmemory
# No output expected

Validate readiness probe passes:

kubectl get pod claude-eval-xxxx -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}'
# Should output "True"

Measure memory consumption during the run:

kubectl top pod claude-eval-xxxx
# Memory should stay below the limit (e.g., 10‑11 Gi of 12 Gi limit)

Prevention – Operational Guardrails

• Resource monitoring. Set Prometheus alerts on container_memory_usage_bytes approaching 90 % of the limit.
• Dataset size validation. Add a CI step that rejects benchmark payloads exceeding --max-input-bytes.
• Probe tuning. Align readinessProbe.timeoutSeconds with the 95th‑percentile inference latency measured in production.
• Batch chunking policy. Enforce a maximum of 10 k prompts per batch (per Batch Evaluation Guide).
• Auto‑scaling. Configure a HorizontalPodAutoscaler that adds replicas when cpu_utilization or memory_utilization exceeds 70 %.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why does the container crash only when the dataset exceeds 10 k records?
   Because each record adds to the tokenization buffer. Beyond the default --max-input-bytes the Java heap grows past the pod’s memory limit, triggering an OOM kill.
2. How can I determine the exact byte size of a batch payload before sending it to Claude?
   Use jq -c . batch.json | wc -c or, in Python, len(json.dumps(batch).encode('utf-8')). Compare against the --max-input-bytes value.
3. Is increasing the Kubernetes memory limit sufficient?
   It prevents OOM kills but does not address the underlying API size limit or probe timeouts. Both the container flag and readiness probe must be tuned in tandem.
4. Can I keep the original large JSON file and avoid sharding?
   Only if you raise --max-input-bytes to a value that still fits within the pod’s heap and you increase the Java heap (-Xmx) accordingly. This is risky; sharding is the recommended, production‑grade approach.
5. What should I monitor to catch similar failures early?
   Set alerts on:
   • container_memory_rss > 85 % of limit
   • Readiness probe failure count > 0
   • Exit codes 137 or 1 in kube_pod_container_status_last_termination_reason