Weaviate Pod Eviction On Edge Node During CPU Pressure

Q: Problem – Weaviate pod eviction on edge node during CPU pressure

In a Kubernetes edge cluster, the Weaviate deployment (vector search engine) is repeatedly evicted when the node experiences high CPU load. The evictions cause query failures, increased latency, and temporary loss of data availability. Edge nodes have limited CPU and memory, and the workload spikes during real‑time ingestion and search bursts.

Q: Root Cause

The eviction originates from the kubelet’s eviction manager which enforces node.allocatable thresholds. When the node’s cpu usage exceeds the configured evictionHard limit (default cpu.available<100m), the kubelet selects pods with the lowest QoS class for termination. The Weaviate pod is running with:

Q: Debug – Investigation Steps 1. Observe eviction events

$ kubectl get events --field-selector reason=Evicted -n weaviate LAST SEEN TYPE REASON OBJECT MESSAGE 2m Warning Evicted pod/weaviate-0-abc123 The node was low on resource: cpu. 2. Inspect node resource pressure $ kubectl top node edge-node-01 NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% edge-node-01 1850m 92% 6.1Gi 78% 3. Review kubelet eviction configuration $ cat /var/lib/kubelet/config.yaml | grep eviction evictionHard: cpu.available: "100m" memory.available: "200Mi" 4. Examine Weaviate pod spec apiVersion: apps/v1 kind: StatefulSet metadata: name: weaviate spec: serviceName: weaviate replicas: 1 template: metadata: labels: app: weaviate spec: containers: - name: weaviate image: semitechnologies/weaviate:1.22.0 resources: requests: cpu: "250m" memory: "1Gi" limits: cpu: "500m" memory: "2Gi" 5. Verify QoS classification $ kubectl get pod weaviate-0-abc123 -o jsonpath='{.status.qosClass}' Burstable 6. Correlate CPU usage with eviction timing $ kubectl logs weaviate-0-abc123 -c weaviate --since=5m | grep "CPU usage" 2024-10-28T12:03:12Z INFO CPU usage: 420m (84% of request) 2024-10-28T12:03:45Z INFO CPU usage: 520m (104% of request) Solution – Preventing Evictions while Preserving Latency and Consistency 1. Align resource requests with observed peak usage Increase the CPU request to a value that covers the 95th‑percentile usage observed during spikes (≈550 m). This upgrades the pod to Guaranteed QoS when limits match requests.

Q: 2. Define a high‑priority class for Weaviate

Create a PriorityClass with a value higher than the default system pods.

Q: 3. Adjust kubelet eviction thresholds for edge nodes

Relax the cpu.available threshold to give a larger buffer, e.g., 200m, and enable systemReserved to reserve CPU for system daemons.

Q: 4. Enable the Vertical Pod Autoscaler (VPA) on the Weaviate StatefulSet

The VPA will automatically adjust requests/limits based on observed usage without manual redeploys.

Q: 5. Deploy a dedicated edge node pool for vector workloads

Label a node with node-role.kubernetes.io/vector=true and use a nodeSelector or affinity rule so that Weaviate runs on a node sized for its CPU demands.

Q: Verify – Confirming the Fix 1. Observe pod QoS and priority

$ kubectl get pod weaviate-0-abc123 -o jsonpath='{.status.qosClass} {.spec.priorityClassName}' Guaranteed weaviate-high-priority 2. Simulate CPU pressure # Run a CPU burn on the edge node for 2 minutes $ kubectl run stress-cpu --image=alpine --restart=Never -- \ sh -c "apk add stress && stress --cpu 4 --timeout 120" Monitor that the Weaviate pod remains Running and no eviction events appear.

Q: 3. Check latency after the change

$ curl -s http://weaviate.edge.local/v1/.well-known/health | jq . { "status": "ok", "latency_ms": 12 } Latency stays within the SLA (< 30 ms) even under load.

Q: 4. Verify VPA recommendations

$ kubectl get vpa weaviate-vpa -o yaml ... status: recommendation: containerRecommendations: - containerName: weaviate target: cpu: "600m" memory: "2Gi" Prevent – Operational Guardrails Resource reservation: Always set CPU requests to the 95th‑percentile of observed usage; keep limits equal to requests for critical services to achieve Guaranteed QoS. Priority classes: Assign a custom priority to any workload that must survive node pressure. Node sizing: Use dedicated node pools for CPU‑intensive vector workloads; avoid co‑locating with unrelated batch jobs. Monitoring & alerts: Create alerts on node_cpu_utilization > 80% and on kube_pod_status_phase{phase="Failed"} for the Weaviate namespace. VPA & HPA: Combine Vertical Pod Autoscaler (for requests) with Horizontal Pod Autoscaler (for replica scaling) to handle both bursty ingestion and query traffic. Eviction threshold review: Periodically audit evictionHard values on edge nodes; adjust based on workload growth. Related Topic Hub: Vector Databases Troubleshooting Hub

In this article

Table of Contents

Problem – Weaviate pod eviction on edge node during CPU pressure

In a Kubernetes edge cluster, the Weaviate deployment (vector search engine) is repeatedly evicted when the node experiences high CPU load. The evictions cause query failures, increased latency, and temporary loss of data availability. Edge nodes have limited CPU and memory, and the workload spikes during real‑time ingestion and search bursts.

Root Cause

The eviction originates from the kubelet’s eviction manager which enforces node.allocatable thresholds. When the node’s cpu usage exceeds the configured evictionHard limit (default cpu.available<100m), the kubelet selects pods with the lowest QoS class for termination. The Weaviate pod is running with:

CPU requests of 250m and limits of 500m.
No priorityClassName defined, so it receives the default Priority=0.
Pod QoS class = Burstable (requests < limits).

During ingestion bursts, the node’s total CPU demand exceeds the evictionHard threshold, causing the kubelet to evict the Burstable Weaviate pod before the Guaranteed system pods. The underlying reason is an insufficient CPU request relative to the actual usage pattern, combined with missing pod priority and aggressive eviction thresholds on the edge node.

Debug – Investigation Steps

1. Observe eviction events


$ kubectl get events --field-selector reason=Evicted -n weaviate
LAST SEEN   TYPE      REASON   OBJECT                     MESSAGE
2m          Warning   Evicted  pod/weaviate-0-abc123      The node was low on resource: cpu.

2. Inspect node resource pressure


$ kubectl top node edge-node-01
NAME            CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
edge-node-01    1850m        92%    6.1Gi           78%

3. Review kubelet eviction configuration


$ cat /var/lib/kubelet/config.yaml | grep eviction
evictionHard:
  cpu.available: "100m"
  memory.available: "200Mi"

4. Examine Weaviate pod spec


apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: weaviate
spec:
  serviceName: weaviate
  replicas: 1
  template:
    metadata:
      labels:
        app: weaviate
    spec:
      containers:
      - name: weaviate
        image: semitechnologies/weaviate:1.22.0
        resources:
          requests:
            cpu: "250m"
            memory: "1Gi"
          limits:
            cpu: "500m"
            memory: "2Gi"

5. Verify QoS classification


$ kubectl get pod weaviate-0-abc123 -o jsonpath='{.status.qosClass}'
Burstable

6. Correlate CPU usage with eviction timing


$ kubectl logs weaviate-0-abc123 -c weaviate --since=5m | grep "CPU usage"
2024-10-28T12:03:12Z INFO CPU usage: 420m (84% of request)
2024-10-28T12:03:45Z INFO CPU usage: 520m (104% of request)

Solution – Preventing Evictions while Preserving Latency and Consistency

1. Align resource requests with observed peak usage

Increase the CPU request to a value that covers the 95th‑percentile usage observed during spikes (≈550 m). This upgrades the pod to Guaranteed QoS when limits match requests.

Before


resources:
  requests:
    cpu: "250m"
    memory: "1Gi"
  limits:
    cpu: "500m"
    memory: "2Gi"

After


resources:
  requests:
    cpu: "550m"
    memory: "1Gi"
  limits:
    cpu: "550m"
    memory: "2Gi"

2. Define a high‑priority class for Weaviate

Create a PriorityClass with a value higher than the default system pods.


apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: weaviate-high-priority
value: 1000000
globalDefault: false
description: "Ensures Weaviate pods are protected from eviction under node pressure."

Attach it to the pod spec:


spec:
  priorityClassName: weaviate-high-priority

3. Adjust kubelet eviction thresholds for edge nodes

Relax the cpu.available threshold to give a larger buffer, e.g., 200m, and enable systemReserved to reserve CPU for system daemons.


evictionHard:
  cpu.available: "200m"
  memory.available: "200Mi"
systemReserved:
  cpu: "200m"
  memory: "500Mi"

4. Enable the Vertical Pod Autoscaler (VPA) on the Weaviate StatefulSet

The VPA will automatically adjust requests/limits based on observed usage without manual redeploys.


apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: weaviate-vpa
spec:
  targetRef:
    apiVersion: "apps/v1"
    kind:       StatefulSet
    name:       weaviate
  updatePolicy:
    updateMode: "Auto"
  resourcePolicy:
    containerPolicies:
    - containerName: weaviate
      controlledResources: ["cpu", "memory"]

5. Deploy a dedicated edge node pool for vector workloads

Label a node with node-role.kubernetes.io/vector=true and use a nodeSelector or affinity rule so that Weaviate runs on a node sized for its CPU demands.


spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: node-role.kubernetes.io/vector
            operator: In
            values: ["true"]

Verify – Confirming the Fix

1. Observe pod QoS and priority


$ kubectl get pod weaviate-0-abc123 -o jsonpath='{.status.qosClass} {.spec.priorityClassName}'
Guaranteed weaviate-high-priority

2. Simulate CPU pressure


# Run a CPU burn on the edge node for 2 minutes
$ kubectl run stress-cpu --image=alpine --restart=Never -- \
  sh -c "apk add stress && stress --cpu 4 --timeout 120"

Monitor that the Weaviate pod remains Running and no eviction events appear.

3. Check latency after the change


$ curl -s http://weaviate.edge.local/v1/.well-known/health | jq .
{
  "status": "ok",
  "latency_ms": 12
}

Latency stays within the SLA (< 30 ms) even under load.

4. Verify VPA recommendations


$ kubectl get vpa weaviate-vpa -o yaml
...
status:
  recommendation:
    containerRecommendations:
    - containerName: weaviate
      target:
        cpu: "600m"
        memory: "2Gi"

Prevent – Operational Guardrails

Resource reservation: Always set CPU requests to the 95th‑percentile of observed usage; keep limits equal to requests for critical services to achieve Guaranteed QoS.
Priority classes: Assign a custom priority to any workload that must survive node pressure.
Node sizing: Use dedicated node pools for CPU‑intensive vector workloads; avoid co‑locating with unrelated batch jobs.
Monitoring & alerts: Create alerts on node_cpu_utilization > 80% and on kube_pod_status_phase{phase="Failed"} for the Weaviate namespace.
VPA & HPA: Combine Vertical Pod Autoscaler (for requests) with Horizontal Pod Autoscaler (for replica scaling) to handle both bursty ingestion and query traffic.
Eviction threshold review: Periodically audit evictionHard values on edge nodes; adjust based on workload growth.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does increasing the CPU limit alone not stop evictions? Eviction decisions are based on requests, not limits. The kubelet guarantees only the requested amount; if the request is too low, the pod is classified as Burstable and becomes a candidate for eviction.
Can I rely on HPA alone to prevent evictions? HPA scales the number of replicas but does not adjust per‑pod resource requests. Under CPU pressure a single pod may still be evicted before a new replica is scheduled.
What is the impact of setting cpu.available to a higher value? It gives the node a larger safety margin before the eviction manager triggers, reducing the frequency of evictions at the cost of potentially higher overall node utilization.
Do I need to restart the Weaviate StatefulSet after changing the priority class? Yes. Updating priorityClassName is not a mutable field; a rolling restart (or kubectl rollout restart) is required for the new priority to take effect.
How does VPA interact with existing resource limits? VPA can operate in Auto mode to update both requests and limits to match recommendations. If limits are set lower than the recommendation, VPA will patch the pod spec on the next rollout.