Elasticsearch Pod Placement Failure During High Concurrency Testing

Q: Problem: Elasticsearch pod placement failures under high‑concurrency load testing

During automated benchmark runs that combine Elasticsearch Rally suites with custom ML vector‑search latency tests, the Kubernetes scheduler repeatedly emits FailedScheduling events and refuses to bind new Elasticsearch pods. The observable symptoms include:

Q: 3. PodDisruptionBudget violations triggered by rapid HPA scaling

The Elasticsearch data tier is protected by a PDB with maxUnavailable: 1. During a simulated load spike, the Horizontal Pod Autoscaler (HPA) scales down two data nodes simultaneously. The PDB blocks the eviction of the second pod, causing the scheduler to reject any new pod creation until the PDB condition clears (SIG Scheduling discussion #2021).

Q: 4. Interaction with strict ResourceQuota objects

Namespace‑level ResourceQuota objects limit total CPU and memory usage. When the benchmark spawns additional vector‑search pods, the quota is exhausted, and the scheduler reports “Insufficient memory” for pending Elasticsearch pods (Stack Overflow 78543201).

Q: Investigation and Debugging Steps Step 1 – Inspect pod events and scheduler logs

kubectl get pods -n benchmark -o wide kubectl describe pod elasticsearch-data-0 -n benchmark Typical describe output excerpt:

Q: Step 2 – Verify node allocatable resources

kubectl top nodes kubectl get nodes -o jsonpath="{.items[*].status.allocatable}" Example output showing saturation:

Q: Step 3 – Check GPU node affinity and taints

kubectl get nodes -l nvidia.com/gpu.present=true -o yaml | grep -i taint kubectl describe pod vector-search-0 -n benchmark Output snippet:

Q: Step 4 – Examine the PodDisruptionBudget status

kubectl get pdb elasticsearch-data-pdb -n benchmark -o yaml Relevant fields:

Q: Step 5 – Review ResourceQuota usage

kubectl get quota -n benchmark -o yaml Sample quota status:

In this article

Table of Contents

Problem: Elasticsearch pod placement failures under high‑concurrency load testing

During automated benchmark runs that combine Elasticsearch Rally suites with custom ML vector‑search latency tests, the Kubernetes scheduler repeatedly emits FailedScheduling events and refuses to bind new Elasticsearch pods. The observable symptoms include:

Pod events such as "FailedScheduling: 0/5 nodes are available: 5 Insufficient cpu" and "FailedScheduling: 0/5 nodes are available: 5 Insufficient memory".
Master‑node pods stuck in Pending with "node(s) had untolerated taints" errors.
PodDisruptionBudget (PDB) violations: "pod disruption budget is violated" when HPA scales down data nodes.
Benchmark timeouts and degraded query latency because shard replicas are reduced.

These failures occur only during the high‑concurrency phases of the test, not during normal operation.

Root Cause Analysis

1. Resource exhaustion on the node pool

The Kubernetes Scheduler documentation defines that a pod cannot be bound if its requests exceed the node’s allocatable CPU or memory. Real‑world incidents (GitHub elastic/cloud-on-k8s#1587) show that a nightly Rally benchmark pushed the cluster to 95 % CPU utilization, causing the scheduler to emit the “Insufficient cpu” error for new data nodes.

2. Conflicting node affinity for GPU‑accelerated vector search

GPU‑enabled inference pods are launched with a node affinity of nvidia.com/gpu (NVIDIA GPU Operator docs). The same nodes also host Elasticsearch master pods that use a generic nodeSelector or no selector at all. When the GPU pods occupy all GPU‑capable nodes, the master pods encounter "node(s) had untolerated taints" because the nodes acquire the nvidia.com/gpu taint without the master pods having a matching toleration (GitHub #1723).

3. PodDisruptionBudget violations triggered by rapid HPA scaling

The Elasticsearch data tier is protected by a PDB with maxUnavailable: 1. During a simulated load spike, the Horizontal Pod Autoscaler (HPA) scales down two data nodes simultaneously. The PDB blocks the eviction of the second pod, causing the scheduler to reject any new pod creation until the PDB condition clears (SIG Scheduling discussion #2021).

4. Interaction with strict ResourceQuota objects

Namespace‑level ResourceQuota objects limit total CPU and memory usage. When the benchmark spawns additional vector‑search pods, the quota is exhausted, and the scheduler reports “Insufficient memory” for pending Elasticsearch pods (Stack Overflow 78543201).

Investigation and Debugging Steps

Step 1 – Inspect pod events and scheduler logs


kubectl get pods -n benchmark -o wide
kubectl describe pod elasticsearch-data-0 -n benchmark

Typical describe output excerpt:


Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  2m    default-scheduler  0/5 nodes are available: 5 Insufficient cpu
  Warning  FailedScheduling  2m    default-scheduler  0/5 nodes are available: 5 Insufficient memory

Step 2 – Verify node allocatable resources


kubectl top nodes
kubectl get nodes -o jsonpath="{.items[*].status.allocatable}"

Example output showing saturation:


NAME          CPU(cores)   MEMORY(bytes)
node-1        2 (100%)     8Gi (96%)
node-2        2 (98%)      8Gi (94%)
...

Step 3 – Check GPU node affinity and taints


kubectl get nodes -l nvidia.com/gpu.present=true -o yaml | grep -i taint
kubectl describe pod vector-search-0 -n benchmark

Output snippet:


Taints:             nvidia.com/gpu=present:NoSchedule
...
Affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: nvidia.com/gpu
          operator: Exists

Step 4 – Examine the PodDisruptionBudget status


kubectl get pdb elasticsearch-data-pdb -n benchmark -o yaml

Relevant fields:


status:
  currentHealthy: 2
  desiredHealthy: 3
  disruptionsAllowed: 0
  expectedPods: 3

Step 5 – Review ResourceQuota usage


kubectl get quota -n benchmark -o yaml

Sample quota status:


status:
  hard:
    cpu: "20"
    memory: 64Gi
  used:
    cpu: "19"
    memory: 60Gi

Resolution

1. Adjust Elasticsearch resource requests and limits

Follow the ECK deployment guide to align CPU/memory requests with the JVM heap size. Reduce the default request from 2CPU/4Gi to 1CPU/2Gi for data nodes during benchmark runs.

# Before (benchmark‑values.yaml)
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  nodeSets:
  - name: data
    count: 3
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"
            limits:
              cpu: "4"
              memory: "8Gi"

# After (benchmark‑values.yaml)
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  nodeSets:
  - name: data
    count: 3
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          resources:
            requests:
              cpu: "1"
              memory: "2Gi"
            limits:
              cpu: "2"
              memory: "4Gi"

2. Separate GPU workloads onto dedicated node pool

Create a label gpu-workload=true on GPU‑only nodes and update the Elasticsearch master node selector to avoid those nodes.

# Label GPU nodes
kubectl label nodes node-1 gpu-workload=true
kubectl label nodes node-2 gpu-workload=true

# Update Elasticsearch master nodeSet
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch
spec:
  nodeSets:
  - name: master
    count: 3
    podTemplate:
      spec:
        nodeSelector:
          gpu-workload: "false"
        tolerations:
        - key: "nvidia.com/gpu"
          operator: "Exists"
          effect: "NoSchedule"
          tolerationSeconds: 300

3. Relax the PodDisruptionBudget during benchmark windows

Use a separate PDB manifest applied only for the benchmark run, setting maxUnavailable: 2 to allow concurrent scaling.

# pdb-benchmark.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: elasticsearch-data-pdb-bench
  namespace: benchmark
spec:
  maxUnavailable: 2
  selector:
    matchLabels:
      elasticsearch.k8s.elastic.co/cluster-name: elasticsearch
      elasticsearch.k8s.elastic.co/node-set: data

4. Increase or split ResourceQuota limits

Either raise the quota for the benchmark namespace or create a separate namespace for GPU inference pods with its own quota.

# Updated quota
apiVersion: v1
kind: ResourceQuota
metadata:
  name: benchmark-quota
  namespace: benchmark
spec:
  hard:
    cpu: "30"
    memory: "96Gi"

Verification

Scheduler health: Ensure no FailedScheduling events for Elasticsearch pods.


kubectl get events -n benchmark --field-selector reason=FailedScheduling

Pod status: All Elasticsearch pods should reach Running and report Ready conditions.


kubectl get pods -l elasticsearch.k8s.elastic.co/cluster-name=elasticsearch -n benchmark

Resource utilization: Verify node CPU/memory headroom remains above 20 % during the load spike.
```
kubectl top nodes
```

PDB compliance: Confirm disruptionsAllowed is non‑zero.


kubectl get pdb elasticsearch-data-pdb-bench -n benchmark -o yaml | grep disruptionsAllowed

GPU pod placement: Check that GPU inference pods land only on nodes with gpu-workload=true.
```
kubectl get pods -l app=vector-search -o wide
```

Prevention and Best Practices

Capacity planning: Use the Elasticsearch resource planning guide to size JVM heap, CPU, and memory before benchmark runs.
Separate node pools: Deploy GPU‑accelerated workloads on a dedicated node pool with its own taints and tolerations to avoid affinity conflicts.
Dynamic PDBs: Automate PDB adjustments via a CI/CD step that applies a relaxed PDB during performance tests and restores the production PDB afterward.
Quota monitoring: Set up alerts on ResourceQuota usage (e.g., when >80 % of CPU or memory is consumed) to catch quota exhaustion before it blocks scheduling.
HPA cooldown: Configure horizontal-pod-autoscaler` with appropriate stabilizationWindowSeconds to avoid rapid simultaneous scale‑down events that trigger PDB violations.
Scheduler profiling: Enable --profiling on the scheduler or use kubectl describe node to watch allocatable vs capacity trends during load tests.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the scheduler report “Insufficient cpu” only during the Rally benchmark?
Because the benchmark creates many concurrent search and indexing threads that push node CPU utilization past the requests of pending Elasticsearch pods. The scheduler checks allocatable - requested at scheduling time, and when the margin is negative it rejects the pod.
Can I keep the same GPU node affinity for both inference pods and Elasticsearch masters?
Only if you add a toleration for the nvidia.com/gpu taint on the master pods and ensure sufficient GPU capacity. A safer pattern is to isolate GPU workloads on a separate node pool to avoid taint conflicts.
How do I know if a PodDisruptionBudget is the bottleneck?
Run kubectl get pdb -n -o wide and look at disruptionsAllowed. If it is zero while HPA is trying to evict pods, the PDB is blocking scheduling. Adjust maxUnavailable or temporarily disable the PDB for the test window.
What metric should I alert on to prevent “Insufficient memory” errors?
Create a Prometheus alert on kube_node_status_allocatable_memory_bytes - sum(kube_pod_container_resource_requests_memory_bytes) < 0.2 * kube_node_status_allocatable_memory_bytes. This triggers when free allocatable memory falls below 20 % of the node’s capacity.
Do I need to change the Elasticsearch JVM heap when I lower the pod’s memory request?
Yes. The JVM heap must be ≤ 50 % of the pod’s memory request (per the Elastic Cloud on Kubernetes guide). Update the esJavaOpts or heapSize setting accordingly to avoid OutOfMemory errors after reducing the request.