Mistral Inference Pod Evictions During Multi-region Replication Sync

Q: Problem – Mistral Inference Pods Evicted During Multi‑Region Replication Sync

The Mistral AI inference service runs as a set of Deployment pods behind a ClusterIP service. When a multi‑region replication job (running as a CronJob that streams model checkpoints from the primary region to secondary regions) starts, operators observe a spike in pod evictions:

Q: Root Cause – Resource Contention Between Inference and Replication Workloads

The replication sync process launches a set of Job pods that perform the following steps for each model version:

Q: Debug – Investigation and Diagnostics 1. Identify eviction events

$ kubectl get events --field-selector reason=Evicted -A --sort-by=.metadata.creationTimestamp NAMESPACE KIND NAME REASON AGE FROM MESSAGE ml-prod Pod mistral-infer-6f7c9b Evicted 2m kubelet, node-az1 Pod evicted: node(s) had insufficient memory (requested: 2Gi, limit: 4Gi) ml-prod Pod mistral-infer-8a2d3e Evicted 2m kubelet, node-az2 Pod evicted: node(s) had insufficient cpu (requested: 1500m, limit: 2000m) 2. Correlate with replication job start time $ kubectl get jobs -n ml-replication -o wide NAME COMPLETIONS DURATION AGE SELECTOR replicate-model-v1 0/15 0s 2m job-name=replicate-model-v1 The timestamps line up with the eviction events, confirming contention.

Q: 3. Inspect node resource pressure

$ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% node-az1-01 2100m 105% 15Gi / 16Gi 94% node-az2-03 1800m 90% 14Gi / 16Gi 88% 4. Review pod priority and preemption policy $ kubectl get pod mistral-infer-6f7c9b -o yaml | grep priorityClassName priorityClassName: inference-high The replication jobs use the default priority class system-cluster-critical, which preempts inference pods.

Q: 5. Check PVC binding and I/O saturation

$ iostat -x 1 3 Device r/s w/s rkB/s wkB/s await svctm %util fs-12345678 0.0 5.2 0.0 312.0 12.3 2.4 30.1 High write I/O aligns with the rsync burst, further stressing the node.

Q: Solution – Mitigation Strategies 1. Separate node pools for replication and inference

Define two distinct NodePool resources (or separate instance groups) with dedicated taints.

Q: 2. Lower replication pod resource requests and enable burst buffering

Adjust the job spec to use streaming mode, reducing memory pressure.

Q: 3. Introduce a PriorityClass for replication lower than inference

apiVersion: scheduling.k8s.io/v1 kind: PriorityClass metadata: name: replication-low value: 500 globalDefault: false description: "Lower priority than inference pods" Update the replication job:

Q: 4. Rate‑limit rsync concurrency

Set --max-children=4 to limit parallel file transfers.

Q: 5. Adjust kubelet eviction thresholds for replication nodes

# /etc/kubernetes/kubelet-config.json { "evictionHard": { "memory.available": "200Mi", "cpu.throttling": "30%" } } Verify – Validation Steps Deploy the new node pools and confirm taints: $ kubectl get nodes -l role=replication -o jsonpath='{.items[*].spec.taints}' [{"key":"replication","value":"true","effect":"NoSchedule"}] Run a replication sync and monitor node metrics: $ kubectl logs job/replicate-model-v1 -n ml-replication -f Expect memory usage on replication nodes to stay below 1Gi and CPU below 1.2 cores.

In this article

Table of Contents

Problem – Mistral Inference Pods Evicted During Multi‑Region Replication Sync

The Mistral AI inference service runs as a set of Deployment pods behind a ClusterIP service. When a multi‑region replication job (running as a CronJob that streams model checkpoints from the primary region to secondary regions) starts, operators observe a spike in pod evictions:


Warning  Evicted  12s   kubelet, node-az1-01  Pod evicted: node(s) had insufficient memory (requested: 2Gi, limit: 4Gi)
Warning  Evicted  8s    kubelet, node-az2-03  Pod evicted: node(s) had insufficient cpu (requested: 1500m, limit: 2000m)

Impact includes:

Transient inference latency spikes as new pods are scheduled.
Cold‑start penalties for model loading (up to 30 seconds per pod).
Occasional request failures when the service scales down to zero during the eviction burst.

Root Cause – Resource Contention Between Inference and Replication Workloads

The replication sync process launches a set of Job pods that perform the following steps for each model version:

Mount a shared PersistentVolumeClaim (PVC) backed by an EFS‑like network file system.
Run rsync to copy the checkpoint files from the source region.
Validate checksum and update a ConfigMap with the new version identifier.

Each replication pod requests 2Gi of memory and 1500m of CPU to keep the copy operation in memory‑buffered mode. In a typical 5‑region deployment, the sync job launches 15 parallel pods per region. The combined request exceeds the node capacity, causing the kube‑scheduler to preempt lower‑priority inference pods (which run with priorityClassName: inference-high but lower than the default system-cluster-critical priority used by the replication jobs).

Additional contributing factors:

Node‑level kubelet eviction thresholds are set to memory.available<500Mi and cpu.throttling<10%, which are crossed quickly during bulk copy.
Replication pods share the same nodeSelector as inference pods, limiting the scheduler’s ability to spread load across the cluster.
Horizontal Pod Autoscaler (HPA) for inference is configured with a minReplicas: 3 but no maxReplicas guard against rapid scale‑down during eviction.

Debug – Investigation and Diagnostics

1. Identify eviction events


$ kubectl get events --field-selector reason=Evicted -A --sort-by=.metadata.creationTimestamp
NAMESPACE   KIND      NAME                     REASON   AGE   FROM                MESSAGE
ml-prod     Pod       mistral-infer-6f7c9b    Evicted  2m    kubelet, node-az1  Pod evicted: node(s) had insufficient memory (requested: 2Gi, limit: 4Gi)
ml-prod     Pod       mistral-infer-8a2d3e    Evicted  2m    kubelet, node-az2  Pod evicted: node(s) had insufficient cpu (requested: 1500m, limit: 2000m)

2. Correlate with replication job start time


$ kubectl get jobs -n ml-replication -o wide
NAME                     COMPLETIONS   DURATION   AGE   SELECTOR
replicate-model-v1       0/15          0s         2m    job-name=replicate-model-v1

The timestamps line up with the eviction events, confirming contention.

3. Inspect node resource pressure


$ kubectl top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-az1-01    2100m        105%   15Gi / 16Gi      94%
node-az2-03    1800m        90%    14Gi / 16Gi      88%

4. Review pod priority and preemption policy


$ kubectl get pod mistral-infer-6f7c9b -o yaml | grep priorityClassName
  priorityClassName: inference-high

The replication jobs use the default priority class system-cluster-critical, which preempts inference pods.

5. Check PVC binding and I/O saturation


$ iostat -x 1 3
Device            r/s   w/s   rkB/s   wkB/s  await  svctm  %util
fs-12345678       0.0   5.2   0.0    312.0   12.3   2.4    30.1

High write I/O aligns with the rsync burst, further stressing the node.

Solution – Mitigation Strategies

1. Separate node pools for replication and inference

Define two distinct NodePool resources (or separate instance groups) with dedicated taints.


# Before (single node pool)
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: mistral-cluster
  region: us-east-1
nodeGroups:
  - name: ml-nodes
    instanceType: m5.4xlarge
    desiredCapacity: 6


# After (dedicated pools)
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: mistral-cluster
  region: us-east-1
nodeGroups:
  - name: inference-nodes
    instanceType: m5.4xlarge
    desiredCapacity: 6
    labels: {role: inference}
  - name: replication-nodes
    instanceType: m5.4xlarge
    desiredCapacity: 4
    taints:
      - key: replication
        value: "true"
        effect: NoSchedule
    labels: {role: replication}

Apply the new configuration and cordon old nodes:


$ kubectl cordon node-az1-01
$ kubectl drain node-az1-01 --ignore-daemonsets --delete-emptydir-data

2. Lower replication pod resource requests and enable burst buffering

Adjust the job spec to use streaming mode, reducing memory pressure.


# Before
resources:
  requests:
    memory: "2Gi"
    cpu: "1500m"
  limits:
    memory: "4Gi"
    cpu: "2000m"

# After – streaming mode
env:
  - name: RSYNC_BUFFER_SIZE
    value: "64M"
resources:
  requests:
    memory: "512Mi"
    cpu: "500m"
  limits:
    memory: "1Gi"
    cpu: "1000m"

3. Introduce a `PriorityClass` for replication lower than inference


apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: replication-low
value: 500
globalDefault: false
description: "Lower priority than inference pods"

Update the replication job:


spec:
  template:
    spec:
      priorityClassName: replication-low

4. Rate‑limit rsync concurrency

Set --max-children=4 to limit parallel file transfers.


command: ["rsync", "-a", "--max-children=4", "/src/", "/dst/"]

5. Adjust kubelet eviction thresholds for replication nodes


# /etc/kubernetes/kubelet-config.json
{
  "evictionHard": {
    "memory.available": "200Mi",
    "cpu.throttling": "30%"
  }
}

Verify – Validation Steps

Deploy the new node pools and confirm taints:


$ kubectl get nodes -l role=replication -o jsonpath='{.items[*].spec.taints}'
[{"key":"replication","value":"true","effect":"NoSchedule"}]

Run a replication sync and monitor node metrics:


$ kubectl logs job/replicate-model-v1 -n ml-replication -f

Expect memory usage on replication nodes to stay below 1Gi and CPU below 1.2 cores.

Verify that inference pods remain running:


$ kubectl get pods -n ml-prod -l app=mistral-inference
NAME                     READY   STATUS    RESTARTS   AGE
mistral-infer-abc123     2/2     Running   0          5m

Check that no new eviction events appear for the inference namespace during the sync window.

Prevent – Operational Guardrails

Node pool segregation: Keep resource‑intensive background jobs on dedicated nodes with lower priority classes.
HPA safeguards: Set maxReplicas on inference HPA to avoid scaling to zero during transient pressure.
Admission controller policy: Enforce a maximum requests.memory of 1Gi for any pod labeled role=replication.
Monitoring: Create alerts on kube_node_status_condition{condition="MemoryPressure"} == 1 for nodes with the replication taint.
Scheduled throttling: Run replication jobs during off‑peak windows (e.g., 02:00‑04:00 UTC) and use CronJob startingDeadlineSeconds to avoid overlap.

FAQ – Common Follow‑Up Questions

Why do inference pods still get evicted after lowering replication pod requests?
If the replication job still spawns more parallel pods than the node can handle, the aggregate request may exceed capacity. Reduce parallelism in the Job spec or increase the size of the replication node pool.
Can I keep a single node pool and avoid evictions?
It is possible by using PodDisruptionBudget and setting a higher memory.available eviction threshold, but this reduces isolation and can still lead to latency spikes for inference.
How do I verify which priority class preempted a pod?
Inspect the eviction event message or the pod’s status.reason. The kubelet logs contain a line such as Preempted by higher priority pod: replication-low.
Is rsync the only way to copy checkpoints?
Alternatives like aws s3 sync or gcsfuse can stream data without large memory buffers, reducing pressure on the node.
What metrics should I add to my SLO dashboard?
Track pod_eviction_total{namespace="ml-prod"}, node_memory_pressure_seconds_total, and replication_job_duration_seconds to correlate evictions with sync duration.

Related Topic Hub: LLM Systems Troubleshooting Hub