Stale inference results from NVIDIA GPU endpoint slice after autoscaling

Problem – Stale Inference Results from NVIDIA GPU EndpointSlice after Autoscaling

In a hybrid‑cloud deployment that mixes on‑premises NVIDIA GPU nodes with cloud‑based orchestration, engineers observed that after a Horizontal Pod Autoscaler (HPA) scale‑up or scale‑down event the Triton Inference Server began returning identical outputs for distinct inputs. The symptom manifested as:

• Log entry from Triton: ModelInstance::Infer: returning cached result for request id 42c1e7
• Kubernetes ingress log: Inference request timed out after 30s, returning previous result
• EndpointSlice controller messages: EndpointSlice update ignored: existing endpoints still considered healthy
• Observed latency drop (responses returned instantly) but the payload was unchanged across different payloads.

The impact was a silent data‑quality regression: downstream services received stale predictions, leading to incorrect business decisions while the system appeared healthy.

Root Cause Analysis

The issue originates from the interaction of three components:

1. Triton model lifecycle – According to the Triton Model Repository documentation, model instances are kept in memory across pod restarts when the model_version_policy permits reuse. If the model files have not changed, Triton may skip a full reload and keep the previous in‑memory state, which includes the result cache used for repeated inference requests.
2. Kubernetes EndpointSlice handling – The EndpointSlice design (Kubernetes SIG Architecture) updates the slice only when the pod’s readiness probe succeeds. During rapid autoscaling, the old pod IP can remain marked healthy while the new pod is still initializing, causing the load balancer to continue routing traffic to the stale pod.
3. GPU Operator & device plugin – The NVIDIA GPU Operator (Autoscaling guide) recreates GPU pods on node drain. The device plugin may reuse shared memory segments, and Triton can inherit a stale shm region, preserving cached inference results even after the pod process restarts.

Combined, these mechanisms allow a pod that has not yet re‑loaded its model to continue serving requests, while the EndpointSlice still directs traffic to it. The result is the “cached result” log entry reported in the community issue #2749 and the EndpointSlice bug discussion #119567.

Investigation and Debugging Steps

1. Verify EndpointSlice contents

kubectl get endpointslice -n inference -l app=triton -o yaml

Typical output before the fix (old pod IP still present):

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: triton-service-abcde
  namespace: inference
addressType: IPv4
ports:
- name: http
  port: 8000
  protocol: TCP
endpoints:
- addresses:
  - 10.1.2.15   # stale pod
  conditions:
    ready: true
- addresses:
  - 10.1.2.22   # new pod (not yet ready)
  conditions:
    ready: false

2. Inspect Triton logs for cache usage

kubectl logs -n inference $(kubectl get pod -l app=triton -o name | head -n1) -c triton | grep "cached result"

Sample log line:

2024-08-12 14:03:27.123456Z [I] ModelInstance::Infer: returning cached result for request id 7f9a3b

3. Check model version timestamps

kubectl exec -n inference $(kubectl get pod -l app=triton -o name | head -n1) -- \
  ls -l /models/my_model/1/model.plan

If the timestamp has not changed, Triton may skip a reload per the model_version_policy rules.

4. Confirm GPU Operator shared memory reuse

kubectl exec -n nvidia-gpu-operator $(kubectl get pod -l app=nvidia-device-plugin -o name | head -n1) -- \
  ls -l /dev/shm | grep triton

Presence of stale triton_shm_* files indicates that the shared memory segment survived pod recreation.

5. Observe load‑balancer routing

curl -s -H "Host: inference.example.com" http:///v2/models/my_model/infer -d @payload1.json
curl -s -H "Host: inference.example.com" http:///v2/models/my_model/infer -d @payload2.json

If both responses are identical despite different payloads, the request is hitting the stale pod.

Solution – Ensuring Fresh Inference After Autoscaling

1. Enforce explicit model reload on pod start

Update the Triton container command to include a post‑start hook that forces a model unload/load cycle.

Before:

containers:
- name: triton
  image: nvcr.io/nvidia/tritonserver:23.09-py3
  args: ["tritonserver", "--model-repository=/models"]

After:

containers:
- name: triton
  image: nvcr.io/nvidia/tritonserver:23.09-py3
  lifecycle:
    postStart:
      exec:
        command:
        - /bin/sh
        - -c
        - |
          tritonserver --model-repository=/models &
          PID=$!
          # Wait for server to be ready
          until curl -s http://localhost:8000/v2/health/ready; do sleep 1; done
          # Force model reload
          curl -X POST http://localhost:8000/v2/repository/models/my_model/load
          wait $PID
  args: ["tritonserver", "--model-repository=/models", "--model-control-mode=explicit"]

Setting --model-control-mode=explicit disables automatic loading based on file timestamps, giving the script full control.

2. Add a readiness probe that blocks traffic until the model is fully loaded

readinessProbe:
  httpGet:
    path: /v2/health/ready
    port: 8000
  initialDelaySeconds: 5
  periodSeconds: 2
  failureThreshold: 30

This ensures the EndpointSlice controller only marks the pod as ready after the explicit load call succeeds.

3. Configure the GPU Operator to clean shared memory on pod termination

Patch the device plugin DaemonSet to delete /dev/shm/triton_* on preStop hook.

containers:
- name: nvidia-device-plugin
  lifecycle:
    preStop:
      exec:
        command: ["/bin/sh", "-c", "rm -f /dev/shm/triton_*"]

4. Force EndpointSlice controller resynchronization after scaling events

Apply a short --resync-period to the controller manager or trigger a manual resync:

kubectl rollout restart deployment/kube-controller-manager -n kube-system
# Or, for a custom controller:
kubectl patch endpointslice triton-service-abcde -n inference --type=merge -p '{"metadata":{"annotations":{"resync":"true"}}}'

5. Disable sticky sessions in the ingress/load balancer

Sticky sessions can keep routing to the stale pod even after it is marked unhealthy.

annotations:
  nginx.ingress.kubernetes.io/affinity: "none"

Verification – Confirming the Fix

1. Check EndpointSlice health – New slice should list only ready pod IPs.
2. Inspect Triton logs – No “cached result” entries after a fresh request.
3. Run functional test – Send two distinct payloads and compare outputs.

Example verification script:

#!/usr/bin/env bash
set -euo pipefail

PAYLOAD1='{"inputs":[{"name":"input","shape":[1,3],"datatype":"FP32","data":[0.1,0.2,0.3]}]}'
PAYLOAD2='{"inputs":[{"name":"input","shape":[1,3],"datatype":"FP32","data":[0.9,0.8,0.7]}]}'

RESP1=$(curl -s -X POST http://lb-ip/v2/models/my_model/infer -d "$PAYLOAD1")
RESP2=$(curl -s -X POST http://lb-ip/v2/models/my_model/infer -d "$PAYLOAD2")

if [[ "$RESP1" == "$RESP2" ]]; then
  echo "❌ Stale result detected"
  exit 1
else
  echo "✅ Fresh inference confirmed"
fi

Successful run prints ✅ Fresh inference confirmed and the logs show a line such as:

2024-08-12 14:15:42.987654Z [I] ModelInstance::Infer: completed inference for request id a3d9f1

Prevention – Operational Guardrails

• Monitoring: Alert on ModelInstance::Infer: returning cached result and on EndpointSlice readiness mismatches.
• Autoscaling policy: Add a scaleDownDelay (e.g., 120s) to give new pods time to load models before old pods are terminated.
• Model version policy: Use latest with explicit version bump on each deployment, forcing Triton to reload.
• Health checks: Combine /v2/health/ready with a custom endpoint that verifies model inference correctness (e.g., a “smoke test” request).
• EndpointSlice controller tuning: Set --endpoint-slice-sync-period=10s in the controller manager to reduce stale endpoint windows.

FAQ – Common Follow‑Up Questions

1. Why does the stale result appear only after a scale‑up and not after a scale‑down?
   Because during scale‑up a new pod is added while the old pod remains ready. The load balancer may continue sending traffic to the old pod until its readiness probe fails, whereas scale‑down removes the pod immediately, exposing the stale instance.
2. Can I rely on Triton’s built‑in model version policy to avoid stale caches?
   The default policy reloads only when the model file timestamp changes. If the file is unchanged, Triton may reuse the in‑memory instance, preserving any cached results. Using --model-control-mode=explicit and triggering a manual load solves this.
3. Is disabling sticky sessions enough?
   It eliminates one cause, but you still need to ensure EndpointSlice health reflects the true readiness of the pod. Combine sticky‑session removal with proper readiness probes and controller resync.
4. How do I clear the shared memory segment without restarting the pod?
   Execute rm -f /dev/shm/triton_* inside the container and then call curl -X POST http://localhost:8000/v2/repository/models/<model>/unload followed by a load request.
5. What alert thresholds should I set for detecting stale inference?
   Alert when the log pattern ModelInstance::Infer: returning cached result appears more than once in a 30‑second window, or when the ratio of identical inference responses for distinct payload hashes exceeds 5% over a minute.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub