Azure VM Endpoint Slice Stale After Model Redeploy On Edge Node

Q: Debug – Investigation Steps 1. Verify the Service and EndpointSlice objects

kubectl get svc model-endpoint -o yaml Check the selector matches the new pod labels.

Q: 2. Inspect controller manager logs for sync errors

kubectl logs -n kube-system -l component=kube-controller-manager \ --since=5m | grep EndpointSlice Look for lines such as:

Q: 3. Check Azure IoT Edge runtime logs (if using IoT Edge)

journalctl -u iotedge --since "5 minutes ago" | grep "EndpointSlice" Typical entry:

Q: 4. Correlate model version logs

kubectl logs deployment/model-endpoint -c inference-container \ | grep "Model version" Sample output showing mixed versions:

Q: 5. Reproduce the routing behavior with a curl trace

curl -v http://model-endpoint:80/predict -d '{"data": [1,2,3]}' Inspect the Server header or a custom version field in the response to confirm which model served the request.

Q: Option B – Patch the Service to Trigger a Slice Re‑generation (zero‑downtime)

# Add a dummy annotation to force Service update kubectl patch svc model-endpoint -p '{"metadata":{"annotations":{"endpoint-slice-refresh":"'"$(date +%s)"'"}}}' The Service controller detects the change, deletes the old EndpointSlice, and creates a fresh one.

Q: Option C – Adjust Controller Settings to Avoid Cache Staleness (long‑term)

Modify the EndpointSlice controller’s --endpoint-slice-sync-period and --endpoint-slice-ttl flags in the AKS control plane (or via a custom kube-controller-manager manifest) to a lower interval, e.g., 30 seconds, ensuring quicker reconciliation.

In this article

Table of Contents

Problem – Stale EndpointSlice Causes Outdated Model Inference on Azure VM Edge Nodes

After redeploying a new machine‑learning model to an Azure VM that hosts an Azure ML online endpoint (often via Azure IoT Edge or AKS), a portion of inference requests continue to be served by the previous model version. The symptom is typically observed as:

Mixed predictions: Model version mismatch: expected v2.1, got v2.0 in container logs.
HTTP 502 errors from the ingress when traffic hits a terminated pod.
Metrics showing a sudden drop in request_success_rate while request_latency remains unchanged.

In production, this manifested as a 30 % error rate for up to 15 minutes after a model redeploy on a manufacturing line edge node.

Root Cause – EndpointSlice Controller Fails to Refresh After Model Redeploy

The Azure ML online endpoint is exposed through a Kubernetes Service. AKS (or the IoT Edge runtime) creates an EndpointSlice object that holds the IP addresses of the pods serving the model. When a new model version is rolled out, the underlying pods are recreated with new IPs, but the EndpointSlice controller sometimes:

Detects a conflict while updating the slice (Failed to sync EndpointSlice: conflict in kube-controller-manager logs).
Considers the existing slice “up‑to‑date” because the cache entry has not expired (Cache hit for endpoint slice, skipping update in Azure IoT Edge runtime logs).
Fails to delete the stale slice during a rapid node‑pool scaling event, leading to a race condition (see the fleet‑wide incident where scaling interfered with slice cleanup).

Consequently, the Service continues to route a fraction of traffic to the old pod IPs retained in the stale EndpointSlice, producing outdated inference results.

Debug – Investigation Steps

1. Verify the Service and EndpointSlice objects

kubectl get svc model-endpoint -o yaml

Check the selector matches the new pod labels.

kubectl get endpointslice -l kubernetes.io/service-name=model-endpoint -o yaml

Typical stale slice output:

apiVersion: discovery.k8s.io/v1
kind: EndpointSlice
metadata:
  name: model-endpoint-abc123
  labels:
    kubernetes.io/service-name: model-endpoint
addressType: IPv4
endpoints:
- addresses: ["10.244.1.12"]   # old pod IP
  conditions:
    ready: true
- addresses: ["10.244.2.45"]   # new pod IP
  conditions:
    ready: true

2. Inspect controller manager logs for sync errors

kubectl logs -n kube-system -l component=kube-controller-manager \
  --since=5m | grep EndpointSlice

Look for lines such as:

Failed to sync EndpointSlice: conflict

3. Check Azure IoT Edge runtime logs (if using IoT Edge)

journalctl -u iotedge --since "5 minutes ago" | grep "EndpointSlice"

Typical entry:

Cache hit for endpoint slice, skipping update

4. Correlate model version logs

kubectl logs deployment/model-endpoint -c inference-container \
  | grep "Model version"

Sample output showing mixed versions:

2026-08-28T12:03:12Z INFO Model version mismatch: expected v2.1, got v2.0

5. Reproduce the routing behavior with a curl trace

curl -v http://model-endpoint:80/predict -d '{"data": [1,2,3]}'

Inspect the Server header or a custom version field in the response to confirm which model served the request.

Solution – Force EndpointSlice Refresh and Ensure Consistent Updates

Option A – Manual Cleanup (quick remediation)

Delete the stale EndpointSlice objects; the controller will recreate them based on the current pod set.

# List stale slices
kubectl get endpointslice -l kubernetes.io/service-name=model-endpoint

# Delete all slices for the service
kubectl delete endpointslice -l kubernetes.io/service-name=model-endpoint

After deletion, verify that only the new pod IPs appear:

kubectl get endpointslice -l kubernetes.io/service-name=model-endpoint -o jsonpath='{.items[*].endpoints[*].addresses}'

Option B – Patch the Service to Trigger a Slice Re‑generation (zero‑downtime)

# Add a dummy annotation to force Service update
kubectl patch svc model-endpoint -p '{"metadata":{"annotations":{"endpoint-slice-refresh":"'"$(date +%s)"'"}}}'

The Service controller detects the change, deletes the old EndpointSlice, and creates a fresh one.

Option C – Adjust Controller Settings to Avoid Cache Staleness (long‑term)

Modify the EndpointSlice controller’s --endpoint-slice-sync-period and --endpoint-slice-ttl flags in the AKS control plane (or via a custom kube-controller-manager manifest) to a lower interval, e.g., 30 seconds, ensuring quicker reconciliation.

# Example snippet for a custom manifest
apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-controller-manager
  namespace: kube-system
data:
  kube-controller-manager.yaml: |
    ...
    - --endpoint-slice-sync-period=30s
    - --endpoint-slice-ttl=2m
    ...

Option D – Use Rolling Deployment with Pod Disruption Budgets

Configure the deployment to replace pods one‑by‑one while keeping the Service selector stable. This reduces the window where old and new pods coexist, minimizing the chance of a stale slice.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: model-endpoint
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: model-endpoint
        version: v2.1   # update label on each redeploy
    spec:
      containers:
      - name: inference-container
        image: myregistry.azurecr.io/model:v2.1

Verify – Confirm the Stale Slice Issue Is Resolved

Re‑run the EndpointSlice inspection; ensure only current pod IPs are listed.
Execute a series of curl requests and check the model version field in the response; all should report the new version.
Monitor the model_endpoint_requests_total metric for a uniform success rate and no spikes in 502 errors.
Check controller logs for the absence of Failed to sync EndpointSlice: conflict or Cache hit for endpoint slice messages.

# Example verification script
for i in {1..20}; do
  curl -s http://model-endpoint/predict -d '{"data":[1,2,3]}' | jq .model_version
done | sort | uniq -c
# Expected output:
# 20 v2.1

Prevent – Operational Guardrails and Best Practices

Enable EndpointSlice metrics (apiserver_requests_total{resource="endpointslices"}) and set alerts on sudden increases in endpoint_slice_stale (custom metric derived from controller logs).
Pin Service selectors to immutable labels (e.g., app=model-endpoint) and change only the version label on redeploy.
Automate EndpointSlice cleanup in CI/CD pipelines: after a successful deployment, run a post‑step that deletes any EndpointSlice objects older than a configurable TTL.
Configure a short sync period for the EndpointSlice controller in AKS clusters that host frequent model updates (see Azure AKS EndpointSlice documentation).
Use health probes that verify the model version inside the pod; failing probes will cause the pod to be removed from the slice promptly.

FAQ – Common Follow‑Up Questions

Why does the stale EndpointSlice appear only after a rapid CI/CD deployment?
Because the controller’s cache may not have expired before the new pods are created, leading to a conflict error and the old slice persisting.
Can I rely on Kubernetes Service DNS to hide EndpointSlice issues?
DNS resolves to the Service IP, but routing still depends on the EndpointSlice contents; stale slices will still direct traffic to old pod IPs.
Is there a way to force a full EndpointSlice rebuild without deleting the Service?
Adding or modifying an annotation on the Service (as shown in Option B) triggers the controller to recreate the slice.
How does Istio or another service mesh interact with stale EndpointSlices?
Istio’s Envoy sidecars watch the Service endpoints; if the EndpointSlice is stale, Envoy continues to route to the old pods until it receives an updated endpoint list.
Do Azure IoT Edge modules suffer from the same issue as AKS?
Yes. The IoT Edge runtime also creates EndpointSlice objects for module services; the same cache‑hit log message appears, and manual slice cleanup or annotation patching resolves it.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub