ONNX Runtime inference fails after CI/CD deployment due to incorrect routing

Problem – ONNX Runtime Inference Fails After CI/CD Deployment

During a routine CI/CD rollout, newly built ONNX model containers are deployed to a Kubernetes cluster. After the deployment completes, API calls that should hit the onnx-runtime service return HTTP 502/504 errors and the ONNX Runtime logs show Failed to load model: file not found. The issue is reproducible only for the freshly deployed version; previous stable releases continue to work.

Typical symptoms observed:

• Client receives 502 Bad Gateway or 504 Gateway Timeout from the ingress endpoint.
• ONNX Runtime container logs contain:

2026-06-07 10:12:34.567 INFO  onnxruntime: SessionOptions: Model file not found: /models/model.onnx
2026-06-07 10:12:34.568 ERROR onnxruntime: Failed to create session

• Ingress controller (NGINX) access logs show requests routed to the model-loader service instead of onnx-runtime:

10.0.2.15 - - [07/Jun/2026:10:12:34 +0000] "POST /v1/predict HTTP/1.1" 502 162 "-" "curl/7.79.1"

Impact includes failed downstream pipelines, SLA breaches, and increased mean time to recovery (MTTR) for model serving.

Root Cause – Misconfigured Ingress Routing Rules

The CI/CD pipeline uses Helm to render a values.yaml file that defines an Ingress resource for each model version. A recent change introduced a templating bug where the path field for the new version incorrectly re‑used the placeholder {{ .Release.Name }} instead of the model‑specific {{ .Values.model.name }}. As a result, the generated Ingress sent /v1/predict traffic to the generic model-loader service, which does not host the ONNX Runtime binary. The old service continued to serve traffic for the previous version, masking the problem until the new version became the default backend.

Key points that led to the failure:

• Dynamic provisioning of model endpoints relies on unique host/path combos per version.
• Ingress rules were generated from a shared Helm chart without strict validation of the serviceName field.
• The CI pipeline promoted the new version to the stable label before verifying that the Ingress correctly pointed to the onnx-runtime service.

Debug – Investigation and Diagnostic Steps

Below is a reproducible debugging workflow that isolates the routing problem.

1. Inspect the Ingress resource created by the deployment

kubectl get ingress onnx-model-v2 -n ml-serving -o yaml

Sample output (incorrect configuration):

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: onnx-model-v2
spec:
  rules:
  - host: models.example.com
    http:
      paths:
      - path: /v1/predict
        pathType: Prefix
        backend:
          service:
            name: model-loader            # <-- should be onnx-runtime
            port:
              number: 80

2. Compare with a known‑good Ingress from a previous release

kubectl get ingress onnx-model-v1 -n ml-serving -o yaml

Correct configuration:

...
            backend:
              service:
                name: onnx-runtime
                port:
                  number: 8000
...

3. Verify the service endpoints

kubectl get svc -n ml-serving

NAME            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
onnx-runtime    ClusterIP   10.96.12.34             8000/TCP   12d
model-loader    ClusterIP   10.96.45.67             80/TCP     12d

4. Capture a request trace through the ingress

kubectl exec -n kube-system $(kubectl get pod -l app=nginx-ingress-controller -n kube-system -o name | head -n1) -- \
  curl -v -X POST http://models.example.com/v1/predict -d '{"inputs":[1,2,3]}'

Expected 200 OK from ONNX Runtime; actual response is 502 Bad Gateway, confirming misrouting.

5. Review Helm template rendering

helm template mymodel ./chart -f values.yaml > rendered.yaml
grep -A3 "backend:" rendered.yaml | grep "service:"

Output shows the erroneous model-loader reference.

Solution – Correcting Ingress Routing

The fix involves updating the Helm chart to reference the proper service name and adding a validation hook to prevent future regressions.

1. Update the Helm template

Before:

{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "model.fullname" . }}
spec:
  rules:
  - host: {{ .Values.ingress.host }}
    http:
      paths:
      - path: {{ .Values.ingress.path }}
        pathType: Prefix
        backend:
          service:
            name: {{ .Release.Name }}            # <-- incorrect
            port:
              number: {{ .Values.service.port }}
{{- end }}

After:

{{- if .Values.ingress.enabled }}
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: {{ include "model.fullname" . }}
spec:
  rules:
  - host: {{ .Values.ingress.host }}
    http:
      paths:
      - path: {{ .Values.ingress.path }}
        pathType: Prefix
        backend:
          service:
            name: {{ .Values.service.name }}      # <-- fixed
            port:
              number: {{ .Values.service.port }}
{{- end }}

2. Add a Helm test hook to assert correct routing

apiVersion: v1
kind: Pod
metadata:
  name: ingress-routing-test
  annotations:
    "helm.sh/hook": test
spec:
  containers:
  - name: curl
    image: curlimages/curl:7.88.1
    command: ["/bin/sh", "-c"]
    args:
      - |
        set -e
        RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" http://{{ .Values.ingress.host }}{{ .Values.ingress.path }})
        if [ "$RESPONSE" != "200" ]; then
          echo "Ingress routing test failed: HTTP $RESPONSE"
          exit 1
        fi
        echo "Ingress routing test passed"

3. Redeploy the corrected chart

helm upgrade --install onnx-model ./chart \
  -n ml-serving \
  -f values.yaml \
  --wait --timeout 5m

4. Verify the new Ingress object

kubectl get ingress onnx-model-v2 -n ml-serving -o yaml | grep "name:"

Output now shows name: onnx-runtime.

Verify – Confirming Successful Inference

Run an end‑to‑end request and inspect both the client response and ONNX Runtime logs.

Functional test

curl -s -X POST http://models.example.com/v1/predict \
  -H "Content-Type: application/json" \
  -d '{"inputs":[[1.0,2.0,3.0]]}'

Expected output (example):

{"outputs":[[0.987, 0.012, 0.001]]}

ONNX Runtime log verification

kubectl logs -l app=onnx-runtime -n ml-serving --tail=20

Relevant lines:

2026-06-07 10:45:12.112 INFO  onnxruntime: Session created successfully for /models/model.onnx
2026-06-07 10:45:12.115 INFO  onnxruntime: Inference completed in 12.4 ms

Metrics check (Prometheus query)

sum(rate(onnx_inference_duration_seconds_sum[5m])) by (model_name)

Should return non‑zero values for the newly deployed model.

Prevent – Best Practices to Avoid Routing Regression

• Template linting*: Use helm lint and helm template diff checks in CI to detect placeholder misuse.
• Ingress validation*: Add a Kubernetes admission controller (e.g., OPA Gatekeeper) rule that enforces the backend.service.name to match a whitelist of allowed services per namespace.
• Canary rollout*: Deploy new model versions behind a separate sub‑path (e.g., /v2/predict) and gradually shift traffic using Istio VirtualService or NGINX weight‑based routing.
• Automated health probes*: Ensure the Ingress health check points to a lightweight /healthz endpoint exposed by the ONNX Runtime container.
• Versioned ConfigMaps*: Store the exact service name and port in a ConfigMap that the Helm chart reads; version the ConfigMap alongside the model artifact.

FAQ – Common Follow‑Up Questions

1. Why did the old model version keep working while the new one failed?
   Because the Ingress rule for the new version overwrote the default backend. Existing pods continued to serve traffic until the new rule became active, after which all requests were misrouted.
2. Can I use a single Ingress resource for multiple model versions?
   Yes, by defining distinct path entries (e.g., /v1/predict, /v2/predict) each pointing to its own service. Ensure path matching is Exact or Prefix as required.
3. How do I debug a 502 error when the ingress controller logs are empty?
   Check the upstream service pods for readiness probes failures and verify that the Service Endpoints list contains the expected pod IPs (kubectl get endpoints <svc>). Missing endpoints often cause 502 responses.
4. Is there a way to automatically test Ingress routing before promotion?
   Integrate Helm test hooks (as shown) or a post‑deploy Kubernetes Job that issues HTTP requests against the ingress host and fails the pipeline if non‑200 responses are returned.
5. What should I do if the model file path is correct but ONNX Runtime still reports “file not found”?
   Confirm the container's security context and volume mounts. The pod may be running as a non‑root user lacking read permission on the mounted model directory. Adjust securityContext.runAsUser or set appropriate fsGroup on the PVC.

Related Topic Hub: Model Serving Troubleshooting Hub