TensorRT Model Serving Fails After Certificate Expiration

Q: Investigation and Debugging 1. Verify certificate validity

openssl x509 -noout -dates -in /etc/triton/certs/server.crt # Expected output: # notBefore=Mar 15 00:00:00 2024 GMT # notAfter=Mar 15 23:59:59 2025 GMT If notAfter is in the past, the cert is expired.

Q: 2. Inspect Triton server logs

journalctl -u triton -f | grep -i "certificate" # Sample output: # Mar 12 08:45:01 triton[1234]: Failed to load TLS certificate: certificate has expired # Mar 12 08:45:01 triton[1234]: Model 'resnet50_trt' failed to load: certificate has expired 3. Check gRPC client error details grpcurl -insecure -proto inference.proto \ -d '{"model_name":"resnet50_trt","inputs":[...]}' # Error: # rpc error: code = Unavailable desc = SSL handshake failed 4. Validate proxy (Envoy) configuration curl -v https://triton-gpu0.example.com/v2/health/ready # Expected 200 OK. # Actual: # * TLS handshake failed (error 35) 5. Confirm load balancer health check failure curl -k https://lb.example.com/healthz # Returns 503 Service Unavailable – upstream TLS error 6. Correlate with certificate rotation scripts Review the cron job that renews the wildcard cert. In the incident, the script exited with a non‑zero status, leaving the old cert in place.

Q: Step 3 – Restart Triton server

systemctl restart triton # Verify startup journalctl -u triton -n 20 | grep "TLS certificate" # Expected: # Mar 12 09:02:15 triton[1234]: Loaded TLS certificate successfully Step 4 – Verify load balancer health checks curl -v https://lb.example.com/healthz # Should now return 200 OK Why the fix works The server and proxy now present a valid certificate chain, allowing the gRPC client to complete the TLS handshake. The health check endpoint becomes reachable, causing the load balancer to mark the replica healthy and resume traffic routing to the TensorRT‑optimized path.

Q: Validation Health endpoint: curl -k https://triton-gpu0.example.com/v2/health/ready returns 200 OK. gRPC request: grpcurl -proto inference.proto -d … triton-gpu0.example.com:8001 succeeds without “SSL handshake failed”. Metrics: inference_requests_success_total returns to baseline; latency drops back to < 10 ms per request. Load balancer logs: No longer report TLS handshake errors; traffic distribution returns to 50/50 A/B split. Operational Experience

During the incident, the initial symptom (“502 Bad Gateway”) led some engineers to suspect a downstream model loading error, but the TLS error in the server logs was the true trigger. The following observations helped narrow the scope:

Problem Description

During a routine A/B test that routes production traffic across two GPU‑accelerated inference endpoints (Triton Server with TensorRT‑optimized models), the following symptoms appeared:

gRPC clients started failing with UNAVAILABLE: SSL handshake failed.
HTTP API gateway returned 502 Bad Gateway – TLS handshake timeout.
Model loading logs showed Failed to load TLS certificate: certificate has expired and model 'resnet50_trt' failed to load: certificate has expired.
Load balancer fell back to a degraded CPU‑only inference path, causing latency spikes >5× and a 30 % traffic loss.
Metrics indicated a sharp drop in inference_requests_success_total and a rise in inference_requests_failed_total.

These errors match the common error messages documented in community sources such as the GitHub issue triton‑inference‑server/server#2541 and the Stack Overflow question “Triton server SSL handshake fails after cert expiration”.

Root Cause Analysis

The failure chain originates from an expired X.509 certificate that is used for TLS termination on the inference endpoint. The relevant components are:

Component	Role	TLS Interaction
Triton Inference Server	Model repository & inference runtime	Loads server certificate at startup (see Triton TLS configuration docs).
Envoy / NGINX sidecar	TLS termination proxy	Uses the same certificate for client‑side TLS; must present a non‑expired cert to upstream.
gRPC client SDK	Inference request sender	Validates server certificate chain during handshake.

According to the NVIDIA Triton documentation (Model Repository and TLS Configuration), the server aborts startup if the supplied certificate is not valid, emitting the log line “Failed to load TLS certificate: certificate has expired”. The server then falls back to an insecure mode, but the surrounding load balancer is configured to reject non‑TLS traffic, causing the observed 502 responses.

Because the A/B router bases its health checks on the TLS endpoint health, the expired certificate marks the affected replica as unhealthy. The orchestrator’s fallback logic routes traffic to a legacy TensorFlow Serving stack, which explains the degradation and latency spikes.

Investigation and Debugging

1. Verify certificate validity

openssl x509 -noout -dates -in /etc/triton/certs/server.crt
# Expected output:
# notBefore=Mar 15 00:00:00 2024 GMT
# notAfter=Mar 15 23:59:59 2025 GMT

If notAfter is in the past, the cert is expired.

2. Inspect Triton server logs

journalctl -u triton -f | grep -i "certificate"
# Sample output:
# Mar 12 08:45:01 triton[1234]: Failed to load TLS certificate: certificate has expired
# Mar 12 08:45:01 triton[1234]: Model 'resnet50_trt' failed to load: certificate has expired

3. Check gRPC client error details

grpcurl -insecure -proto inference.proto \
  -d '{"model_name":"resnet50_trt","inputs":[...]}'
# Error:
# rpc error: code = Unavailable desc = SSL handshake failed

4. Validate proxy (Envoy) configuration

curl -v https://triton-gpu0.example.com/v2/health/ready
# Expected 200 OK.
# Actual:
# * TLS handshake failed (error 35)

5. Confirm load balancer health check failure

curl -k https://lb.example.com/healthz
# Returns 503 Service Unavailable – upstream TLS error

6. Correlate with certificate rotation scripts

Review the cron job that renews the wildcard cert. In the incident, the script exited with a non‑zero status, leaving the old cert in place.

Resolution

Step 1 – Renew the certificate

Obtain a new certificate (e.g., via Let’s Encrypt or internal PKI) and place it in the shared location.

# Before (expired)
cp /etc/triton/certs/server.crt /backup/server.crt.expired
# After (new)
cp /etc/letsencrypt/live/triton.example.com/fullchain.pem /etc/triton/certs/server.crt
cp /etc/letsencrypt/live/triton.example.com/privkey.pem /etc/triton/certs/server.key
chmod 640 /etc/triton/certs/server.key

Step 2 – Update proxy sidecar

If the sidecar mounts the certificate separately, synchronize the files:

# Before (sidecar still has old cert)
ls -l /etc/nginx/certs/server.crt
# After (copy new cert)
cp /etc/triton/certs/server.crt /etc/nginx/certs/server.crt
cp /etc/triton/certs/server.key /etc/nginx/certs/server.key
nginx -s reload

Step 3 – Restart Triton server

systemctl restart triton
# Verify startup
journalctl -u triton -n 20 | grep "TLS certificate"
# Expected:
# Mar 12 09:02:15 triton[1234]: Loaded TLS certificate successfully

Step 4 – Verify load balancer health checks

curl -v https://lb.example.com/healthz
# Should now return 200 OK

Why the fix works

The server and proxy now present a valid certificate chain, allowing the gRPC client to complete the TLS handshake. The health check endpoint becomes reachable, causing the load balancer to mark the replica healthy and resume traffic routing to the TensorRT‑optimized path.

Validation

Health endpoint: curl -k https://triton-gpu0.example.com/v2/health/ready returns 200 OK.
gRPC request: grpcurl -proto inference.proto -d … triton-gpu0.example.com:8001 succeeds without “SSL handshake failed”.
Metrics: inference_requests_success_total returns to baseline; latency drops back to < 10 ms per request.
Load balancer logs: No longer report TLS handshake errors; traffic distribution returns to 50/50 A/B split.

Operational Experience

During the incident, the initial symptom (“502 Bad Gateway”) led some engineers to suspect a downstream model loading error, but the TLS error in the server logs was the true trigger. The following observations helped narrow the scope:

Only the GPU‑0 replica reported “certificate has expired”; GPU‑1 continued serving, confirming a per‑node cert mount issue.
Sidecar NGINX logs showed “SSL_ERROR_RX_RECORD_TOO_LONG”, a classic indicator of a mismatched TLS termination point.
Production monitoring did not have an alert for certificate_not_after expiry, which delayed detection by several days.

Best Practices and Prevention

Certificate expiry monitoring: Export notAfter as a Prometheus metric (tls_certificate_expiry_seconds) and set an alert for expires_in < 7d.
Automated rotation: Use a Kubernetes cert-manager Issuer or an internal renewal script that updates both the server and sidecar volumes atomically.
Rolling restart strategy: Deploy new certs via a RollingUpdate with maxUnavailable: 0 to avoid a full outage.
Health check design: Separate TLS health from model health; configure the load balancer to consider a replica unhealthy only after both /v2/health/ready and a successful TLS handshake.
Logging consistency: Include the full TLS handshake error path in server logs (e.g., grpc_server.cc:TLS handshake failed: certificate expired) to aid future triage.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the gRPC client report “UNAVAILABLE: SSL handshake failed” instead of a certificate‑specific error?
The client aborts the connection as soon as the TLS handshake cannot be completed; the underlying OpenSSL error (“certificate has expired”) is not propagated through gRPC’s error mapping.
Can I keep the server running in insecure mode while I rotate the certificate?
Yes, but only if the load balancer and client SDKs are configured to allow insecure connections. This defeats the security model and is not recommended for production.
How do I verify which cipher suite was negotiated after renewal?
Run openssl s_client -connect triton-gpu0.example.com:443 -servername triton.example.com -tls1_2 and look for the Cipher line in the output.
Does TensorRT runtime need any special handling for TLS certificates?
TensorRT itself does not manage TLS; the responsibility lies with Triton or the surrounding proxy. However, the runtime will refuse to load a model if the server process cannot start TLS, as shown in the “Model initialization failed: unable to verify server certificate” log entry.
What should I do if only some replicas report certificate expiration?
Check the volume mounts for each replica. In Kubernetes, ensure the secret containing the certificate is mounted with readOnly: true and that all pods reference the same secret name.