TLS Handshake Timeout In AMD GPU Inference Pods On Kubernetes

Q: Debug 1. Verify library versions inside the pod

$ kubectl exec -it inference-abc123 -- bash root@inference-abc123:/# ldd /app/inference-binary | grep libcrypto libcrypto.so.1.0.2 => /usr/lib/amdgpu/libcrypto.so.1.0.2 (0x7f8c2d400000) root@inference-abc123:/# openssl version OpenSSL 3.0.2 15 Mar 2022 The binary is linked against the driver’s libcrypto.so.1.0.2 despite the system OpenSSL being newer.

Q: 2. Capture TLS handshake with openssl s_client

$ openssl s_client -connect storage.example.com:443 -tls1_3 -servername storage.example.com CONNECTED(00000003) ... handshake failure Forcing TLS 1.3 fails, confirming the library cannot negotiate the required cipher suite.

Q: 3. Inspect kernel TLS offload status

$ ethtool -k eth0 | grep tls tls-segmentation: off [fixed] tls-hw-tx-offload: off [fixed] tls-hw-rx-offload: off [fixed] The driver disables hardware TLS offload, forcing user‑space TLS which is already broken by the library mismatch.

Q: 4. Review pod logs for timeout pattern

2024-10-12T08:15:23.412Z ERROR inference-service: TLS handshake timeout after 10s 2024-10-12T08:15:23.415Z ERROR inference-service: certificate verification failed: unable to get local issuer certificate 2024-10-12T08:15:23.417Z INFO inference-service: retrying model download (attempt 1/3) The retry loop confirms the client treats the failure as a transient network error.

Q: Step 3 – Rebuild and redeploy the inference service

$ docker build -t registry.example.com/inference:latest . $ kubectl set image deployment/inference inference=registry.example.com/inference:latest --record Step 4 – (Optional) Enable kernel TLS offload if supported If the kernel is rebuilt with CONFIG_TLS=y, enable offload on the amdgpu interface:

Problem

AI inference pods that run on AMD GPU nodes in a Kubernetes cluster intermittently fail to establish HTTPS connections to external model storage services. The containers emit errors such as:


2024-10-12T08:15:23.412Z ERROR inference-service: TLS handshake timeout after 10s
2024-10-12T08:15:23.415Z ERROR inference-service: certificate verification failed: unable to get local issuer certificate

The failure manifests during model download at pod startup and during periodic telemetry uploads. The pod remains in Running state, but the application reports 503 Service Unavailable to downstream consumers.

Root Cause

The timeout originates from a mismatch between the AMD GPU driver’s user‑space runtime (amdgpu-pro) and the OpenSSL library bundled in the inference container. The driver installs a custom libcrypto.so version (1.0.2) that is loaded before the container’s own OpenSSL 3.0.2. This older library lacks support for TLS 1.3 and several modern cipher suites required by the external storage endpoint (e.g., AES256-GCM-SHA384).

When the inference binary links against the driver‑provided libcrypto.so, the TLS handshake proceeds until the CertificateVerify step, where the server requests a cipher suite not present in the older library. OpenSSL aborts the handshake, and the client library reports a generic timeout after the configured 10‑second deadline.

Additional contributing factors:

AMD GPU nodes run a hardened kernel with CONFIG_TLS=0, disabling kernel‑offloaded TLS for the amdgpu network interface.
The container image uses FROM ubuntu:22.04 which pulls OpenSSL 3.x, but the dynamic linker resolves the older library from /usr/lib/amdgpu due to LD_LIBRARY_PATH set by the driver installer.
Only AMD GPU nodes exhibit the problem; identical pods on Intel or NVIDIA nodes succeed because they ship with compatible OpenSSL versions.

Debug

1. Verify library versions inside the pod


$ kubectl exec -it inference-abc123 -- bash
root@inference-abc123:/# ldd /app/inference-binary | grep libcrypto
    libcrypto.so.1.0.2 => /usr/lib/amdgpu/libcrypto.so.1.0.2 (0x7f8c2d400000)
root@inference-abc123:/# openssl version
OpenSSL 3.0.2 15 Mar 2022

The binary is linked against the driver’s libcrypto.so.1.0.2 despite the system OpenSSL being newer.

2. Capture TLS handshake with `openssl s_client`


$ openssl s_client -connect storage.example.com:443 -tls1_3 -servername storage.example.com
CONNECTED(00000003)
...
handshake failure

Forcing TLS 1.3 fails, confirming the library cannot negotiate the required cipher suite.

3. Inspect kernel TLS offload status


$ ethtool -k eth0 | grep tls
tls-segmentation: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]

The driver disables hardware TLS offload, forcing user‑space TLS which is already broken by the library mismatch.

4. Review pod logs for timeout pattern


2024-10-12T08:15:23.412Z ERROR inference-service: TLS handshake timeout after 10s
2024-10-12T08:15:23.415Z ERROR inference-service: certificate verification failed: unable to get local issuer certificate
2024-10-12T08:15:23.417Z INFO inference-service: retrying model download (attempt 1/3)

The retry loop confirms the client treats the failure as a transient network error.

Solution

The fix consists of ensuring the inference binary uses a modern OpenSSL library and, optionally, disabling the driver’s library path injection.

Step 1 – Remove driver‑provided OpenSSL from `LD_LIBRARY_PATH`

Edit the DaemonSet that installs the AMD GPU runtime and unset the environment variable for inference pods.


# Before (DaemonSet snippet)
env:
  - name: LD_LIBRARY_PATH
    value: "/usr/lib/amdgpu:/usr/local/lib:/usr/lib"

# After
env:
  - name: LD_LIBRARY_PATH
    value: "/usr/local/lib:/usr/lib"

Step 2 – Install a compatible OpenSSL runtime inside the container

Update the Dockerfile to copy the host’s OpenSSL 3.x libraries into the image and link them explicitly.


# Before
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y ca-certificates

# After
FROM ubuntu:22.04
RUN apt-get update && apt-get install -y ca-certificates libssl3
# Copy host OpenSSL 3.x libraries to avoid driver shadowing
COPY /usr/lib/x86_64-linux-gnu/libssl.so.3 /usr/lib/
COPY /usr/lib/x86_64-linux-gnu/libcrypto.so.3 /usr/lib/
ENV LD_LIBRARY_PATH="/usr/lib:${LD_LIBRARY_PATH}"

Step 3 – Rebuild and redeploy the inference service


$ docker build -t registry.example.com/inference:latest .
$ kubectl set image deployment/inference inference=registry.example.com/inference:latest --record

Step 4 – (Optional) Enable kernel TLS offload if supported

If the kernel is rebuilt with CONFIG_TLS=y, enable offload on the amdgpu interface:


$ sysctl -w net.ipv4.tcp_fastopen=3
$ ethtool -K eth0 tls-hw-tx-offload on tls-hw-rx-offload on

This step is not required for the handshake fix but improves performance for large model transfers.

Verify

1. Confirm library linkage


$ kubectl exec -it inference-abc123 -- bash
root@inference-abc123:/# ldd /app/inference-binary | grep libcrypto
    libcrypto.so.3 => /usr/lib/libcrypto.so.3 (0x7f9d2c600000)
root@inference-abc123:/# openssl version
OpenSSL 3.0.2 15 Mar 2022

2. Successful TLS handshake test


$ openssl s_client -connect storage.example.com:443 -tls1_3 -servername storage.example.com
CONNECTED(00000003)
...
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    ...

3. Application log after fix


2024-10-12T08:22:10.001Z INFO inference-service: model download started
2024-10-12T08:22:10.123Z INFO inference-service: TLS handshake completed in 45ms
2024-10-12T08:22:10.456Z INFO inference-service: model loaded successfully

4. End‑to‑end functional test

Run a curl request from within the pod to the storage endpoint:


$ curl -f -s https://storage.example.com/models/latest.bin -o /tmp/model.bin && echo "download ok"
download ok

Prevent

Pin OpenSSL version at image build time. Include the required libssl.so.3 and libcrypto.so.3 in the container and set LD_LIBRARY_PATH explicitly.
Avoid host library injection. Do not rely on driver‑installed LD_LIBRARY_PATH for production pods; instead use securityContext.runAsUser with a non‑root user that cannot read host library directories.
Monitor TLS handshake latency. Create a Prometheus alert on http_client_tls_handshake_seconds_bucket{le="5"} == 0 to catch future regressions.
Validate cipher suite compatibility. Add a CI step that runs openssl s_client against all external HTTPS endpoints used by the service.
Document driver version requirements. Keep a matrix of AMDGPU driver releases vs. supported OpenSSL versions and upgrade the driver when a newer OpenSSL is required.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

Why does the TLS handshake succeed on Intel/NVIDIA nodes?
Those nodes do not install the AMDGPU driver, so the system OpenSSL 3.x library is used by default. No older libcrypto.so.1.0.2 is injected, allowing TLS 1.3 negotiation.
Can I keep the AMD driver but still use OpenSSL 3.x?
Yes. Unset LD_LIBRARY_PATH for the pod or mount a custom /usr/lib with the newer libraries. The driver itself does not depend on OpenSSL for user‑space TLS.
Is kernel TLS offload required for the fix?
No. The handshake failure is caused by the user‑space library mismatch. Enabling kernel TLS offload is optional and only improves throughput.
What if the external service only supports TLS 1.2?
OpenSSL 3.x still supports TLS 1.2 and a wide range of cipher suites. After the fix, the client will negotiate the highest mutually supported protocol, which includes TLS 1.2.
How do I detect similar library shadowing issues in other pods?
Add a startup script that runs ldd on the main binary and fails if any library path contains /usr/lib/amdgpu. Emit the result to a readiness probe so the pod does not become ready with a broken TLS stack.