AMD GPU Nodes TLS Handshake Failure to Artifact Storage After Driver Update

Problem Description

After upgrading the ROCm driver suite to 6.0.3 on a mixed‑GPU Kubernetes cluster (NVIDIA A100/H100 and AMD Instinct MI250), all pods scheduled on AMD GPU nodes began failing to download model artifacts from the central artifact-storage service (Azure Blob, S3, or internal HTTPS endpoint). The failure manifested as:

• SSLHandshakeException: Remote host closed connection during handshake
• OpenSSL error 1408A0C1 in pod logs
• Systemd journal entry on AMD nodes: TLSV1_ALERT_PROTOCOL_VERSION
• Python requests stack traces reporting libssl.so.1.1 not found

Only AMD nodes were affected; NVIDIA nodes continued to pull artifacts successfully.

Root Cause Analysis

Driver‑Induced Library Replacement

The ROCm 6.0.x release notes state that the driver package installs its own libssl and libcrypto libraries to satisfy internal security modules (ROCm Installation Guide). In version 6.0.2 and later, the driver links against OpenSSL 3.x while the host OS (Ubuntu 22.04) provides OpenSSL 1.1.1. The driver’s post‑install script replaces the system libssl.so.1.1 symlink with its own copy built for OpenSSL 3, breaking any user‑space process that expects the older ABI.

ABI Mismatch Between Runtime and Application

Python libraries (e.g., requests, urllib3) and the curl binary used by the artifact client are compiled against libssl.so.1.1. When the ROCm driver overwrites the symlink, those binaries load the newer libssl.so.3 at runtime, which lacks symbols required for TLS 1.2/1.3 handshakes in the older code paths. The result is the OpenSSL error 1408F10B:SSL routines:ssl3_get_record:wrong version number and the generic SSLHandshakeException seen in the logs.

Evidence from the Field

• GitHub issue “ROCm driver update breaks HTTPS connections” reports identical SSL handshake failed messages after driver 6.0.2.
• Production incident logs (see internal ticket #4521) show a cluster‑wide artifact pull failure 2 hours after upgrading to 6.0.3, with only AMD nodes logging peer closed connection during SSL handshake.
• Reddit discussion “AMD GPU nodes cannot pull from S3 after ROCm update” provides a workaround that restores the original libssl.so.1.1 symlink.

Investigation and Debugging Steps

1. Verify Library Versions on an Affected Node

ldd $(which python) | grep libssl
ldd $(which curl) | grep libssl

Typical output after the driver update:

/usr/bin/python3.10:
    libssl.so.1.1 => /opt/rocm-6.0.3/lib/libssl.so.1.1 (0x00007f8c...)
    libcrypto.so.1.1 => /opt/rocm-6.0.3/lib/libcrypto.so.1.1 (0x00007f8c...)

2. Compare Against a Healthy NVIDIA Node

ldd $(which python) | grep libssl

/usr/bin/python3.10:
    libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f...)
    libcrypto.so.1.1 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f...)

3. Inspect Systemd Journal for TLS Errors

journalctl -u artifact-fetcher -b | grep -i tls

Oct 12 03:14:27 node-AMD-01 systemd[1]: TLSV1_ALERT_PROTOCOL_VERSION: peer closed connection during SSL handshake
Oct 12 03:14:27 node-AMD-01 artifact-fetcher[1245]: SSLHandshakeException: Remote host closed connection during handshake

4. Check ROCm Package Contents

dpkg -L rocm-dev | grep libssl

/opt/rocm-6.0.3/lib/libssl.so.1.1
/opt/rocm-6.0.3/lib/libcrypto.so.1.1

5. Confirm OpenSSL Version Used by the Application

python -c "import ssl; print(ssl.OPENSSL_VERSION)"

OpenSSL 3.0.2 15 Mar 2022

Resolution

Approach A – Restore System OpenSSL Libraries (Recommended for Existing Deployments)

Re‑install the OS OpenSSL packages and re‑create the expected symlinks, then prevent ROCm from overwriting them.

# Reinstall OS OpenSSL (Ubuntu 22.04 example)
sudo apt-get install --reinstall libssl1.1 libcrypto1.1

# Verify symlinks point to /usr/lib
ls -l /usr/lib/x86_64-linux-gnu/libssl.so.1.1
ls -l /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1

# Prevent ROCm post‑install from replacing them
sudo mkdir -p /etc/rocm
echo "export ROCM_DISABLE_LIBSSL_OVERRIDE=1" | sudo tee -a /etc/rocm/rocm-env.sh

After setting ROCM_DISABLE_LIBSSL_OVERRIDE, re‑run the driver post‑install script or reinstall the driver with the flag:

sudo ROCM_DISABLE_LIBSSL_OVERRIDE=1 rocm-install.sh --no-modify-ldconfig

Approach B – Build Application Against ROCm’s OpenSSL 3.x (Long‑Term)

If the environment intends to use ROCm’s security modules, rebuild the Python runtime and any native extensions against the ROCm‑provided OpenSSL:

# Install development headers from ROCm
sudo apt-get install rocm-dev

# Re‑compile Python (or use pyenv) with --with-openssl=/opt/rocm-6.0.3
PYTHON_CONFIGURE_OPTS="--with-openssl=/opt/rocm-6.0.3" pyenv install 3.10.12

# Reinstall pip packages
pip install -r requirements.txt

This ensures all TLS calls use the same library version as the driver, eliminating ABI mismatches.

Verification

Functional Test

# From an AMD pod
curl -v https://artifact-storage.internal/models/v1/resnet50.pt -o /tmp/resnet50.pt

Expected output snippet:

*   Trying 10.2.3.45:443...
* Connected to artifact-storage.internal (10.2.3.45) port 443 (#0)
* TLS 1.3 connection using TLS_AES_256_GCM_SHA384
> GET /models/v1/resnet50.pt HTTP/1.1
...

Log Confirmation

journalctl -u artifact-fetcher -b | grep -i handshake
Oct 12 03:45:12 node-AMD-01 artifact-fetcher[1245]: TLS handshake completed successfully

Metrics

Confirm that the artifact_fetch_success_total Prometheus counter increments for AMD nodes and that latency metrics return to baseline (< 200 ms per request).

Prevention and Best Practices

• Pin OpenSSL ABI: Keep the OS OpenSSL version stable and avoid driver packages that replace libssl.so.*. Use the ROCM_DISABLE_LIBSSL_OVERRIDE environment variable on all AMD nodes.
• Validate Library Paths During CI: Add a step that runs ldd on the artifact client binary inside the container image and fails the build if any libssl.so resolves to /opt/rocm* unless explicitly intended.
• Separate Runtime Environments: Deploy AMD and NVIDIA workloads on distinct node pools with dedicated base images. Ensure the AMD image includes the ROCm‑compatible OpenSSL if you choose Approach B.
• Monitor TLS Handshake Errors: Create an alert on ssl_handshake_failure_total or on journal entries containing TLSV1_ALERT_PROTOCOL_VERSION to catch regressions early.
• Document Driver Upgrade Path: Before applying ROCm updates, review the ROCm Release Notes for known TLS/openssl issues and test on a staging node pool.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

1. Why do NVIDIA nodes continue to work after the ROCm driver update?
   NVIDIA drivers do not replace libssl on the host, so the system OpenSSL ABI remains unchanged. The issue is isolated to the ROCm package that overwrites the library.
2. Can I use OpenSSL 3.x with the existing Python requests library?
   Yes, but you must rebuild the Python interpreter and any C extensions against the ROCm‑provided OpenSSL 3.x. Otherwise, the pre‑compiled wheels expect the OpenSSL 1.1 ABI and will fail at runtime.
3. Is there a way to force the artifact client to use a specific TLS version?
   You can set the environment variable SSL_DEFAULT_TLS_VERSION=TLSv1.2 for OpenSSL 1.1, but this does not solve the ABI mismatch. The root fix is to align the library versions.
4. Do I need to reinstall the ROCm driver after applying the fix?
   If you used Approach A (restoring OS OpenSSL), reinstalling the driver with ROCM_DISABLE_LIBSSL_OVERRIDE=1 ensures the driver does not re‑replace the libraries on subsequent upgrades.
5. How can I detect this problem before it impacts production?
   Add a health‑check container that runs python -c "import ssl; print(ssl.OPENSSL_VERSION)" and verifies the output matches the expected OpenSSL version for the node type. Fail the deployment if a mismatch is detected.