AMD GPU Batch Ingestion Failure In Air-gapped Environment

Q: 2. Transfer and Install on the Air‑Gapped Node

# On the target node sudo dpkg -i /mnt/usb/*.deb # Resolve dependencies from the same media sudo apt-get install -f After installation verify:

Q: 4. Configure Runtime Paths System‑wide

# /etc/profile.d/rocm.sh export PATH=$PATH:/opt/rocm/bin:/opt/rocm/hip/bin export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/rocm/lib:/opt/rocm/hip/lib Reload the profile or restart the node to apply.

In this article

Table of Contents

Problem: Batch Ingestion Fails on AMD GPU in an Air‑Gapped Environment

In a secure, air‑gapped compute cluster the nightly AI batch‑ingestion pipeline aborts with errors indicating missing ROCm runtime components. Typical log excerpts look like:


Traceback (most recent call last):
  File "/opt/pipeline/ingest.py", line 42, in 
    import torch
ImportError: libamdhip64.so: cannot open shared object file: No such file or directory


[2026-06-04 02:13:57] ERROR: Failed to initialize ROCm runtime: HSA_STATUS_ERROR_OUT_OF_RESOURCES

Additional symptoms observed on the node:

Command rocm-smi returns “command not found”.
Kernel module amdgpu is loaded, but lsmod | grep amdgpu shows no rocm driver entries.
Package manager reports “Unable to locate package rocm‑dev‑5.4”.

The environment has no internet access, preventing on‑the‑fly download of updated drivers or libraries.

Root Cause Analysis

Why the failure occurs

ROCm (Radeon Open Compute) consists of a kernel driver stack, runtime libraries (e.g., libamdhip64.so), and user‑space utilities (rocm-smi). In an air‑gapped deployment the following assumptions often break:

Version alignment: The batch pipeline was built against ROCm 5.4 (as documented in the ROCm Release Notes), but the offline repository only contains ROCm 5.2 packages. Mismatched versions cause “ROCm version mismatch: required 5.4, found 5.2”.
Offline package completeness: The ROCm Offline Installation Guide requires that the repository include rocm-dkms, rocm-dev, and the hip runtime. Missing rocm-dev leads directly to the ImportError: libamdhip64.so seen in the logs.
Driver‑GPU compatibility: The AMD GPU model (e.g., Instinct MI100) requires a minimum driver version listed in the Driver Compatibility Matrix. An outdated driver bundled in the offline repo cannot bind to the GPU, resulting in “Failed to initialize ROCm runtime: HSA_STATUS_ERROR_OUT_OF_RESOURCES”.
Environment variable propagation: Offline installations often forget to add ROCm library paths (/opt/rocm/lib, /opt/rocm/hip/lib) to LD_LIBRARY_PATH. When the batch script runs under a non‑interactive scheduler, the variables are absent, causing the runtime loader to miss libamdhip64.so.

Debugging and Investigation

Step‑by‑step diagnostic workflow

Verify kernel driver load
```
lsmod | grep amdgpu
dmesg | grep -i amdgpu
```
Expected output includes amdgpu and rocm modules. Absence indicates driver not installed or not compatible with the current kernel.
Check ROCm package versions
```
dpkg -l | grep rocm
# or rpm -qa | grep rocm
```
Compare the listed version against the version required by the application (e.g., 5.4).
Inspect library presence
```
ls /opt/rocm/lib/libamdhip64.so
ldconfig -p | grep libamdhip64
```
If the file is missing, the rocm-dev package was not installed.

Validate environment variables

echo $LD_LIBRARY_PATH
echo $PATH

Typical ROCm‑required values:

LD_LIBRARY_PATH=/opt/rocm/lib:/opt/rocm/hip/lib
PATH=$PATH:/opt/rocm/bin

Run a minimal HIP program

cat > test_hip.c <<'EOF'
#include <hip/hip_runtime.h>
int main() {
    int device;
    hipGetDevice(&device);
    printf("HIP device %d\n", device);
    return 0;
}
EOF
hipcc test_hip.c -o test_hip
./test_hip

Successful output confirms a functional runtime.

Capture system logs
```
journalctl -u rocm-dkms -b
```
Look for errors such as “Failed to load amdgpu kernel module”.

Solution: Offline ROCm Installation and Environment Alignment

1. Prepare a Complete Offline Repository

On a machine with internet access, follow the official offline guide to mirror the required ROCm release:

# On a connected host (Ubuntu 22.04 example)
mkdir -p /tmp/rocm-offline
cd /tmp/rocm-offline
# Choose the exact version required by the pipeline, e.g., 5.4
wget -qO- https://repo.radeon.com/rocm/apt/5.4/rocm.gpg.key | sudo apt-key add -
echo "deb [arch=amd64] https://repo.radeon.com/rocm/apt/5.4/ jammy main" | sudo tee /etc/apt/sources.list.d/rocm.list
sudo apt-get update
apt-get download rocm-dkms rocm-dev rocm-utils rocm-smi hipblas hipfft
# Copy .deb files to a USB stick

Include the kernel‑module package that matches the target kernel version (check uname -r on the air‑gapped node).

2. Transfer and Install on the Air‑Gapped Node

# On the target node
sudo dpkg -i /mnt/usb/*.deb
# Resolve dependencies from the same media
sudo apt-get install -f

After installation verify:

dpkg -l | grep rocm

3. Align Versions with the Application

If the pipeline was built with ROCm 5.4, ensure the offline repo contains the exact same major/minor release. Use the Release Notes to confirm GPU support.

4. Configure Runtime Paths System‑wide

# /etc/profile.d/rocm.sh
export PATH=$PATH:/opt/rocm/bin:/opt/rocm/hip/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/rocm/lib:/opt/rocm/hip/lib

Reload the profile or restart the node to apply.

5. Re‑build or Re‑link the Batch Application (if needed)

If the binary was linked against a different ROCm version, rebuild it against the newly installed libraries:

# Example for a PyTorch‑based pipeline
pip uninstall torch
pip install torch==2.2.0+rocm5.4 -f https://download.pytorch.org/whl/rocm5.4/torch_stable.html

Before / After Comparison

Condition	Before Fix	After Fix
Library availability	`ImportError: libamdhip64.so not found`	Runtime loads `libamdhip64.so` successfully
Utility presence	`rocm-smi: command not found`	`rocm-smi` reports GPU health
Version match	Required 5.4, found 5.2 → batch abort	Both application and system at 5.4 → batch runs
Environment variables	Empty `LD_LIBRARY_PATH`	Includes ROCm library paths system‑wide

Verification: Confirming Successful Batch Ingestion

Run a quick inference test:

python -c "import torch; print(torch.cuda.is_available())"
# Expected output: True

Execute the first stage of the batch pipeline manually and watch logs for ROCm initialization messages:

2026-06-05 01:02:13 INFO: ROCm runtime initialized (version 5.4)
2026-06-05 01:02:13 INFO: Batch job started on GPU 0

Check rocm-smi for GPU health:

# rocm-smi
GPU  Temp   AvgPower  GPUUtil  MemUtil
0    45C    120W      78%      65%

Validate scheduler integration (e.g., Slurm):

scontrol show job <jobid> | grep GPU
   Gres=gpu:1

Prevention and Best Practices

Version pinning: Store the exact ROCm version required by your containers or binaries in a version‑controlled manifest.
Offline repository hygiene: Periodically refresh the mirrored ROCm packages after each upstream release, using the offline guide to avoid stale libraries.
Automated validation: Add a CI step that runs the minimal HIP program on a staging air‑gapped node after any repository update.
Environment consistency: Deploy /etc/profile.d/rocm.sh via configuration management (Ansible, Chef) to guarantee PATH and LD_LIBRARY_PATH are present for all users and scheduler jobs.
Kernel‑driver alignment: Keep a mapping of kernel versions to the corresponding rocm-dkms package; use the Driver Compatibility Matrix as a reference.
Monitoring: Set up alerts on rocm-smi metrics (GPU temperature, utilization) and on systemd unit rocm-dkms.service failures.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

Why does ImportError: libamdhip64.so appear only after a kernel upgrade?
The kernel upgrade may have removed the previously installed rocm-dkms module, causing the driver package to be rebuilt. If the offline repo does not contain the matching rocm-dkms for the new kernel, the runtime libraries are not re‑installed, leading to the missing libamdhip64.so error.
Can I use container images (e.g., Docker) to avoid offline ROCm installation?
Yes, but the host must still provide the kernel driver and the /dev/kfd device. Containers cannot ship the kernel module; you still need a correctly installed ROCm driver on the host.
How do I verify which ROCm version a compiled binary expects?
Run ldd on the binary and look for libamdhip64.so.X.Y. The suffix (e.g., .so.5.4) indicates the required ROCm version. Matching this against dpkg -l | grep rocm confirms compatibility.
What should I do if rocm-smi is present but reports “GPU not found”?
Check that the amdgpu kernel module is loaded and that the PCI device is visible (lspci -nn | grep -i amd). A mismatched driver version or a disabled IOMMU in the BIOS can hide the GPU from ROCm.
Is it safe to disable LD_LIBRARY_PATH and rely on /etc/ld.so.conf.d/rocm.conf?
Yes, after installing ROCm you can add /opt/rocm/lib and /opt/rocm/hip/lib to /etc/ld.so.conf.d/rocm.conf and run ldconfig. This removes the need for per‑session environment variables and avoids scheduler‑related path issues.