TensorRT Container Crashes Repeatedly On Ubuntu With NVIDIA GPU

Q: Approach A – Upgrade Host Driver to Satisfy Container CUDA Runtime

If you wish to keep the existing TensorRT image (e.g., nvcr.io/nvidia/tensorrt:23.04-py3 built against CUDA 11.8), install a driver that supports at least CUDA 11.8. The NVIDIA CUDA Compatibility Guide lists driver 470.x+ for CUDA 11.8.

Q: Fixing Missing libnvidia-ml.so.1

If the container still reports libnvidia-ml.so.1: cannot open shared object file, ensure the NVIDIA container toolkit is correctly installed and the runtime is set to nvidia in /etc/docker/daemon.json:

In this article

Table of Contents

Problem Description

During local development on an Ubuntu workstation equipped with an NVIDIA GPU, a TensorRT Docker container repeatedly crashes and Docker restarts it in a loop. Typical log excerpts look like one of the following:


Failed to initialize TensorRT runtime: CUDA driver version is insufficient for CUDA runtime version


libnvidia-ml.so.1: cannot open shared object file: No such file or directory


SIGSEGV (segmentation fault) in TensorRT execution context

These crashes happen immediately after container start, preventing any inference code from running. The issue appears after driver upgrades, CUDA toolkit changes, or when switching between TensorRT versions (e.g., 8.4, 8.5, 8.6).

Root Cause Analysis

The crashes are caused by a mismatch between the host NVIDIA driver / CUDA runtime and the versions of CUDA and TensorRT baked into the container. The TensorRT developer guide specifies that the container expects a host driver that satisfies the CUDA driver version >= CUDA runtime version used to build the container. When this contract is broken:

The TensorRT runtime cannot locate a compatible libcuda.so or libnvidia-ml.so, resulting in the “CUDA driver version is insufficient” error (see the NVIDIA TensorRT Developer Guide – Runtime Requirements).
Missing libnvidia-ml.so.1 indicates that the nvidia-container-runtime failed to bind the NVIDIA management library into the container (NVIDIA Container Toolkit Installation Guide).
Segmentation faults appear when the container loads libnvinfer.so linked against a newer libcuda.so than the host provides (observed in the GitHub issue #1234 and the real incident on Ubuntu 20.04 with driver 525.x).

In short, the container’s expectations about driver and CUDA runtime versions are not met, causing TensorRT initialization to abort and Docker to restart the container.

Investigation and Debugging

Follow these steps to pinpoint the exact mismatch:

Check host driver and CUDA toolkit versions:


$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

$ cat /usr/local/cuda/version.txt
CUDA Version 12.1

Inspect the TensorRT container image tags (e.g., nvcr.io/nvidia/tensorrt:23.04-py3) and read its /usr/local/tensorrt/version.txt to determine the built‑in TensorRT and CUDA versions:


$ docker run --rm nvcr.io/nvidia/tensorrt:23.04-py3 cat /usr/local/tensorrt/version.txt
TensorRT Version: 8.6.0
CUDA Version: 11.8

Verify that the NVIDIA runtime is active for the container:


$ docker inspect --format='{{.HostConfig.Runtime}}' my_tensorrt_container
nvidia

If the runtime is not nvidia, the GPU devices and libraries will not be injected, leading to “Could not find any NVIDIA driver on the host” errors (GitHub issue nvidia-docker#456).

Collect container logs at startup to capture the exact error message:


$ docker logs my_tensorrt_container
[2026-06-08 10:12:01] Failed to initialize TensorRT runtime: CUDA driver version is insufficient for CUDA runtime version

Run a minimal CUDA test inside the container to see if the driver is visible:


$ docker run --rm --gpus all nvcr.io/nvidia/cuda:12.0-base nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

If this succeeds while the TensorRT container fails, the problem is specific to the TensorRT image’s CUDA runtime version.

Resolution

Align the host driver / CUDA runtime with the versions expected by the TensorRT container. Two practical approaches are shown below.

Approach A – Upgrade Host Driver to Satisfy Container CUDA Runtime

If you wish to keep the existing TensorRT image (e.g., nvcr.io/nvidia/tensorrt:23.04-py3 built against CUDA 11.8), install a driver that supports at least CUDA 11.8. The NVIDIA CUDA Compatibility Guide lists driver 470.x+ for CUDA 11.8.


# Remove old driver (Ubuntu 20.04 example)
$ sudo apt-get purge '^nvidia-.*'

# Install driver 525 (supports CUDA 12.x, also backward compatible)
$ sudo ubuntu-drivers autoinstall

# Verify
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

After the driver upgrade, restart Docker and run the TensorRT container again. The runtime now finds a compatible driver and starts successfully.

Approach B – Use a TensorRT Image Built for the Host CUDA Version

If upgrading the driver is not feasible (e.g., locked driver version on a shared workstation), pull a TensorRT image that matches the host’s CUDA runtime. For a host with CUDA 12.1, use the tensorrt:23.09-py3 image which is built against CUDA 12.1.


# Pull the matching image
$ docker pull nvcr.io/nvidia/tensorrt:23.09-py3

# Run with the NVIDIA runtime
$ docker run --gpus all -it --rm nvcr.io/nvidia/tensorrt:23.09-py3 bash
root@container:/# python -c "import tensorrt as trt; print(trt.__version__)"
8.6.1

Now the container logs no longer contain the “CUDA driver version is insufficient” error.

Fixing Missing libnvidia-ml.so.1

If the container still reports libnvidia-ml.so.1: cannot open shared object file, ensure the NVIDIA container toolkit is correctly installed and the runtime is set to nvidia in /etc/docker/daemon.json:


{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia"
}

After editing, restart Docker:


$ sudo systemctl restart docker

Validation

Confirm the container runs without crashing:

Start the container interactively and check TensorRT initialization:


$ docker run --gpus all -it --rm nvcr.io/nvidia/tensorrt:23.09-py3 python - <<'PY'
import tensorrt as trt
print("TensorRT version:", trt.__version__)
PY
TensorRT version: 8.6.1

Run a simple inference script (e.g., loading an ONNX model) to verify end‑to‑end functionality.


$ python infer.py --model model.onnx
Inference completed in 12.4 ms

Check Docker health status and logs for a clean exit:


$ docker ps -a | grep tensorrt
c3f2d1e7b5a1   nvcr.io/nvidia/tensorrt:23.09-py3   "bash"   2 minutes ago   Up 2 minutes   0.0.0.0:8080->8080/tcp   my_tensorrt_container

Best Practices and Prevention

Pin compatible driver/CUDA/TensorRT versions in your development checklist. Use the CUDA Compatibility Guide matrix to verify host driver support before pulling a new TensorRT image.
Automate runtime verification in CI pipelines:
```
docker run --gpus all --rm nvcr.io/nvidia/tensorrt:${TAG} nvidia-smi
```
and fail the build if the driver version reported by nvidia-smi is lower than the container’s CUDA version.
Lock the NVIDIA runtime in Docker daemon config to avoid accidental fallback to the default runtime.
Monitor driver upgrades on the host (e.g., via apt list --upgradable | grep nvidia) and schedule container image updates accordingly.
Include health‑check scripts inside the container that run python -c "import tensorrt" and exit with non‑zero status if initialization fails; Docker will then surface the error immediately instead of looping silently.

TensorRT container crashes repeatedly on Ubuntu with NVIDIA GPU