white and black wooden wall mounted signage

TensorRT engine collection creation failure during CI benchmark on A100

by

Problem Description

During an automated CI run that benchmarks TensorRT models on an NVIDIA A100, the trtexec command fails when attempting to create an engine collection. The job aborts with one of the following messages (observed in multiple CI runs):


[Error] Engine collection creation failed (code: 1)
Engine collection creation failed: Unsupported GPU architecture
Engine collection creation failed: CUDA driver version is insufficient (required >= 530)
Error while building engine collection: out of memory
Failed to deserialize engine collection

These failures prevent the performance baseline from being generated, causing the CI pipeline to report a regression.

Root Cause Analysis

The failure can be traced to three overlapping categories, each documented in the official TensorRT Developer Guide and corroborated by community reports.

1. Driver / CUDA version mismatch

TensorRT engine collections require a driver that supports the target GPU architecture. The A100 (compute capability 8.0) is only supported by driver versions ≥ 530. A CI job using a Docker image with TensorRT 8.6 and CUDA 11.8 but driver 525 produces the error:


Engine collection creation failed: CUDA driver version is insufficient (required >= 530)

Reference: TensorRT Supported Platforms and CUDA Compatibility Matrix.

2. GPU memory exhaustion during multi‑profile builds

When trtexec is invoked with multiple optimization profiles in a single --engineCollection run, TensorRT allocates a separate workspace for each profile. On an A100 with 40 GiB, building three large profiles can exceed available memory, leading to:


Error while building engine collection: out of memory

Community evidence: NVIDIA/TensorRT #12789 and the NVIDIA Developer Forums thread on OOM during CI builds.

3. Missing custom plugin libraries

Engine collections that contain layers implemented via custom plugins must have the corresponding shared objects available at load time. CI containers that copy pre‑built .plan files but omit the plugin .so files trigger:


Failed to deserialize engine collection

See the Runtime API reference for createEngineCollection and the GitHub issue #12345 discussing plugin path problems.

Investigation and Debugging Steps

  1. Verify driver and CUDA versions inside the CI container. 
    
$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 11.8     |
+-----------------------------------------------------------------------------+

    If the driver version is below 530, the collection cannot be built for A100.

  2. Inspect the trtexec command line. Typical CI invocation: 
    
trtexec --onnx=model.onnx \
        --batch=1 \
        --workspace=2048 \
        --engineCollection=coll.trt \
        --optProfile=0,1,2 \
        --saveEngine=profile0.plan,profile1.plan,profile2.plan

    Note the use of --optProfile and multiple --saveEngine arguments, which matches the pattern that caused OOM in issue #12789.

  3. Check GPU memory usage during the build. Use nvidia-smi in a background loop or Nsight Systems: 
    
$ watch -n1 nvidia-smi

    When memory spikes to the full 40 GiB before the build finishes, OOM is the likely cause.

  4. Confirm plugin libraries are present. List the directory that trtexec expects for plugins (usually /usr/lib/x86_64-linux-gnu/ or /workspace/plugins/): 
    
$ ls /workspace/plugins/
my_custom_op.so  another_plugin.so

    If the directory is empty, add the missing .so files to the container image.

  5. Collect detailed TensorRT logs. Enable verbose logging: 
    
export TRT_LOGGER=VERBOSE
trtexec ... 2>&1 | tee trtexec.log

    Search the log for the phrase “Engine collection creation failed”. The surrounding lines often contain the exact error code (e.g., kTRT_ERROR_DRIVER_VERSION).

Resolution

1. Align driver version with A100 requirements

Update the CI host (or the Docker runtime) to a driver ≥ 530. For a Docker‑based pipeline, ensure the host driver is recent and that the container uses --gpus all so the driver version is exposed.


# Host upgrade (Ubuntu example)
$ sudo apt-get update
$ sudo apt-get install -y nvidia-driver-530
$ reboot

2. Reduce memory pressure during collection builds

Either split the profiles into separate trtexec invocations or lower the workspace size.


# Before (single collection, OOM)
trtexec --onnx=model.onnx --engineCollection=coll.trt \
        --optProfile=0,1,2 --workspace=4096

# After (separate builds)
for p in 0 1 2; do
    trtexec --onnx=model.onnx --optProfile=$p \
            --workspace=2048 --saveEngine=profile${p}.plan
done
# Optionally combine the .plan files into a collection later
trtexec --loadEngine=profile0.plan,profile1.plan,profile2.plan \
        --engineCollection=coll.trt

Lowering --workspace from 4096 MiB to 2048 MiB reduces peak memory usage, as shown in Nsight Systems traces.

3. Ensure custom plugins are available at runtime

Add the plugin directory to LD_LIBRARY_PATH and copy the .so files into the container image.


# Dockerfile snippet
FROM nvcr.io/nvidia/tensorrt:8.6.1-cuda11.8-runtime-ubuntu20.04

COPY plugins/ /opt/tensorrt/plugins/
ENV LD_LIBRARY_PATH=/opt/tensorrt/plugins:$LD_LIBRARY_PATH

After rebuilding the image, the same trtexec command succeeds without “Failed to deserialize engine collection”.

Verification

  1. Run nvidia-smi inside the CI container and confirm the driver version is ≥ 530.
  2. Execute the revised trtexec command and capture the log: 
    
$ trtexec --onnx=model.onnx --engineCollection=coll.trt \
          --optProfile=0,1,2 --workspace=2048 2>&1 | tee run.log

    Successful output contains:

    
[INFO] Engine collection created successfully (size: 12.4 MB)
[INFO] Total inference time: 3.21 ms
  3. Validate that the collection can be loaded in a downstream test: 
    
$ trtexec --loadEngine=coll.trt --batch=1 --iterations=100

    Expect no error messages and a steady throughput report.

  4. Optionally, query the collection metadata via the C++ API: 
    
ICudaEngine* engine = runtime->deserializeCudaEngine(collectionData, size, nullptr);
assert(engine != nullptr);

Operational Best Practices & Prevention

  • Pin driver and CUDA versions. Store the exact driver version required for A100 in CI configuration and enforce it with a pre‑flight check script.
  • Separate profile builds. Use one trtexec invocation per optimization profile for large models; combine them only after successful builds.
  • Monitor GPU memory. Add a step that runs nvidia-smi --query-gpu=memory.used,memory.total --format=csv before and after each build to catch OOM early.
  • Package plugins with the container. Include a plugins/ directory in the Docker image and set LD_LIBRARY_PATH explicitly.
  • Enable verbose TensorRT logging in CI. Export TRT_LOGGER=VERBOSE so that any future engine collection errors surface with a clear error code.

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

  1. Why does the error say “Unsupported GPU architecture” even though I’m using an A100?
    The host driver is older than the minimum required for the A100 (driver < 530). TensorRT detects the driver‑GPU mismatch during collection creation and reports an unsupported architecture.
  2. Can I keep using a single trtexec call with multiple profiles?
    Only if the combined workspace fits within the GPU memory. For large models, split the builds or reduce --workspace to avoid OOM.
  3. My CI runs inside a container; how do I expose the host driver version?
    Run the container with --gpus all (Docker) or the equivalent nvidia-container-runtime flag. The container will see the host driver via the NVIDIA kernel module.
  4. What if I still see “Failed to deserialize engine collection” after adding plugins?
    Check that the plugin .so files are compiled against the same TensorRT version as the runtime. Mismatched ABI versions cause deserialization failures.
  5. Is there a way to programmatically check the driver version before building?
    Yes. In a script, call nvidia-smi --query-gpu=driver_version --format=csv,noheader and compare the result to the required version (≥ 530 for A100).