black and silver sony cassette player

Inference queue backlog on NVIDIA GPU after model reload in local dev

by

Problem – Inference Queue Backlog on NVIDIA GPU After Model Reload (Local Development)

During local development of a deep‑learning service that uses NVIDIA GPUs (e.g., Triton Inference Server, PyTorch TorchScript, or TensorFlow Serving), engineers observed a sudden increase in pending inference requests after hot‑reloading a model. The symptoms include:

  • GPU utilization drops to 0 % while nvidia-smi shows many active processes.
  • Server logs repeatedly emit:

[2026-08-31 14:22:13] [I] tritonserver.cc:1234] Inference request queue full (size=256, max=256)
[2026-08-31 14:22:13] [E] cuda_runtime.cc:567] CUDA_ERROR_LAUNCH_TIMEOUT: kernel launch timed out
  • Latency spikes from ~5 ms to >1 s for new requests.
  • CPU threads block on cudaStreamSynchronize calls that never return.

These symptoms prevent timely processing of inference requests, effectively stalling the development workflow.

Root Cause – How the GPU Queue Becomes Saturated After a Reload

Hot‑reloading a model on a GPU does not automatically flush the CUDA streams that were used by the previous model version. The following mechanisms interact to create the backlog:

  1. CUDA Stream Queue Depth – Each GPU maintains a finite number of pending kernel launches per stream (see CUDA Stream and Event Management). When a model is reloaded, the server typically creates new streams for the new engine while the old streams may still contain kernels that have not completed.
  2. Missing Synchronization – If the reload code omits cudaStreamSynchronize() or cudaDeviceSynchronize(), pending kernels from the previous version remain in the stream queue. The queue depth quickly reaches its limit, causing new kernels to be queued but never launched.
  3. Driver Watchdog & Safe‑Mode – The NVIDIA driver watchdog (documented in the GPU Driver Release Notes) aborts kernels that exceed a timeout, putting the GPU into a safe‑mode state. Subsequent launches are queued but the watchdog prevents execution, resulting in the “CUDA_ERROR_LAUNCH_TIMEOUT” error.
  4. Memory Fragmentation – Concurrent reloads can fragment GPU memory (see the real incident “Concurrent model reloads in local dev triggered memory fragmentation”). Fragmented memory leads to allocation failures for stream resources, which surface as “CUDA out of memory while allocating stream resources”.
  5. Server‑Level Queue Limits – Triton’s per‑GPU request queue defaults to 256 entries (GPU Queue Management). After a reload, the backlog can instantly fill this limit, causing the server to reject new requests with “Inference request queue full”.

In summary, the backlog is caused by a combination of stale kernels occupying CUDA streams, driver watchdog interruptions, and server‑level queue saturation.

Debug – Systematic Investigation Steps

1. Capture Server and Driver Logs


$ tail -f /var/log/tritonserver.log
[2026-08-31 14:20:45] [I] model_repository.cc:210] Loading model 'resnet50' version 2
[2026-08-31 14:20:47] [I] model_repository.cc:215] Model 'resnet50' loaded successfully
[2026-08-31 14:21:12] [I] model_repository.cc:210] Unloading model 'resnet50' version 1
[2026-08-31 14:21:13] [E] cuda_runtime.cc:567] CUDA_ERROR_LAUNCH_TIMEOUT: kernel launch timed out
[2026-08-31 14:22:13] [I] tritonserver.cc:1234] Inference request queue full (size=256, max=256)

2. Inspect CUDA Stream Status with Nsight Systems

Run a short capture around the reload event:


$ nsight-systems-cli -c cuda -t 30s -o reload_capture.nsys-rep

In the timeline, look for a burst of kernels marked “Pending” that never transition to “Running”.

3. Verify Stream Queue Depth via CUDA API


#include <cuda_runtime.h>
int pending = 0;
cudaError_t err = cudaStreamQuery(my_stream);
if (err == cudaErrorNotReady) {
    // Stream still has pending work
    pending++;
}
printf("Pending kernels on stream: %d\n", pending);

Running this snippet before and after reload highlights the increase in pending kernels.

4. Check Driver Watchdog Activity


$ dmesg | grep -i "NVRM"
[  1234.567890] NVRM: Xid (PCI:0000:65:00): 79, pid=1234, name=tritonserver
[  1234.568001] NVRM: Xid (PCI:0000:65:00): 79, pid=1234, name=tritonserver

Error code 79 corresponds to a watchdog timeout (see CUDA_ERROR_LAUNCH_TIMEOUT).

5. Confirm Server Queue Size


$ curl -v http://localhost:8000/v2/models/resnet50/ready
# No response, request hangs → queue is saturated

Solution – Eliminating the Backlog

1. Explicitly Synchronize and Destroy Old Streams

Modify the model reload routine to wait for all pending work and release resources before creating new streams.

Before:

bool ReloadModel(const std::string& path) {
    // Load new engine
    auto engine = LoadEngine(path);
    // Replace pointer without sync
    current_engine_ = engine;
    return true;
}

After:

bool ReloadModel(const std::string& path) {
    // Ensure all previous kernels have finished
    for (auto& stream : old_streams_) {
        cudaError_t err = cudaStreamSynchronize(stream);
        if (err != cudaSuccess) {
            LOG_ERROR("Stream sync failed: %s", cudaGetErrorString(err));
        }
        cudaStreamDestroy(stream);
    }
    old_streams_.clear();

    // Load new engine and create fresh streams
    auto engine = LoadEngine(path);
    current_engine_ = engine;
    CreateStreamsForEngine(engine);
    return true;
}

This guarantees that no kernels remain queued when the new model starts accepting requests.

2. Reset the GPU Between Reloads (Development Only)

For local development, a quick nvidia-smi --gpu-reset clears all pending work and stream state.


$ sudo nvidia-smi --gpu-reset -i 0
GPU reset successful

Note: This command is not suitable for production environments.

3. Increase Triton Queue Limits Temporarily

If the workload legitimately spikes during reload, raise the queue size in model_config.pbtxt:


max_batch_size: 32
instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
]
dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
}
# Add explicit queue size
backend_parameter: {
  key: "gpu_queue_depth"
  value: "512"
}

After the reload, revert to the default to avoid unbounded memory growth.

4. Upgrade Driver to a Version Without the Known Queue‑Backlog Bug

The driver release notes (Known Issues with CUDA Kernel Queue Backlog) indicate that version 525.89.02 resolves a watchdog‑related queue stall. Verify the driver version:


$ nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
+-----------------------------------------------------------------------------+

Verify – Confirming the Issue Is Resolved

  1. Run a Load Test – Issue 200 concurrent inference requests before and after a reload.
  2. Check Queue Depth – Triton exposes /v2/metrics; look for triton_inference_queue_size staying below the max.
  3. Observe Latency – Latency should remain within the pre‑reload baseline (e.g., <5 ms).
  4. Confirm No Watchdog Errors – dmesg should not contain “Xid (PCI…) 79”.

Sample verification output:


$ curl http://localhost:8002/metrics | grep triton_inference_queue_size
triton_inference_queue_size{model_name="resnet50",model_version="2"} 12

The queue size of 12 is well below the default max of 256, indicating healthy operation.

Prevent – Best Practices for Ongoing Development

  • Always synchronize streams before destroying or reusing them. Wrap reload logic in a helper that enforces cudaStreamSynchronize and error checking.
  • Use a dedicated “reload” CUDA stream. Isolate hot‑reload work from inference streams to avoid cross‑contamination.
  • Monitor driver watchdog events. Set up a systemd service that watches dmesg for Xid 79 and alerts the team.
  • Limit concurrent reloads. Serialize model updates in local dev; use a lock file or CI gate.
  • Pin a stable driver version. Avoid automatic driver upgrades that may re‑introduce the queue‑backlog bug.
  • Enable Triton’s “model_control_mode = explicit”. This forces the server to pause request handling while a model is being loaded/unloaded.

FAQ – Common Follow‑Up Questions

  1. Why does the backlog only appear after a hot reload and not on a cold start?
    Because existing CUDA streams retain pending kernels from the previous model version. A cold start begins with fresh streams, so the queue is empty.
  2. Can I avoid the watchdog timeout without disabling the driver watchdog?
    Yes. Keep kernel execution times below the watchdog threshold (≈2 s on most consumer GPUs) by ensuring that model loading does not launch long‑running kernels, and always synchronize before reloading.
  3. Is increasing gpu_queue_depth a safe long‑term solution?
    It mitigates symptom visibility but does not address the root cause. Excessive queue depth can exhaust GPU memory and hide underlying synchronization bugs.
  4. How do I know which CUDA stream is still busy after a reload?
    Use Nsight Systems or the CUDA API cudaStreamQuery on each stream handle. Streams returning cudaErrorNotReady still have pending work.
  5. What if I cannot modify the server code (e.g., using a pre‑built Triton image)?
    Wrap the reload command with nvidia-smi --gpu-reset or use CUDA_LAUNCH_BLOCKING=1 to force synchronous launches, then restart the server to clear stale streams.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub