NVIDIA GPU Inference Queue Backlog Under High Concurrency Serving Transformers

Q: Problem – Inference Queue Backlog on NVIDIA GPUs Under High Concurrency

During a load test that simulated 10 k RPS of real‑time requests to a BERT‑large transformer served via Triton Inference Server, the following symptoms appeared:

Q: 1. Increase Model Instance Replication

Original config.pbtxt (single instance):

In this article

Table of Contents

Problem – Inference Queue Backlog on NVIDIA GPUs Under High Concurrency

During a load test that simulated 10 k RPS of real‑time requests to a BERT‑large transformer served via Triton Inference Server, the following symptoms appeared:

Latency spikes from ~30 ms (baseline) to >2 s.
HTTP 504 timeout errors from the client library.
Server logs showing a growing request_queue_size metric.
GPU utilization plateauing at ~55 % despite the high request rate.

Sample log excerpt (Triton):

2024-10-17T12:04:23.874Z [I] inference_server.cc:1245] model 'bert_large' queue depth: 128 (max 64)
2024-10-17T12:04:24.001Z [W] request_rate_limiter.cc:212] request dropped: queue full (size=128)
2024-10-17T12:04:24.003Z [I] metrics.cc:312] inference_queue_backlog_seconds 12.3

These symptoms indicate that the GPU is not keeping up with the incoming inference workload, leading to a backlog that eventually triggers timeouts.

Root Cause – Why the Backlog Grows

The backlog originates from a combination of runtime and configuration factors that become visible only under sustained high concurrency:

CUDA Stream Saturation: Triton creates a single CUDA stream per model instance by default. When many concurrent requests are dispatched, kernels are serialized on that stream, preventing the GPU from overlapping execution.
Insufficient Model Instance Replication: The default instance_group of count: 1 limits parallelism to a single inference engine per GPU. Under 10 k RPS the single instance becomes a bottleneck.
Memory Pool Fragmentation: Large transformer models allocate >10 GB of GPU memory per instance. The default cuda_memory_pool_size (1 GB) forces frequent cudaMalloc/cudaFree cycles, causing GPU memory paging and additional kernel launch latency.
CPU‑side Thread Pool Exhaustion: Triton’s request handling thread pool defaults to num_threads = 8. With thousands of concurrent HTTP connections, the pool stalls, queuing requests before they even reach the GPU.
Dynamic Batching Disabled: Without dynamic batching, each request is processed individually, increasing kernel launch overhead and reducing overall throughput.

Collectively, these issues cause the GPU to appear under‑utilized while the server’s request queue expands.

Debug – Systematic Investigation Steps

The following checklist reproduces the diagnostic path used in production:

Collect GPU Utilization and Memory Metrics

# Continuous monitoring
watch -n1 nvidia-smi --query-gpu=utilization.gpu,memory.total,memory.used,temperature.gpu --format=csv

Expected output when the backlog occurs:

timestamp, utilization.gpu [%], memory.total [MiB], memory.used [MiB], temperature.gpu [C]
2024-10-17 12:04:24, 55, 24576, 16384, 68

Inspect Triton Metrics

curl -s http://localhost:8002/metrics | grep inference_queue_backlog_seconds
# Example output
inference_queue_backlog_seconds{model_name="bert_large",model_version="1"} 12.3

Check Request Queue Depth

curl -s http://localhost:8002/v2/models/bert_large/ready | jq .
# Returns false when queue is saturated
{
  "ready_state": false,
  "reason": "Queue depth exceeds max allowed (64)"
}

Trace Kernel Execution

# Use Nsight Systems for a 30‑second capture
nsys profile -t cuda -o trace ./run_load_test.sh
# Look for long gaps between kernel launches on the same stream

In the capture you will see a single stream (ID 0) with back‑to‑back cudaLaunchKernel calls, confirming serialization.

Validate Thread Pool Saturation

journalctl -u tritonserver -f | grep "ThreadPool"
# Example warning
2024-10-17T12:04:25.112Z [W] thread_pool.cc:87] all worker threads busy, request queued

Solution – Configuration and Code Changes to Eliminate the Backlog

Applying the following changes resolves the bottlenecks. Each change is shown before and after the modification.

1. Increase Model Instance Replication

Original config.pbtxt (single instance):

name: "bert_large"
platform: "pytorch_libtorch"
max_batch_size: 8
input [
  {
    name: "input_ids"
    data_type: TYPE_INT32
    dims: [ -1 ]
  }
]
output [
  {
    name: "logits"
    data_type: TYPE_FP32
    dims: [ -1, 30522 ]
  }
]
instance_group [
  {
    kind: KIND_GPU
    count: 1
  }
]

Updated configuration with three parallel instances and explicit CUDA stream allocation:

instance_group [
  {
    kind: KIND_GPU
    count: 3
    gpus: [ 0 ]
    # Assign a distinct stream per instance
    cuda_streams: [ 0, 1, 2 ]
  }
]

2. Enable Dynamic Batching

Before (disabled):

# No dynamic_batching block

After (enabled with 4‑ms max latency):

dynamic_batching {
  preferred_batch_size: [ 8, 16, 32 ]
  max_queue_delay_microseconds: 4000
}

3. Expand CUDA Memory Pool

Default server launch:

tritonserver --model-repository=/models

Adjusted launch with a 8 GB pool (sufficient for three BERT‑large instances):

tritonserver \
  --model-repository=/models \
  --cuda-memory-pool-size=8192

4. Increase CPU Thread Pool Size

Original launch (8 threads):

tritonserver --model-repository=/models --backend-config=pytorch,default-max-batch-size=8

After adding a larger pool (32 threads) and enabling request queueing:

tritonserver \
  --model-repository=/models \
  --backend-config=pytorch,default-max-batch-size=8 \
  --http-thread-count=32 \
  --grpc-thread-count=32 \
  --allow-metrics=true

5. Deploy TensorRT Optimized Engine (Optional)

For further latency reduction, convert the PyTorch model to TensorRT:

# Convert with torch2trt
python -m torch2trt \
  --model bert_large.pt \
  --output bert_large_trt.plan \
  --max-batch-size 32 \
  --fp16

Update config.pbtxt to use the TensorRT backend:

backend: "tensorrt"

Verify – Confirming the Fix

After applying the changes, repeat the load test and observe the following metrics:

# GPU utilization should rise above 90%
watch -n1 nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv
2024-10-17 12:15:03, 92, 16384

# Queue depth metric should stay below the configured max (64)
curl -s http://localhost:8002/metrics | grep inference_queue_backlog_seconds
inference_queue_backlog_seconds{model_name="bert_large",model_version="1"} 0.12

Latency distribution from the client side should flatten:

# Example output from a perf test script
p50: 28ms
p90: 35ms
p99: 48ms
p999: 62ms

No HTTP 504 errors should appear in the logs.

Prevent – Operational Guardrails to Avoid Future Backlogs

Metric‑Based Autoscaling: Create an alert on inference_queue_backlog_seconds > 1.0 and trigger horizontal pod autoscaling (or GPU instance scaling) before latency degrades.
Capacity Planning: Record the maximum sustainable RPS per GPU instance after tuning; keep a safety margin of 20 %.
Periodic Kernel Profiling: Schedule nightly nsys captures on a synthetic load to detect new kernel serialization patterns after model updates.
Configuration Validation CI: Include a step that launches Triton with the target config and asserts that instance_group.count * max_batch_size yields a theoretical throughput > expected peak.
Resource Quotas: Enforce per‑namespace GPU memory quotas to prevent accidental over‑provisioning that could fragment the CUDA pool.

FAQ – Common Follow‑Up Questions

Why does the backlog appear only under peak load and not during steady‑state traffic?
Because the single CUDA stream and limited instance count can handle average request rates, but they cannot absorb the bursty spikes where concurrent kernel launches exceed the stream’s capacity.
Can I keep a single model instance and still achieve high throughput?
Only if you enable multiple CUDA streams per instance (via --cuda-streams) and use TensorRT with fused kernels; otherwise parallel instances are the simplest way to increase concurrency.
How do I choose the right max_batch_size for dynamic batching?
Measure the latency‑throughput curve for your model; pick the smallest batch size that still yields >90 % of the peak throughput while keeping the 99th‑percentile latency within SLA.
What impact does enabling FP16 have on queue depth?
FP16 reduces kernel execution time and memory pressure, which often lowers queue depth, but you must verify that the model’s accuracy remains acceptable.
Is the CUDA memory pool size the only memory‑related setting I need to adjust?
For large transformers you may also need to increase --workspace-size for TensorRT and ensure the OS‑level cudaMallocAsync pool is not exhausted; monitor cudaMalloc failures in dmesg.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub