Docker Inference Worker Queue Backlog Causing Latency Spikes In Streaming

Q: Observed Symptoms Average end‑to‑end latency jumps from ~45 ms to >200 ms during spikes. Prometheus metric inference_queue_length climbs past 500 items (target < 50). Container logs contain repeated warnings: [2026-09-28T14:32:11.842Z] WARN inference_worker: queue backlog = 342 (max 300) [2026-09-28T14:32:12.001Z] WARN inference_worker: processing latency 178ms (budget 50ms) Occasional OOMKilled events in docker ps -a output. Downstream downstream consumer (e.g., WebRTC) reports “frame dropped” warnings. Root Cause: Mismatch Between Ingestion Rate, Worker Concurrency, and Resource Limits

The inference workers use a single‑process Python multiprocessing.Queue to buffer frames. The queue is bounded at maxsize=300. When the upstream frame producer (ffmpeg + Kafka) bursts to >10 k frames/s, the queue fills faster than the GPU‑accelerated model can consume frames (≈ 4 k fps per GPU). The container is limited to cpu_quota=200000 (2 vCPU) and memory=2GiB. Under load the worker process spawns additional Python threads for pre‑processing, but the cgroup CPU limit throttles them, causing the effective inference throughput to drop to ~2 k fps.

Q: Debug: Investigation Steps 1. Verify Queue Metrics

# Prometheus query inference_queue_length{job="inference_worker"} > 250 2. Inspect Container Resource Usage docker stats $(docker ps -q --filter "name=worker") # Expected: CPU ~ 100% (throttled), MEM ~ 1.8GiB 3. Check cgroup throttling counters docker exec -it worker_1 cat /sys/fs/cgroup/cpu/docker/$(docker inspect -f '{{.Id}}' worker_1)/cpu.stat # Sample output nr_periods 12000 nr_throttled 8450 throttled_time 3420000000 4. Profile Inference Throughput # Inside container python -m cProfile -s time inference_worker.py # Look for time spent in pre_process() vs model.infer() 5. Reproduce Burst Scenario Locally # Generate a burst of 12k frames/s for 30 seconds ffmpeg -re -i sample.mp4 -f rawvideo -pix_fmt rgb24 - | \ pv -L 12000000 | \ docker exec -i worker_1 python ingest.py 6. Examine Docker Run Configuration docker inspect worker_1 | jq '.[0].HostConfig' # Look for CpuQuota, Memory, DeviceRequests (GPU) Solution: Align Concurrency, Queue Size, and Resource Allocation 1. Increase Worker Concurrency and Enable Dynamic Scaling Modify the worker entrypoint to read WORKER_CONCURRENCY and spawn that many inference subprocesses, each bound to a dedicated GPU stream if available.

Q: 2. Raise Queue Capacity and Apply Back‑Pressure

Set maxsize=1000 and implement a drop‑policy when the queue exceeds 900 items, logging the event.

Q: 4. Enable Autoscaling of Worker Pods (K8s example)

# HorizontalPodAutoscaler apiVersion: autoscaling/v2beta2 kind: HorizontalPodAutoscaler metadata: name: inference-worker-hpa spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: inference-worker minReplicas: 2 maxReplicas: 10 metrics: - type: Pods pods: metric: name: inference_queue_length target: type: AverageValue averageValue: "200" Verify: Confirming the Fix Queue Length Stabilization # After deployment, watch the metric for 5 minutes inference_queue_length{job="inference_worker"} < 250 Latency Return to Budget # Prometheus query histogram_quantile(0.95, sum(rate(inference_latency_seconds_bucket[1m])) by (le)) # Expected 95th percentile ≈ 0.045s No OOMKilled Events docker ps -a --filter "status=exited" --filter "name=worker" # Should show zero entries with "OOMKilled" GPU Utilization nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 5 # Utilization ~ 85‑95%, memory within limits End‑to‑End Frame Delivery # WebRTC stats (via chrome://webrtc-internals) framesSent: 2998 / framesDropped: 2 # Drop rate 30 s. CPU Throttling Alert: Monitor cpu.throttled_time from cgroup; alert if > 20 % of total time. Autoscaling Thresholds: Keep HPA target at 150‑200 items to maintain headroom. Load‑Test Before Release: Use locust or custom frame burst generator to validate queue behavior under peak expected load. Graceful Degradation: Implement a fallback path that streams lower‑resolution frames when the queue exceeds 950 items, reducing processing cost. Related Topic Hub: Distributed Systems Troubleshooting Hub

In this article

Table of Contents

Problem: Inference Queue Backlog Causing Latency Spikes in Real‑Time Streaming

A high‑throughput video streaming service runs a fleet of Docker containers, each hosting an AI inference worker that processes incoming frames. Over the past weeks the pipeline has exhibited intermittent latency spikes and dropped frames. Monitoring shows a growing backlog in the internal inference queue, eventually exhausting the container’s CPU and memory resources.

Observed Symptoms

Average end‑to‑end latency jumps from ~45 ms to >200 ms during spikes.
Prometheus metric inference_queue_length climbs past 500 items (target < 50).
Container logs contain repeated warnings:


[2026-09-28T14:32:11.842Z] WARN inference_worker: queue backlog = 342 (max 300)
[2026-09-28T14:32:12.001Z] WARN inference_worker: processing latency 178ms (budget 50ms)

Occasional OOMKilled events in docker ps -a output.
Downstream downstream consumer (e.g., WebRTC) reports “frame dropped” warnings.

Root Cause: Mismatch Between Ingestion Rate, Worker Concurrency, and Resource Limits

The inference workers use a single‑process Python multiprocessing.Queue to buffer frames. The queue is bounded at maxsize=300. When the upstream frame producer (ffmpeg + Kafka) bursts to >10 k frames/s, the queue fills faster than the GPU‑accelerated model can consume frames (≈ 4 k fps per GPU). The container is limited to cpu_quota=200000 (2 vCPU) and memory=2GiB. Under load the worker process spawns additional Python threads for pre‑processing, but the cgroup CPU limit throttles them, causing the effective inference throughput to drop to ~2 k fps.

Two interacting factors amplify the problem:

Static concurrency configuration: The worker_concurrency environment variable is set to 1. The code does not auto‑scale based on GPU availability.
Insufficient cgroup limits: The Docker run command caps CPU at 2 vCPU, which is below the requirement for the pre‑processing pipeline (≈ 3 vCPU per worker) when handling burst traffic.

The combination leads to a queue that never drains, causing latency spikes and eventual OOM due to memory growth from queued frames.

Debug: Investigation Steps

1. Verify Queue Metrics


# Prometheus query
inference_queue_length{job="inference_worker"} > 250

2. Inspect Container Resource Usage


docker stats $(docker ps -q --filter "name=worker")
# Expected: CPU ~ 100% (throttled), MEM ~ 1.8GiB

3. Check cgroup throttling counters


docker exec -it worker_1 cat /sys/fs/cgroup/cpu/docker/$(docker inspect -f '{{.Id}}' worker_1)/cpu.stat
# Sample output
nr_periods 12000
nr_throttled 8450
throttled_time 3420000000

4. Profile Inference Throughput


# Inside container
python -m cProfile -s time inference_worker.py
# Look for time spent in pre_process() vs model.infer()

5. Reproduce Burst Scenario Locally


# Generate a burst of 12k frames/s for 30 seconds
ffmpeg -re -i sample.mp4 -f rawvideo -pix_fmt rgb24 - | \
pv -L 12000000 | \
docker exec -i worker_1 python ingest.py

6. Examine Docker Run Configuration


docker inspect worker_1 | jq '.[0].HostConfig'
# Look for CpuQuota, Memory, DeviceRequests (GPU)

Solution: Align Concurrency, Queue Size, and Resource Allocation

1. Increase Worker Concurrency and Enable Dynamic Scaling

Modify the worker entrypoint to read WORKER_CONCURRENCY and spawn that many inference subprocesses, each bound to a dedicated GPU stream if available.

# before (single process)
worker_concurrency = int(os.getenv("WORKER_CONCURRENCY", "1"))
for _ in range(worker_concurrency):
    Process(target=run_worker).start()

# after (dynamic scaling, GPU‑aware)
import torch

def available_gpus():
    return [i for i in range(torch.cuda.device_count())]

gpus = available_gpus()
worker_concurrency = max(len(gpus), int(os.getenv("WORKER_CONCURRENCY", "1")))

for gpu_id in gpus:
    Process(target=run_worker, args=(gpu_id,)).start()

2. Raise Queue Capacity and Apply Back‑Pressure

Set maxsize=1000 and implement a drop‑policy when the queue exceeds 900 items, logging the event.

# before
frame_queue = Queue(maxsize=300)

# after
MAX_QUEUE = 1000
DROP_THRESHOLD = 900
frame_queue = Queue(maxsize=MAX_QUEUE)

def enqueue(frame):
    if frame_queue.qsize() >= DROP_THRESHOLD:
        logger.warning("queue near capacity, dropping frame")
        return False
    frame_queue.put(frame, block=False)
    return True

3. Adjust Docker Resource Limits

Update the deployment manifest (Docker Compose or Kubernetes) to allocate sufficient CPU and memory, and request GPU devices.

# docker-compose.yml (excerpt)
services:
  inference_worker:
    image: myorg/inference:latest
    deploy:
      resources:
        limits:
          cpus: '4.0'
          memory: 4G
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - WORKER_CONCURRENCY=4
      - QUEUE_MAXSIZE=1000

4. Enable Autoscaling of Worker Pods (K8s example)

# HorizontalPodAutoscaler
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: inference-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inference-worker
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Pods
    pods:
      metric:
        name: inference_queue_length
      target:
        type: AverageValue
        averageValue: "200"

Verify: Confirming the Fix

Queue Length Stabilization


# After deployment, watch the metric for 5 minutes
inference_queue_length{job="inference_worker"} < 250

Latency Return to Budget


# Prometheus query
histogram_quantile(0.95, sum(rate(inference_latency_seconds_bucket[1m])) by (le))
# Expected 95th percentile ≈ 0.045s

No OOMKilled Events


docker ps -a --filter "status=exited" --filter "name=worker"
# Should show zero entries with "OOMKilled"

GPU Utilization


nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv -l 5
# Utilization ~ 85‑95%, memory within limits

End‑to‑End Frame Delivery


# WebRTC stats (via chrome://webrtc-internals)
framesSent: 2998 / framesDropped: 2
# Drop rate < 0.1%

Prevent: Operational Guardrails and Monitoring

Alert on Queue Length: Fire an alert when inference_queue_length > 800 for >30 s.
CPU Throttling Alert: Monitor cpu.throttled_time from cgroup; alert if > 20 % of total time.
Autoscaling Thresholds: Keep HPA target at 150‑200 items to maintain headroom.
Load‑Test Before Release: Use locust or custom frame burst generator to validate queue behavior under peak expected load.
Graceful Degradation: Implement a fallback path that streams lower‑resolution frames when the queue exceeds 950 items, reducing processing cost.

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

Why does increasing the queue size alone not solve the latency spikes?
Because the root issue is insufficient processing capacity. A larger queue merely buffers more frames, extending the time each frame spends waiting, which worsens latency.
Can I keep a single‑process worker and still meet the latency budget?
Only if the hardware can sustain the required throughput. On a single GPU the maximum sustainable fps is limited by model inference time; scaling concurrency or adding GPUs is required for >5 k fps workloads.
How do I know if CPU throttling is the bottleneck versus GPU saturation?
Check nvidia-smi GPU utilization. If GPU is < 70 % while cpu.throttled_time is high, the CPU is the bottleneck. Conversely, GPU at 100 % with low CPU usage indicates the model inference itself is the limit.
What is the recommended drop‑policy for real‑time streaming?
Drop the newest frames when the queue exceeds a high watermark (e.g., 90 % of maxsize) to keep downstream latency low. Log the drop count for observability.
Should I use a message broker (Kafka, RabbitMQ) instead of an in‑process queue?
For cross‑host scaling, a broker provides durability and back‑pressure across the fleet. However, it adds latency; for intra‑host buffering the lightweight multiprocessing.Queue with proper sizing is sufficient.