Intermittent Qwen Query Latency Spikes During On‑the‑fly Evaluation

Q: Problem – Intermittent Qwen Query Latency Spikes in an On‑the‑Fly Evaluation Loop

During a continuous training run, the Qwen model is invoked for on‑the‑fly evaluation after each training epoch. Most inference calls complete within ~30 ms, but occasional spikes of 300‑800 ms appear, causing the overall training step time to increase by up to 5×. The spikes are non‑deterministic, appear roughly every 5‑15 minutes, and disappear without any restart.

Q: Debug – Step‑by‑Step Investigation 1. Capture Latency Distribution

import time, statistics latencies = [] for i in range(1000): start = time.time() _ = qwen_model.generate(prompt) latencies.append(time.time() - start) print("p95:", statistics.quantiles(latencies, n=100)[94]) print("p99:", statistics.quantiles(latencies, n=100)[98]) Output showed a long tail:

Q: 2. Correlate with System Metrics

# CPU and GPU utilisation (prometheus query) rate(node_cpu_seconds_total{mode="user"}[1m]) gpu_utilization_percent{device="GPU0"} Spikes coincided with CPU utilisation > 95 % and GPU memory usage approaching 95 %.

Q: 3. Inspect Torch Thread‑Pool State

import torch print("Default intra‑op threads:", torch.get_num_threads()) print("Default inter‑op threads:", torch.get_num_interop_threads()) Result:

Q: Solution – Isolate Inference and Adjust Thread Pools 1. Run Inference in a Dedicated Worker Process

Spawn a lightweight multiprocessing worker that loads the model once and receives prompts over a Unix domain socket.

In this article

Table of Contents

Problem – Intermittent Qwen Query Latency Spikes in an On‑the‑Fly Evaluation Loop

During a continuous training run, the Qwen model is invoked for on‑the‑fly evaluation after each training epoch. Most inference calls complete within ~30 ms, but occasional spikes of 300‑800 ms appear, causing the overall training step time to increase by up to 5×. The spikes are non‑deterministic, appear roughly every 5‑15 minutes, and disappear without any restart.

Root Cause – Thread‑Pool Starvation and GPU Context Re‑initialisation

The training process runs on a single GPU and shares the same process space with the inference client. The inference client creates a new torch.cuda stream for each request and relies on the default torch thread pool (size = number of CPU cores). When the training loop is at full GPU utilisation, the inference thread pool becomes saturated, causing the inference request to block until a training thread releases a CPU core. In addition, the Qwen runtime lazily re‑initialises its CUDA context when the GPU memory pressure exceeds a threshold, adding an extra ~200 ms overhead that surfaces as a latency spike.

Key points:

Training and inference share the same process → shared CPU thread pool.
Default torch thread pool size is insufficient for concurrent training + inference workloads.
GPU memory fragmentation triggers a context re‑initialisation path in the Qwen runtime.

Debug – Step‑by‑Step Investigation

1. Capture Latency Distribution

import time, statistics
latencies = []
for i in range(1000):
    start = time.time()
    _ = qwen_model.generate(prompt)
    latencies.append(time.time() - start)

print("p95:", statistics.quantiles(latencies, n=100)[94])
print("p99:", statistics.quantiles(latencies, n=100)[98])

Output showed a long tail:

p95: 0.032
p99: 0.785   # spike

2. Correlate with System Metrics

# CPU and GPU utilisation (prometheus query)
rate(node_cpu_seconds_total{mode="user"}[1m])
gpu_utilization_percent{device="GPU0"}

Spikes coincided with CPU utilisation > 95 % and GPU memory usage approaching 95 %.

3. Inspect Torch Thread‑Pool State

import torch
print("Default intra‑op threads:", torch.get_num_threads())
print("Default inter‑op threads:", torch.get_num_interop_threads())

Result:

Default intra‑op threads: 32
Default inter‑op threads: 8

4. Detect CUDA Context Re‑initialisation

Qwen writes a debug line when it re‑creates the CUDA context:

2026-10-28 14:12:03,421 [QwenRuntime] INFO  Re‑initialising CUDA context due to memory pressure

These log entries appeared exactly before the latency spikes.

5. Reproduce in Isolation

Running inference in a separate process eliminated the spikes, confirming the contention is intra‑process.

Solution – Isolate Inference and Adjust Thread Pools

1. Run Inference in a Dedicated Worker Process

Spawn a lightweight multiprocessing worker that loads the model once and receives prompts over a Unix domain socket.

Before (in‑process inference):

def evaluate(prompt):
    return qwen_model.generate(prompt)   # executed in training process

After (out‑of‑process inference):

# inference_worker.py
import torch, socket, json, os
from qwen import QwenModel

model = QwenModel.from_pretrained("Qwen-7B")
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
sock.bind("/tmp/qwen.sock")
sock.listen()

while True:
    conn, _ = sock.accept()
    data = conn.recv(4096)
    payload = json.loads(data)
    response = model.generate(payload["prompt"])
    conn.sendall(json.dumps({"output": response}).encode())
    conn.close()

# training_loop.py (snippet)
import socket, json, time

def evaluate(prompt):
    client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    client.connect("/tmp/qwen.sock")
    client.sendall(json.dumps({"prompt": prompt}).encode())
    resp = client.recv(8192)
    client.close()
    return json.loads(resp)["output"]

2. Increase Torch Thread‑Pool Sizes for Training Only

Set larger intra‑op threads for the training process and keep inference worker at default.

# training entrypoint
import torch
torch.set_num_threads(64)          # training CPU parallelism
torch.set_num_interop_threads(16)

3. Pre‑warm the CUDA Context

Execute a dummy forward pass once at process start to allocate the maximum required memory and avoid lazy re‑initialisation.

# inference_worker.py (add after model load)
dummy_input = torch.randn(1, 1024, device="cuda")
with torch.no_grad():
    _ = model(dummy_input)

4. Enable CUDA Memory Pooling

Set the PYTORCH_CUDA_ALLOC_CONF environment variable to keep a stable memory pool.

# .env for inference worker
PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.8,max_split_size_mb:128

Verify – Confirm the Latency Spike Is Resolved

1. Re‑run Latency Test

latencies = []
for i in range(2000):
    start = time.time()
    _ = evaluate("What is the capital of France?")
    latencies.append(time.time() - start)

print("p95:", statistics.quantiles(latencies, n=100)[94])
print("p99:", statistics.quantiles(latencies, n=100)[98])

Expected output after fix:

p95: 0.031
p99: 0.045   # no spike

2. Monitor Logs for Context Re‑initialisation

No longer see the “Re‑initialising CUDA context” line.

3. Observe System Metrics

CPU utilisation stays < 80 % for training threads.
GPU memory usage stabilises at ~85 % without sudden jumps.
Inference worker CPU usage stays < 10 %.

Prevent – Operational Guardrails for Future Deployments

Process Isolation: Keep long‑running inference services in separate containers or processes from training jobs.
Thread‑Pool Configuration: Explicitly set torch.set_num_threads and torch.set_num_interop_threads per service rather than relying on defaults.
CUDA Warm‑up: Perform a dummy forward pass during service start‑up to allocate the full memory pool.
Memory Pool Settings: Use PYTORCH_CUDA_ALLOC_CONF to avoid fragmentation‑driven context resets.
Tail‑Latency Monitoring: Alert on p99 inference latency > 100 ms over a 5‑minute window.
Resource Quotas: Enforce cgroup CPU limits for training jobs to leave headroom for inference threads.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why do latency spikes disappear after a few minutes without any change?
The spikes are caused by occasional CUDA context re‑initialisation triggered by memory pressure. Once the context is rebuilt, the memory layout stabilises until the next pressure event.
Can increasing the GPU memory solve the problem?
More memory reduces the frequency of context re‑initialisation but does not address the underlying thread‑pool contention. Process isolation is a more reliable fix.
Is it safe to set torch.set_num_threads to a very high value?
Only for the process that performs the heavy computation (training). Over‑allocating threads can lead to OS‑level CPU thrashing; benchmark to find the sweet spot (usually 2‑4 × physical cores).
Do I need to restart the training job after applying the fixes?
Yes. The CUDA context must be recreated with the new memory pool settings, and the inference worker must be launched as a separate process.
How can I detect similar latency issues for other models?
Instrument each inference call with timestamps, log any “Re‑initialising CUDA context” messages, and monitor p99 latency. Correlate spikes with CPU/GPU utilisation to spot contention patterns.