DeepSeek streaming response cuts off during concurrent multi‑GPU training

Problem: DeepSeek streaming response cuts off during concurrent multi‑GPU training

In a production node equipped with 4 × NVIDIA A100 GPUs, DeepSeek is used for two workloads:

• Fine‑tuning or continued pre‑training (torch.distributed, NCCL)
• Real‑time chat inference via the DeepSeek streaming API (stream=True)

When a training job runs on any GPU while inference requests are served, the token‑by‑token stream terminates early. Typical symptoms include:

2024-09-20 14:32:11,842 | INFO | inference_worker.py:112 - Streaming response started for request 7f3c...
2024-09-20 14:32:13,017 | ERROR | inference_worker.py:158 - GenerationCancelledError: Generation stopped due to resource contention
2024-09-20 14:32:13,018 | INFO | grpc_server.py:87 - Stream closed unexpectedly for client 10.2.5.12:54321

Clients receive a truncated JSON payload, e.g.:

{
  "id": "msg-12345",
  "choices": [
    {"index":0,"delta":{"content":"The quick brown"}}
  ],
  "finish_reason": "stop"
}

The stream stops after ~2 seconds regardless of prompt length, causing a poor user experience and breaking downstream pipelines.

Root Cause Analysis

Interaction between training NCCL traffic and inference CUDA streams

The DeepSeek official streaming documentation states that token generation runs on a dedicated CUDA stream with torch.cuda.Stream(priority=0). During multi‑GPU training, the NCCL backend creates high‑priority collective streams (e.g., nccl:all_reduce) that compete for the same GPU SM resources.

Evidence from the DeepSeek GPU Memory Management FAQ shows that when torch.distributed is launched with the default NCCL --gpus-per-node setting, NCCL may saturate the PCIe/NVLink bandwidth and pre‑empt lower‑priority streams. This priority inversion leads to:

• CUDA stream starvation for the inference worker
• Premature termination of the token generation kernel
• Propagation of GenerationCancelledError from the vLLM scheduler (see vLLM issue #487)

Additional contributing factors observed in real incidents:

Why the issue appears only under concurrent load

When the node is idle, the inference CUDA stream runs uninterrupted, matching the behavior described in the DeepSeek streaming guide. As soon as a training process launches a collective operation (e.g., torch.distributed.all_reduce), NCCL injects kernels into the same SM queue. Because NCCL streams default to priority=-1 (higher than the inference stream), the scheduler aborts the lower‑priority token generation after a timeout, raising GenerationCancelledError.

Investigation and Debugging Steps

1. Reproduce the failure in a controlled environment

# Launch a 4‑GPU training job
torchrun --nnodes=1 --nproc_per_node=4 \
    --master_port=29500 \
    train.py --model deepseek-7b --batch-size 32

# In a separate terminal, start the inference server
python -m deepseek.server --model deepseek-7b --stream True

Send a streaming request with curl:

curl -N -X POST http://localhost:8000/generate \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Explain quantum entanglement","stream":true}'

Observe the truncated output and the error log shown earlier.

2. Verify CUDA stream priorities

import torch
for i in range(torch.cuda.device_count()):
    default_stream = torch.cuda.default_stream(i)
    print(f"GPU{i} default stream priority: {default_stream.priority()}")

Typical output on A100:

GPU0 default stream priority: 0
GPU1 default stream priority: 0
GPU2 default stream priority: 0
GPU3 default stream priority: 0

Now inspect NCCL streams (requires nvprof or nsight):

nvprof --profile-from-start off -o trace.nvvp \
    python -c "import torch.distributed as dist; dist.init_process_group('nccl');"

In the trace, NCCL kernels appear with priority -1, confirming higher priority.

3. Check GPU memory fragmentation

nvidia-smi --query-gpu=memory.used,memory.free,memory.total --format=csv

If free memory drops below the inference buffer size (≈2 GB for DeepSeek‑7B), the server may hit RuntimeError: Unexpected EOF while reading from stream.

4. Examine NCCL bandwidth utilization

nccl-tests/build/all_reduce_perf -b 8 -e 8M -f 2 -g 4

Peak utilization > 90 % indicates contention that can starve other streams.

Solution: Isolate inference from training resources

Approach A – Separate process groups and CUDA devices

Assign inference to a dedicated GPU (or subset) that is excluded from the training torch.distributed launch.

# training launch (GPUs 0‑2)
torchrun --nnodes=1 --nproc_per_node=3 \
    --master_port=29500 \
    train.py --model deepseek-7b --gpus 0,1,2

# inference server (GPU 3 only)
CUDA_VISIBLE_DEVICES=3 python -m deepseek.server \
    --model deepseek-7b --stream True

Result: No stream cancellation, because NCCL traffic never reaches GPU 3.

Approach B – Lower NCCL stream priority

Set the NCCL environment variable NCCL_LAUNCH_MODE=GROUP and explicitly lower its priority so inference streams retain higher priority.

export NCCL_LAUNCH_MODE=GROUP
export NCCL_STREAM_PRIORITY=-2   # lower than inference (0)
torchrun ... train.py

After the change, the same trace shows NCCL kernels with priority -2, allowing the inference stream to pre‑empt them when needed.

Approach C – Use separate CUDA streams for inference with higher priority

Patch the DeepSeek server to create a high‑priority stream:

# deepseek/server/inference.py
import torch

# Existing:
# inference_stream = torch.cuda.Stream()

# Updated:
inference_stream = torch.cuda.Stream(priority=-1)  # higher than NCCL default
torch.cuda.set_stream(inference_stream)

Rebuild and redeploy the server. The higher priority prevents NCCL from pre‑empting token generation.

Approach D – Throttle training communication

Insert a torch.distributed.barrier() only after a batch of inference requests has been served, or use torch.distributed.reduce_scatter with async_op=True to avoid blocking the GPU SM for long periods.

Before / After Comparison

# Before (shared GPUs)
torchrun ... train.py
CUDA_VISIBLE_DEVICES=0,1,2,3 python -m deepseek.server --stream True

# After (inference on GPU3 only)
torchrun ... train.py --gpus 0,1,2
CUDA_VISIBLE_DEVICES=3 python -m deepseek.server --stream True

Verification

Functional test

curl -N -X POST http://localhost:8000/generate \
    -H "Content-Type: application/json" \
    -d '{"prompt":"Summarize the plot of Inception","stream":true}' | head -n 20

Expected output: continuous token flow until the model emits " or stop token, with no premature termination.

Metrics validation

• Monitor inference_stream_latency_seconds (Prometheus) – should remain stable under training load.
• Check nvidia-smi dmon for GPU memory usage; inference GPU should stay under 80 %.
• Verify absence of GenerationCancelledError in journalctl -u deepseek.service logs.

Prevention and Best Practices

• Device isolation: Reserve at least one GPU exclusively for inference when running concurrent training.
• CUDA stream priority management: Set NCCL_STREAM_PRIORITY lower than inference streams or explicitly create high‑priority inference streams.
• Memory budgeting: Allocate separate memory pools using torch.cuda.set_per_process_memory_fraction for training and inference.
• Network throttling: Limit NCCL bandwidth with NCCL_IB_TIMEOUT and NCCL_SOCKET_IFNAME to avoid PCIe/NVLink saturation.
• Health checks: Add a lightweight endpoint that issues a short streaming request every 30 seconds; alert on GenerationCancelledError or truncated JSON.
• Process group segregation: Use distinct torch.distributed process groups for training and inference if they must share GPUs.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why does the streaming response fail only when training is active?
   Training launches NCCL collective kernels on high‑priority CUDA streams. Those kernels pre‑empt the lower‑priority inference stream, causing the token generation kernel to be aborted and raising GenerationCancelledError.
2. Can I keep inference on the same GPUs as training without isolation?
   Yes, by lowering NCCL stream priority (NCCL_STREAM_PRIORITY=-2) or by creating a higher‑priority inference stream (torch.cuda.Stream(priority=-1)). However, this reduces training throughput and is not recommended for production.
3. What metric should I watch to detect early signs of stream truncation?
   Monitor the rate of GenerationCancelledError occurrences and the latency of the inference_stream_latency_seconds histogram. A sudden spike indicates contention.
4. Is mixed‑precision training a factor?
   Mixed‑precision reduces memory pressure but can increase NCCL bandwidth usage because gradients are communicated more frequently. The fintech incident showed that mixed‑precision combined with shared memory buffers caused Unexpected EOF while reading from stream. Isolating memory pools mitigates this.
5. Do I need to modify the DeepSeek Docker image?
   No. The fixes are applied via environment variables and optional code patches. If you use the official image, set CUDA_VISIBLE_DEVICES and NCCL_STREAM_PRIORITY at container start.