Milvus upsert timeout after each epoch in 4‑node DDP training
Problem
During distributed fine‑tuning of an OpenAI GPT‑3.5 model with PyTorch DistributedDataParallel (DDP) on a 4‑node cluster (8 GPUs per node), the training loop succeeds for an epoch and then attempts to upsert the generated embeddings into a Milvus vector store. After each epoch the upsert call blocks for ~30 seconds and eventually raises a timeout exception:
Traceback (most recent call last):
File "train.py", line 312, in main
vector_store.upsert(embeddings, ids)
File ".../milvus_sdk/client.py", line 78, in upsert
resp = self._client.insert(collection_name, vectors, ids, timeout=20)
File ".../milvus_sdk/grpc_client.py", line 145, in insert
raise MilvusException("Insert timeout after 20s")
MilvusException: Insert timeout after 20s
Symptoms observed across all nodes:
- CPU usage spikes on the Milvus server during the upsert window.
- Network interface on the training nodes shows a burst of outbound traffic (~2 GB) followed by a pause.
- Training proceeds normally after the timeout, but the epoch‑level metrics are missing the newly stored vectors.
Root Cause
The timeout originates from a combination of:
- Batch size mismatch: Each node attempts to upsert
batch_size * world_sizevectors (≈ 64 k vectors) in a single RPC. Milvus’s defaultmax_insert_batch_sizeis 16 k, causing the server to reject the request after processing the first 16 k and leaving the client waiting until the RPC deadline expires. - Connection pool exhaustion: The DDP launcher creates one
MilvusClientper process. With 32 processes concurrently issuing upserts, the server’s gRPC thread pool (default 8 workers) becomes saturated, leading to queuing delays that exceed the client‑side timeout. - Network MTU fragmentation: The training nodes use an MTU of 1500 bytes, while the serialized protobuf payload for 64 k vectors exceeds the maximum transmission unit, causing TCP segmentation and retransmission spikes that further delay the RPC.
Debug
Step‑by‑step investigation that isolated the three factors:
1. Log inspection on Milvus
[2024-11-16 02:14:35.112] [INFO] grpc_server.cpp:1125 - Received Insert request: collection=embeddings, rows=65536
[2024-11-16 02:14:35.115] [WARN] insert.cpp:237 - Batch size 65536 exceeds max_insert_batch_size 16384, truncating.
[2024-11-16 02:14:35.117] [ERROR] thread_pool.cpp:78 - No idle worker threads, request queued.
[2024-11-16 02:14:55.124] [INFO] grpc_server.cpp:1150 - Insert request timed out after 20s.
2. Client‑side metrics
$ ss -tnp | grep 19530
ESTAB 0 0 10.0.1.12:56784 10.0.2.5:19530 users:(("python",pid=8421,fd=5))
Only 8 ESTABLISHED connections were observed despite 32 processes, confirming connection pooling limits.
3. Packet capture (first 5 seconds of upsert)
12:34:56.001 IP 10.0.1.12.56784 > 10.0.2.5.19530: Flags [P.], seq 1:1450, ack 1, win 229, length 1449
12:34:56.002 IP 10.0.1.12.56784 > 10.0.2.5.19530: Flags [.], ack 1450, win 229
...
12:34:58.123 IP 10.0.1.12.56784 > 10.0.2.5.19530: Flags [R], seq 0, ack 0, win 0
Repeated retransmissions indicated fragmentation and loss.
Solution
The fix consists of three coordinated changes:
1. Reduce per‑process upsert batch size
Configure the client to split the embedding array into chunks that respect Milvus’s max_insert_batch_size (default 16 k). The SDK provides a helper batch_upsert method.
# Before
vector_store.upsert(embeddings, ids) # embeddings.shape = (65536, 1536)
# After
def chunked_upsert(vs, vectors, ids, chunk=16384):
for i in range(0, len(vectors), chunk):
vs.upsert(vectors[i:i+chunk], ids[i:i+chunk])
chunked_upsert(vector_store, embeddings, ids)
2. Increase Milvus gRPC worker pool
Modify milvus.yaml (or the Helm values) to allocate more workers:
# milvus.yaml
grpc:
maxConcurrentGrpc: 32 # default 8
maxInsertBatchSize: 16384
After applying the new config and restarting the Milvus service, the server can handle concurrent upserts from all 32 processes.
3. Align network MTU and enable protobuf compression
Set the training nodes’ network interface MTU to 9000 (jumbo frames) to accommodate larger payloads, and enable gRPC compression on the client side.
# Increase MTU
sudo ip link set dev eth0 mtu 9000
# Enable compression in Milvus client
client = MilvusClient(
uri="tcp://10.0.2.5:19530",
compression="gzip" # options: none, gzip, snappy
)
Verify
After deploying the three changes, run a single epoch and observe the following:
- No timeout exception;
upsertreturns within 2 seconds. - Milvus logs show successful batch inserts:
[2024-11-16 03:02:10.021] [INFO] insert.cpp:112 - Inserted 16384 rows (batch 1/4) into collection=embeddings
[2024-11-16 03:02:10.023] [INFO] insert.cpp:112 - Inserted 16384 rows (batch 2/4) into collection=embeddings
...
[2024-11-16 03:02:10.045] [INFO] insert.cpp:112 - Inserted 16384 rows (batch 4/4) into collection=embeddings
Validate that the vectors are searchable:
$ python - <<'PY'
from pymilvus import Collection
col = Collection("embeddings")
res = col.search(
data=[[0.1]*1536],
anns_field="embedding",
param={"metric_type":"IP","params":{"nprobe":10}},
limit=5
)
print(res)
PY
Returned results contain the newly upserted IDs, confirming persistence.
Prevent
- Monitor insert latency: Create a Prometheus alert on
milvus_grpc_insert_latency_seconds> 5 s for more than 3 consecutive minutes. - Enforce batch size policy: Wrap all upsert calls in a utility that checks
len(vectors) ≤ max_insert_batch_sizeand raises a warning during CI. - Capacity planning: Size the gRPC worker pool to at least
world_size * processes_per_node / 2to avoid saturation under peak load. - Network hygiene: Standardize MTU across the cluster (prefer jumbo frames) and enable gRPC compression for large protobuf payloads.
FAQ
- Why does the timeout only appear after the first epoch?
The first epoch uses a warm‑up learning rate and generates fewer embeddings (e.g., due to gradient accumulation). Subsequent epochs produce the full batch size, exposing the batch‑size limit. - Can I increase
max_insert_batch_sizeinstead of chunking?
Yes, but the server must allocate proportional memory. In production environments the default 16 k is a safe upper bound; increasing it risks OOM on the Milvus node. - Is gRPC compression mandatory?
It is not required if the network MTU is large enough, but it reduces payload size and mitigates fragmentation, especially when jumbo frames are unavailable. - How many gRPC workers should I allocate for a 64‑GPU cluster?
A practical rule isworkers = min(2 * total_processes, 128). Adjust based on observed CPU utilization and insert latency. - What if I cannot change the MTU on the network?
Fallback to smaller chunk sizes (e.g., 8 k) and enable compression; also consider using the MilvusBulkInsertAPI which streams data in smaller frames.
Related Topic Hub: LLM Systems Troubleshooting Hub