a bird of prey is flying in the air

Milvus memory allocation failure during batch insert after multi‑GPU training

by

Problem: Milvus Memory Allocation Failure During Batch Insert After Multi‑GPU Training

In a production pipeline, a distributed multi‑GPU training job streams high‑dimensional embeddings (e.g., 1536‑dim float vectors) to a Milvus vector database running on a single node with 64 GiB RAM. When the training job attempts to insert a batch of ~200 k vectors, Milvus logs show allocation errors and the insert request fails with status=FAILED, message="memory allocation failed". The failure also propagates to index construction, causing the downstream similarity search to be unavailable.

Root Cause

The failure stems from a combination of three factors:

  1. In‑memory buffer overflow: Milvus Insert API first stores incoming vectors in an internal InsertBuffer before flushing to disk. The default buffer size is 8 GiB. A batch of 200 k vectors × 1536 dimensions × 4 bytes/float ≈ 1.2 GiB, but the buffer also holds temporary structures (row IDs, timestamps, primary keys) that roughly double the memory footprint. When multiple concurrent insert streams arrive (e.g., from several GPU workers), the buffer quickly exceeds the configured limit.
  2. Memory fragmentation: Milvus uses a custom memory pool (jemalloc) for vector storage. Continuous allocation/deallocation of large contiguous blocks during batch inserts leads to fragmentation, reducing the amount of usable contiguous memory even though total free RAM appears sufficient.
  3. Insufficient swap / OOM killer: The host OS has swap disabled. When the process requests a large contiguous block, the kernel denies the allocation, and Milvus reports a memory allocation failure instead of falling back to disk.

Debug: Investigation Steps

1. Examine Milvus logs


2024-10-12 14:23:45.123 INFO  milvus::engine::insert - Insert request received, collection=images, batch_size=200000
2024-10-12 14:23:45.124 WARN  milvus::engine::memory - InsertBuffer size (8192 MiB) exceeded, current usage=10240 MiB
2024-10-12 14:23:45.125 ERROR milvus::engine::memory - Memory allocation failed: requested 1258291200 bytes
2024-10-12 14:23:45.126 INFO  milvus::engine::index - Index build aborted due to previous errors

2. Check OS memory statistics


$ free -h
              total        used        free      shared  buff/cache   available
Mem:           63Gi       45Gi       2.1Gi       0.5Gi        15Gi        12Gi
Swap:            0B          0B          0B

The available column shows 12 GiB, but the process cannot allocate a contiguous 1.2 GiB block because the memory pool is fragmented.

3. Inspect Milvus configuration


# milvus.yaml (excerpt)
cache:
  cache_size: 8GiB          # InsertBuffer size
  insert_buffer_size: 8GiB
engine:
  use_blas_threshold: 1024
  gpu_resource:
    enable: true
    gpu_ids: [0,1,2,3]

4. Reproduce with a minimal client script


import numpy as np
from pymilvus import Collection, connections, utility

connections.connect(host='127.0.0.1', port='19530')
collection = Collection('images')
vectors = np.random.random((200000, 1536)).astype('float32')
ids = list(range(200000))

try:
    collection.insert([ids, vectors])
except Exception as e:
    print('Insert failed:', e)

Running the script reproduces the same error, confirming that the issue is not specific to the training pipeline.

Solution: Adjust Memory Management and Insertion Strategy

1. Reduce InsertBuffer size and enable auto‑flush

Configure Milvus to flush smaller batches to disk, preventing the buffer from growing beyond physical memory.


# milvus.yaml (updated)
cache:
  cache_size: 4GiB               # Reduce total cache
  insert_buffer_size: 2GiB       # Smaller per‑collection buffer
  insert_buffer_flush_interval: 30   # seconds
  insert_buffer_flush_threshold: 0.75 # Flush when 75% full

2. Enable memory_pool fragmentation mitigation

Switch the allocator to mmap for large allocations, which bypasses jemalloc fragmentation.


# milvus.yaml (additional)
engine:
  memory_pool:
    enable: true
    allocator: mmap
    mmap_threshold: 256MiB

3. Add swap space as a safety net


# Create a 16GiB swap file
$ sudo fallocate -l 16G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile

# Verify
$ swapon --show
FILE       TYPE SIZE USED PRIO
/swapfile  file 16G   0B   -2

Swap does not replace RAM but prevents immediate OOM termination when large contiguous allocations are requested.

4. Split large batches on the client side

Modify the training pipeline to send smaller chunks (e.g., 50 k vectors) and wait for acknowledgment before sending the next chunk.


BATCH_SIZE = 50000

def stream_embeddings(embeddings):
    total = len(embeddings)
    for i in range(0, total, BATCH_SIZE):
        batch = embeddings[i:i+BATCH_SIZE]
        ids = list(range(i, i+len(batch)))
        collection.insert([ids, batch])
        # optional: collection.flush()

5. Restart Milvus with the new configuration


$ docker-compose down
$ docker-compose up -d   # Assuming Milvus runs in Docker

Verify: Confirming the Fix

1. Insert a test batch


import numpy as np
from pymilvus import Collection, connections

connections.connect(host='127.0.0.1', port='19530')
col = Collection('images')
vectors = np.random.random((50000, 1536)).astype('float32')
ids = list(range(50000))

result = col.insert([ids, vectors])
print('Inserted:', result)

Expected output:


Inserted: {'insert_count': 50000, 'primary_keys': [...], 'status': 'Success'}

2. Monitor Milvus metrics


$ curl http://localhost:9091/metrics | grep milvus_insert_buffer
milvus_insert_buffer_size_bytes 2097152000
milvus_insert_buffer_used_bytes 1048576000

The used buffer stays below the configured threshold.

3. Verify index build


$ curl -X POST http://127.0.0.1:19121/collections/images/indexes \
  -d '{"field_name":"embedding","index_type":"IVF_FLAT","metric_type":"IP","params":{"nlist":1024}}'

Response should be {"status":"success"} and no allocation errors appear in the logs.

Prevent: Operational Guardrails

  • Capacity planning: Keep cache_size ≤ 50 % of physical RAM on nodes that host Milvus.
  • Batch size limits: Enforce a maximum client batch size (e.g., 50 k vectors) via API gateway or SDK wrapper.
  • Monitoring: Alert when milvus_insert_buffer_used_bytes exceeds 80 % of cache_size or when milvus_memory_allocation_failed_total increments.
  • Periodic health checks: Run milvusctl healthcheck and verify InsertBuffer metrics.
  • Swap provisioning: Ensure at least 1 GiB swap per 8 GiB of RAM for safety, especially on nodes with high‑dimensional vectors.
  • Version lock: Use Milvus 2.3.0 or later, which includes improved memory pool fragmentation handling.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the failure happen only with multi‑GPU training and not with a single GPU? Multi‑GPU training often produces embeddings faster, increasing the number of concurrent insert streams. The higher concurrency fills the InsertBuffer more quickly, exposing the buffer size limit and fragmentation issues.
  2. Can I increase the RAM on the Milvus node instead of changing the buffer size? Adding RAM helps but does not solve fragmentation. Adjusting insert_buffer_size and enabling mmap allocation are more reliable because they control how memory is requested.
  3. Is disabling swap a best practice for Milvus? Disabling swap eliminates latency spikes but removes a safety net for large contiguous allocations. For production workloads with large batches, a modest swap file is recommended.
  4. How do I know if memory fragmentation is the culprit? Observe the milvus_memory_fragmentation_ratio metric (available in Milvus 2.3+). Values > 0.6 indicate significant fragmentation.
  5. Will reducing the batch size impact training throughput? Yes, but the impact is usually minor compared to the cost of failed inserts. Use asynchronous pipelines to overlap training and insertion, keeping overall throughput high.