FAISS Index Build Timeout After Migrating To Hybrid GPU/CPU Cloud

Q: 1. Verify GPU utilization and memory pressure

$ nvidia-smi --query-gpu=timestamp,name,utilization.gpu,memory.total,memory.used --format=csv -l 5 2026-10-09 14:32:05, GPU-0, 12 %, 16384 MiB, 8192 MiB 2026-10-09 14:32:10, GPU-0, 45 %, 16384 MiB, 12288 MiB ... Observation: Utilization spikes to 80 % only after the first batch is fully transferred, confirming that most of the time is spent waiting for data.

Q: 2. Capture network latency and throughput

$ iperf3 -c 10.0.2.5 -t 30 [ 5] 0.00-30.00 sec 2.1 GBytes 600 Mbits/sec 0.00 ms 0/0 (0%) The effective bandwidth (≈ 600 Mbit/s) is far below the PCIe bandwidth, indicating the VPN link is the choke point.

Q: 3. Inspect FAISS logs for fallback messages

[2026-10-09 14:33:12] INFO faiss.gpu: Using CPU for training step 2 – tempMemory insufficient (256 MiB) [2026-10-09 14:33:12] INFO faiss.gpu: Copying intermediate results back to host This confirms the default temporary memory size is inadequate for the dataset size.

Q: 4. Review Kubernetes pod scheduling

$ kubectl get pods -n search -o wide NAME READY STATUS RESTARTS AGE IP NODE faiss-indexer-0 1/1 Running 0 2h 10.1.0.12 gpu-node-01 faiss-indexer-1 1/1 Running 0 2h 10.1.0.13 gpu-node-01 Two indexer pods share the same GPU, leading to serialized CUDA streams.

In this article

Table of Contents

Problem

After migrating the vector search pipeline to a hybrid cloud architecture—on‑premises GPU nodes for heavy‑weight indexing and cloud‑based CPU instances for ingestion and query routing—FAISS index build jobs started exceeding the allocated execution time. The orchestration system aborts the job with a TimeoutError: index build exceeded 1800 s. The failure manifests during both bulk index construction (e.g., IndexIVFFlat training) and batch query processing that triggers on‑the‑fly index updates.

Typical symptoms observed in the job logs:


[2026-10-09 14:32:07] INFO  faiss.index_builder: Starting training for IVF index (nlist=4096) on GPU:0
[2026-10-09 14:32:07] INFO  faiss.index_builder: Transferring 1.2 B vectors to GPU memory
[2026-10-09 14:38:12] WARN  scheduler: Job runtime 370 s, approaching timeout threshold
[2026-10-09 14:45:03] ERROR scheduler: Job failed – TimeoutError: index build exceeded 1800 s

Impact includes delayed data availability, increased ingestion latency, and downstream query timeouts.

Root Cause

The timeout originates from a combination of three interacting factors:

GPU‑CPU data transfer bottleneck: The on‑premises GPU node receives vector batches over a high‑latency VPN (average RTT ≈ 120 ms). Each batch of ~10 M vectors (≈ 1.2 GB) is copied to GPU memory via PCIe, but the network link becomes saturated, causing transfer times of > 5 min per batch.
Resource scheduling contention: The orchestration layer (Kubernetes with gpu-node taint) schedules multiple indexing pods on the same GPU. FAISS uses a single CUDA stream by default, so concurrent jobs serialize on the device, further extending runtime.
Incompatible FAISS build configuration: The index was built with faiss::gpu::StandardGpuResources using the default tempMemory size (256 MB). When the training data exceeds this buffer, FAISS falls back to CPU for intermediate steps, incurring additional host‑to‑device copies and CPU memory pressure, which dramatically slows down training.

These conditions were not present in the legacy all‑CPU environment, where data stayed local and no GPU contention existed.

Debug

Systematic investigation steps that revealed the root cause:

1. Verify GPU utilization and memory pressure


$ nvidia-smi --query-gpu=timestamp,name,utilization.gpu,memory.total,memory.used --format=csv -l 5
2026-10-09 14:32:05, GPU-0, 12 %, 16384 MiB, 8192 MiB
2026-10-09 14:32:10, GPU-0, 45 %, 16384 MiB, 12288 MiB
...

Observation: Utilization spikes to 80 % only after the first batch is fully transferred, confirming that most of the time is spent waiting for data.

2. Capture network latency and throughput


$ iperf3 -c 10.0.2.5 -t 30
[  5]   0.00-30.00  sec  2.1 GBytes   600 Mbits/sec  0.00 ms  0/0 (0%)

The effective bandwidth (≈ 600 Mbit/s) is far below the PCIe bandwidth, indicating the VPN link is the choke point.

3. Inspect FAISS logs for fallback messages


[2026-10-09 14:33:12] INFO  faiss.gpu: Using CPU for training step 2 – tempMemory insufficient (256 MiB)
[2026-10-09 14:33:12] INFO  faiss.gpu: Copying intermediate results back to host

This confirms the default temporary memory size is inadequate for the dataset size.

4. Review Kubernetes pod scheduling


$ kubectl get pods -n search -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE
faiss-indexer-0          1/1     Running   0          2h    10.1.0.12    gpu-node-01
faiss-indexer-1          1/1     Running   0          2h    10.1.0.13    gpu-node-01

Two indexer pods share the same GPU, leading to serialized CUDA streams.

Solution

Address each contributing factor with targeted changes.

1. Increase temporary GPU memory allocation

Modify the FAISS resource configuration to allocate a larger tempMemory buffer (e.g., 4 GB) and enable multi‑stream execution.


// before
faiss::gpu::StandardGpuResources res;
res.setTempMemory(256 * 1024 * 1024); // 256 MiB

// after
faiss::gpu::StandardGpuResources res;
res.setTempMemory(4ULL * 1024 * 1024 * 1024); // 4 GiB
res.setDefaultStreamPerDevice(true); // enable separate streams per GPU

2. Pin a dedicated GPU to each indexing job

Update the Kubernetes deployment to request exclusive GPU resources using the nvidia.com/gpu resource limit and a nodeSelector that isolates GPU nodes per job.


// before (shared GPU)
resources:
  limits:
    nvidia.com/gpu: 1

// after (exclusive GPU)
resources:
  limits:
    nvidia.com/gpu: 1
nodeSelector:
  gpu-type: dedicated
tolerations:
- key: "gpu-node"
  operator: "Exists"
  effect: "NoSchedule"

3. Reduce network latency by colocating ingestion and indexing

Deploy a lightweight “ingestion‑proxy” on the same on‑premises subnet as the GPU node. The proxy buffers incoming vectors and streams them over a high‑throughput internal link (10 Gbps) instead of the VPN.


# ingestion-proxy Dockerfile snippet
FROM python:3.11-slim
RUN pip install fastapi uvicorn
COPY proxy.py /app/
CMD ["uvicorn", "proxy:app", "--host", "0.0.0.0", "--port", "8080"]

4. Adjust the orchestration timeout

Increase the job timeout to accommodate the worst‑case data transfer time, but keep it bounded to avoid runaway jobs.


// before
job.timeoutSeconds = 1800

// after
job.timeoutSeconds = 7200   // 2 hours
job.activeDeadlineSeconds = 10800 // hard cap

Verify

After applying the fixes, run a controlled indexing job and confirm the following:

GPU utilization stays high throughout the run (≥ 70 %).
Data transfer time drops from ~5 min per batch to < 30 s, as shown by nvidia-smi timestamps aligning with log entries.
No fallback messages appear in FAISS logs.
Job completes well within the new timeout (e.g., 1 200 s for a 100 M vector dataset).

Sample successful run log excerpt:


[2026-10-10 09:12:01] INFO  faiss.index_builder: Starting training for IVF index (nlist=4096) on GPU:0
[2026-10-10 09:12:01] INFO  faiss.index_builder: Transferring 1.2 B vectors to GPU memory (30.2 s)
[2026-10-10 09:12:35] INFO  faiss.index_builder: Training completed in 124 s
[2026-10-10 09:14:59] INFO  scheduler: Job completed successfully in 179 s

Prevent

Implement guardrails to avoid recurrence:

Monitoring: Add Prometheus alerts for gpu_memory_used_bytes approaching 90 % and for faiss_index_build_duration_seconds exceeding 30 min.
Capacity planning: Automate sizing of tempMemory based on vector_dim * n_vectors using a CI step that validates the configuration before deployment.
Resource isolation: Enforce a PodDisruptionBudget that guarantees at most one indexing pod per GPU node.
Network health checks: Periodically run iperf3 between ingestion proxies and GPU nodes; trigger a fallback to CPU-only indexing if bandwidth drops below 500 Mbit/s.
Timeout policy: Use a dynamic timeout calculated as base + (data_size / effective_bandwidth) * safety_factor rather than a static constant.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Q1: Why does the index build succeed on a pure‑CPU cluster but fails after adding GPUs?
A: On CPU‑only runs the data never leaves the host, so there is no network transfer overhead. Adding GPUs introduces host‑to‑device copies and, with insufficient tempMemory, FAISS falls back to CPU for intermediate steps, dramatically increasing runtime.

Q2: Can I keep the default tempMemory size and still avoid timeouts?
A: Only if the dataset fits within the temporary buffer (roughly dim * n_vectors * 4 bytes). For large corpora, increasing tempMemory or switching to a multi‑GPU configuration is required.

Q3: How do I know whether my job is being throttled by GPU contention?
A: Check nvidia-smi for multiple processes using the same GPU and inspect CUDA stream activity via nvprof. If you see serialized kernels and low utilization, contention is likely.

Q4: Is it safe to increase the job timeout arbitrarily?
A: Increasing the timeout masks the underlying performance problem. Use a calculated timeout based on data size and observed transfer rates, and pair it with alerts to catch abnormal spikes.

Q5: Will enabling setDefaultStreamPerDevice(true) affect other workloads on the same GPU?
A: It isolates streams per device, reducing interference between concurrent FAISS jobs. However, if multiple unrelated workloads share the GPU, you should still enforce exclusive GPU allocation to guarantee deterministic performance.