A red kite soars against a green forest

Milvus index rebuild failure in air‑gapped deployment missing binary dependencies

by

Problem

In an air‑gapped deployment of Milvus (v2.3.x), attempts to rebuild a vector index on an existing collection fail with errors indicating missing binary dependencies and permission denials. The failure occurs during the milvusctl index rebuild operation and leaves the collection without a usable index.

Typical Symptoms

  • CLI returns non‑zero exit code:
milvusctl index rebuild --collection my_vectors
Error: failed to execute index builder: exec: "faiss_build": executable file not found in $PATH
  • Milvus logs contain lines similar to:
[2024-09-12 14:32:07.123] [ERROR] [index_builder] failed to launch binary '/opt/milvus/bin/faiss_build': permission denied
[2024-09-12 14:32:07.124] [WARN]  [index_builder] falling back to Python wrapper, which is not available in this build
  • Metrics for index_build_duration_seconds never transition to success, remaining at 0.
  • Subsequent search queries return IndexNotFound errors.

Root Cause

The index rebuild process in Milvus relies on native binary executables (e.g., faiss_build, hnsw_build) that are packaged separately from the core server. In an air‑gapped environment:

  1. Missing binaries: The internal package repository does not contain the milvus-index-builder package, so the binaries are never installed on the node.
  2. Incorrect permissions: Even when the binaries are manually copied, they inherit restrictive file system ACLs (e.g., root:root 600) that prevent the Milvus service user (milvus) from executing them.
  3. Library dependencies: The binaries depend on shared libraries such as libopenblas.so and libgomp.so. Without access to an internal yum/apt mirror, these libraries are absent, causing the loader to abort with ENOENT.
  4. Resource constraints: The default temporary directory for the builder (/tmp/milvus_index_build) resides on a small partition, leading to ENOSPC errors that are sometimes masked by the missing‑binary message.

Debug

Systematic investigation steps that reproduced the failure:

1. Verify binary presence and permissions

# Check expected location
ls -l /opt/milvus/bin/faiss_build
-rw------- 1 root root 12M Sep 10 08:15 /opt/milvus/bin/faiss_build

# Attempt to run as milvus user
sudo -u milvus /opt/milvus/bin/faiss_build --version
bash: /opt/milvus/bin/faiss_build: Permission denied

2. Inspect dynamic linker errors

# Use ldd to reveal missing shared objects
ldd /opt/milvus/bin/faiss_build
    not a dynamic executable
# If it were dynamic:
# ldd: /opt/milvus/bin/faiss_build: libopenblas.so.0 => not found

3. Review Milvus server logs for loader failures

[2024-09-12 14:32:07.123] [ERROR] [index_builder] execve("/opt/milvus/bin/faiss_build"): No such file or directory
[2024-09-12 14:32:07.124] [ERROR] [index_builder] execve failed: ENOENT

4. Confirm temporary storage capacity

# df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda2        2G   1.9G  100M  96% /tmp

5. Check internal repository for the builder package

# yum list available | grep milvus-index-builder
No matches found.

Solution

Apply the following corrective actions in the order presented.

1. Provision the index builder binaries from an approved source

Download the milvus-index-builder tarball on a machine with internet access, verify its checksum, and transfer it via air‑gap media.

# On a connected workstation
curl -O https://github.com/milvus-io/milvus/releases/download/v2.3.0/milvus-index-builder-linux-amd64.tar.gz
sha256sum milvus-index-builder-linux-amd64.tar.gz
# => 3b7e1f9c... (record this value)

# Transfer the tarball to the air‑gapped host (e.g., via USB)

2. Install binaries with correct ownership and mode

# On the Milvus node (as root)
tar -xzf milvus-index-builder-linux-amd64.tar.gz -C /opt/milvus/bin
chown -R milvus:milvus /opt/milvus/bin
chmod 755 /opt/milvus/bin/faiss_build /opt/milvus/bin/hnsw_build

3. Satisfy shared‑library dependencies from the internal repo

Copy the required libraries from a trusted build environment into a local directory and add it to LD_LIBRARY_PATH for the Milvus service.

# Gather libraries on a build host
mkdir -p /opt/milvus/lib
cp /usr/lib64/libopenblas.so.0 /opt/milvus/lib/
cp /usr/lib64/libgomp.so.1   /opt/milvus/lib/

# Update systemd unit
cat >> /etc/systemd/system/milvus.service.d/override.conf <<EOF
[Service]
Environment="LD_LIBRARY_PATH=/opt/milvus/lib:/usr/lib64"
EOF

systemctl daemon-reload
systemctl restart milvus

4. Expand the temporary build directory

Mount a larger volume for index building and point Milvus to it via the index_build_tmp_dir configuration key.

# Create mount point
mkdir -p /mnt/milvus_build_tmp
mount /dev/sdb1 /mnt/milvus_build_tmp   # 20 GB partition

# Update milvus.yaml
index_build_tmp_dir: "/mnt/milvus_build_tmp"

5. Re‑run the index rebuild

milvusctl index rebuild --collection my_vectors
[INFO] Starting rebuild for collection 'my_vectors'
[INFO] Using binary '/opt/milvus/bin/faiss_build'
[INFO] Index rebuild completed in 132.4s

Verify

Confirm that the index is now usable and that no residual errors appear.

1. Check collection status via Milvus SDK

from pymilvus import Collection
c = Collection("my_vectors")
print(c.indexes)  # Should list the newly built index

2. Run a search query

results = c.search(
    data=[[0.1]*128],
    anns_field="embedding",
    param={"metric_type":"IP","params":{"nprobe":10}},
    limit=5
)
print(len(results[0]))  # Expect 5 hits

3. Inspect logs for absence of previous errors

journalctl -u milvus -f | grep index_builder
# No new error lines should appear

4. Verify temporary storage usage

df -h /mnt/milvus_build_tmp
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdb1        20G   1.2G   18G   7% /mnt/milvus_build_tmp

Prevent

Implement guardrails to avoid recurrence in future air‑gapped rollouts.

Preventive Action Implementation Detail
Package mirroring Maintain an internal mirror of milvus-index-builder and its dependent libraries (OpenBLAS, libgomp). Use createrepo or apt-mirror to keep versions in sync with Milvus releases.
Permission hardening checklist Automate a post‑install script that sets chmod 755 and chown milvus:milvus on all binaries under /opt/milvus/bin.
Temporary storage sizing Document required index_build_tmp_dir size (≈ 10 GB per 1 B vectors) and provision dedicated mount points during cluster provisioning.
Health‑check augmentation Add a systemd health check that verifies the presence and executability of each builder binary at startup.
CI validation Include a test stage that runs milvusctl index rebuild --dry-run against a minimal collection in the isolated environment.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does Milvus fall back to the Python wrapper during rebuild? The server first attempts to launch the native binary. If execve fails, it logs a warning and tries the pure‑Python implementation, which is disabled in production builds for performance reasons, resulting in an immediate failure.
  2. Can I use a container image that already contains the builder binaries? Yes. Build a custom Milvus image that copies the binaries into /opt/milvus/bin and sets appropriate permissions. Ensure the image also includes the required shared libraries.
  3. What if my internal repository only provides .deb packages but the host runs RHEL? Extract the binaries from the .deb using dpkg-deb -x on a compatible machine, then transfer the extracted files to the RHEL host.
  4. How do I know which binary is required for a given index type? The mapping is static:
    Index Type Builder Binary
    IVF_FLAT, IVF_PQ, IVF_SQ8 faiss_build
    HNSW hnsw_build
    DISKANN diskann_build

    Use milvusctl index list-types to view supported types.

  5. Is it safe to set LD_LIBRARY_PATH globally on the host? Prefer configuring it only for the Milvus service via a systemd drop‑in file to avoid affecting other processes.