FAISS search hangs when stop sequence missing during autoscaling

Problem

FAISS‑based vector search service is deployed as a stateful microservice in Kubernetes with horizontal pod autoscaling (HPA). When a query contains a predefined stop sequence (e.g., a special token that should terminate the search early), the service fails to detect it. The search continues until the maximum k results are collected, often exhausting CPU and memory, causing the pod to appear hung and the HPA to scale out unnecessarily.

Typical symptoms:

  • API request latency spikes from < 50 ms to > 30 s or times out.
  • Pod CPU usage reaches 100 % while the request is in flight.
  • HPA scales the deployment from 2 → 10 replicas, then crashes due to OOM.
  • Logs contain the search loop but never log “stop sequence detected”.

Root Cause

The search implementation iterates over the top‑k nearest vectors and appends results until either k is reached or a stop sequence token is found. The stop‑sequence check relied on a std::unordered_set<std::string> populated from a ConfigMap at container start. During an autoscaling event, new pods are created from a fresh image that does not mount the ConfigMap (PVC mount order race). Consequently, the in‑memory stop‑sequence set remains empty, so the conditional break never executes.

Additional contributing factors:

  • The container entrypoint loads the stop‑sequence list lazily; if the file is missing, the code silently continues without error.
  • FAISS index files are stored on a PVC, but the ConfigMap is mounted on a separate volume; the pod spec did not declare a dependency, leading to a race condition during pod startup.
  • No health‑check validates that the stop‑sequence list is non‑empty before accepting traffic.

Debug

Step‑by‑step investigation performed in the affected cluster.

1. Observe pod logs


$ kubectl logs faiss-search-5d9f7c9d7b-9kz2m
[INFO] query_id=42 start_search k=100
[DEBUG] retrieved_id=12345 score=0.987
[DEBUG] retrieved_id=12346 score=0.985
...
# No "stop sequence detected" line appears

2. Verify ConfigMap mount


$ kubectl exec -it faiss-search-5d9f7c9d7b-9kz2m -- ls /etc/faiss/stop_sequences
ls: cannot access '/etc/faiss/stop_sequences': No such file or directory

3. Check pod spec for volume ordering

Volume Name Mount Path Source
faiss-index /data/faiss PersistentVolumeClaim
stop-seq-cm /etc/faiss/stop_sequences ConfigMap

The faiss-index volume is declared before stop-seq-cm. Kubernetes does not guarantee mount order, and the entrypoint reads the stop‑sequence file before the ConfigMap is attached.

4. Examine HPA events


$ kubectl describe hpa faiss-search
Events:
  Type    Reason             Age   From               Message
  ----    ------             ----  ----               -------
  Normal  SuccessfulRescale  5m    horizontal-pod-autoscaler  New size: 5; reason: cpu utilization above target

5. Reproduce locally

Running the same query against a pod started without the ConfigMap reproduces the hang.


$ curl -X POST http://localhost:8080/search -d '{"query":"foo bar ", "k":100}'
# client hangs until timeout

Solution

Three coordinated changes fix the issue:

1. Enforce ConfigMap availability at startup

Modify the entrypoint script to abort if the stop‑sequence file is missing.

# before/after comparison

# Before
if [ -f /etc/faiss/stop_sequences/list.txt ]; then
    cp /etc/faiss/stop_sequences/list.txt /app/stop.txt
fi

# After
if [ -f /etc/faiss/stop_sequences/list.txt ]; then
    cp /etc/faiss/stop_sequences/list.txt /app/stop.txt
else
    echo "ERROR: stop sequence file missing – aborting"
    exit 1
fi

2. Load stop sequences synchronously after all volumes are mounted

Update the C++ search component to read the file at the first request rather than at static initialization.

// before
static std::unordered_set<std::string> stop_set = load_stop_set();

// after
std::unordered_set<std::string> get_stop_set() {
    static std::unordered_set<std::string> stop_set;
    static std::once_flag flag;
    std::call_once(flag, [](){
        std::ifstream f("/app/stop.txt");
        std::string line;
        while (std::getline(f, line)) {
            stop_set.insert(line);
        }
    });
    return stop_set;
}

// search loop
for (auto &result : results) {
    if (get_stop_set().count(result.token)) {
        LOG_INFO("stop sequence detected, breaking");
        break;
    }
    // …process result…
}

3. Adjust pod spec to guarantee mount order and add liveness probe

# before (partial)
volumes:
- name: faiss-index
  persistentVolumeClaim:
    claimName: faiss-pvc
- name: stop-seq-cm
  configMap:
    name: faiss-stop-seq

# after (partial)
volumes:
- name: stop-seq-cm
  configMap:
    name: faiss-stop-seq
- name: faiss-index
  persistentVolumeClaim:
    claimName: faiss-pvc
volumeMounts:
- name: stop-seq-cm
  mountPath: /etc/faiss/stop_sequences
  readOnly: true
- name: faiss-index
  mountPath: /data/faiss
livenessProbe:
  exec:
    command: ["cat", "/app/stop.txt"]
  initialDelaySeconds: 5
  periodSeconds: 10

Placing the ConfigMap volume first ensures it is mounted before the index PVC. The liveness probe forces the container to fail fast if the file is absent, preventing the pod from entering a ready state.

Verify

After applying the changes, run the following checks.

1. Deploy new version


$ kubectl rollout restart deployment/faiss-search
$ kubectl get pods -w

2. Confirm stop‑sequence file presence


$ kubectl exec -it faiss-search-6c7d9f8b9c-2xk5l -- cat /app/stop.txt

3. Execute a query that includes the stop token


$ time curl -s -X POST http://faiss-search.default.svc.cluster.local/search \
    -d '{"query":"alpha beta  gamma", "k":100}'
{
  "results": [
    {"id":123,"score":0.99},
    {"id":124,"score":0.98}
  ],
  "stopped": true
}
real    0m0.112s

The response includes "stopped": true and latency is sub‑100 ms, confirming the break condition works.

4. Observe HPA behavior


$ kubectl get hpa faiss-search
NAME          REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
faiss-search  Deployment/faiss-search  10%/80%   2         5         2          10m

No unnecessary scaling events occur under normal load.

Prevent

  • Startup validation: Always abort container start if required configuration files (stop sequence list, index metadata) are missing.
  • Volume ordering: Declare ConfigMap volumes before PVCs when the application depends on both during initialization.
  • Health checks: Use a liveness probe that verifies the stop‑sequence file can be read.
  • Unit & integration tests: Add tests that inject a query containing the stop token and assert that the search terminates early.
  • Monitoring: Alert on search latency > 500 ms or on CPU > 80 % for more than 30 s; correlate with HPA scaling events.
  • Configuration drift guardrails: Store the stop‑sequence ConfigMap version in a ConfigMap checksum annotation on the Deployment; let the rollout controller restart pods on checksum change.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the issue appear only after autoscaling?
    New pods are created from the base image without the ConfigMap mounted early enough. Existing pods that started before the bug have the file, so the problem is only visible on scale‑out.
  2. Can I rely on FAISS’s own metadata to store stop sequences?
    FAISS indexes do not have a concept of stop tokens; they are pure vector structures. Stop‑sequence handling must be performed in the application layer.
  3. What if I need multiple stop‑sequence sets per tenant?
    Mount a per‑tenant ConfigMap using a ProjectedVolume or load the list from a sidecar that queries a central store (e.g., etcd) before the first request.
  4. Is a timeout a safe fallback?
    Adding a hard timeout (e.g., 2 s) prevents indefinite hangs but masks the underlying bug. Use it as a secondary safety net, not a primary solution.
  5. How do I test the fix in CI?
    Spin up a local Kubernetes cluster (kind or minikube), deploy the manifest with the ConfigMap, and run a curl request containing the stop token. Assert that the response contains "stopped": true and that the container logs the “stop sequence detected” line.