Milvus vector insert failures when CronJobs overlap during schedule change

Problem

After a schedule change in the data‑pipeline, two CronJob resources that drive Milvus batch ingestion started to overlap. When the second job launched while the first was still inserting vectors, Milvus returned INSERT FAILED: RESOURCE_EXHAUSTED errors and the corresponding Kubernetes pod terminated with a non‑zero exit code. The symptom manifested as missing vectors in the target collection and intermittent spikes in CPU/memory usage on the Milvus server pods.

Root Cause

The overlap occurs because the new cron expression reduces the interval between runs without adjusting the startingDeadlineSeconds or adding a guard that prevents concurrent execution. Milvus processes each batch in a single Insert RPC that holds an exclusive write lock on the target collection. When two jobs run simultaneously:

  • Both jobs acquire the same write lock; the second blocks until the lock timeout (default 30 s) expires.
  • Milvus’ internal ResourceManager detects the contention and aborts the second Insert request, returning RESOURCE_EXHAUSTED.
  • The Kubernetes CronJob controller treats the non‑zero exit as a failure and retries according to backoffLimit, further increasing load.

In short, the schedule change introduced a timing window where the previous batch had not yet completed, leading to resource contention and failed vector insertions.

Debug

Below is a typical debugging workflow that isolates the overlap.

1. Observe the failing pod logs

kubectl logs -n data-pipeline $(kubectl get pods -n data-pipeline -l job-name=milvus-ingest-20231109-01 -o jsonpath='{.items[0].metadata.name}')
2026-11-09T03:12:45.123Z [ERROR] Insert failed: RESOURCE_EXHAUSTED: write lock timeout
2026-11-09T03:12:45.124Z [INFO]  Batch size: 50000, collection: movies

2. Check CronJob schedule and last run times

kubectl get cronjob milvus-ingest -n data-pipeline -o yaml | grep -E 'schedule|lastScheduleTime|startingDeadlineSeconds'
schedule: "*/30 * * * *"
lastScheduleTime: "2026-11-09T03:00:00Z"
startingDeadlineSeconds: 300

3. Correlate with Milvus server metrics

curl -s http://milvus-metrics:9090/api/v1/query?query=milvus_write_lock_wait_seconds_total
{
  "status":"success",
  "data":{"result":[{"metric":{"instance":"milvus-0"},"value":[170, "12"]}]}
}

The metric shows 12 seconds of lock wait during the failure window.

4. Verify overlapping jobs

kubectl get jobs -n data-pipeline --field-selector=status.successful=0 -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp
milvus-ingest-20231109-01   2026-11-09T03:00:12Z
milvus-ingest-20231109-02   2026-11-09T03:30:05Z

Both jobs are active, confirming concurrency.

Solution

The fix consists of two parts: (1) make the CronJob mutually exclusive, and (2) adjust Milvus resource limits to tolerate brief overlaps if they occur.

1. Add a concurrencyPolicy and a lock file

Update the CronJob manifest to use Forbid concurrency and implement a simple file‑based lock in the ingestion script.

Before

apiVersion: batch/v1
kind: CronJob
metadata:
  name: milvus-ingest
spec:
  schedule: "*/30 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: ingest
            image: registry.example.com/milvus-ingest:1.2.0
            args: ["--batch-size=50000"]
          restartPolicy: OnFailure

After

apiVersion: batch/v1
kind: CronJob
metadata:
  name: milvus-ingest
spec:
  schedule: "*/30 * * * *"
  concurrencyPolicy: Forbid   # <-- prevents overlapping runs
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 5
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: ingest
            image: registry.example.com/milvus-ingest:1.2.0
            command: ["/bin/sh","-c"]
            args:
            - |
              LOCKFILE=/tmp/ingest.lock
              if [ -e $LOCKFILE ]; then
                echo "$(date) - another ingest job is running, exiting."
                exit 1
              fi
              touch $LOCKFILE
              trap "rm -f $LOCKFILE" EXIT
              /app/ingest --batch-size=50000
          restartPolicy: OnFailure

2. Increase Milvus write‑lock timeout (optional)

If occasional overlap cannot be avoided (e.g., manual trigger), raise the timeout in Milvus configuration.

# milvus.yaml
engine:
  write_lock_timeout_seconds: 120   # default 30

After editing, restart Milvus pods:

kubectl rollout restart statefulset milvus -n milvus

Verify

Confirm that only one job runs at a time and that insertions succeed.

1. Observe CronJob status

kubectl get cronjob milvus-ingest -n data-pipeline -o yaml | grep concurrencyPolicy
concurrencyPolicy: Forbid

2. Check that the lock file prevents a second run

kubectl logs -n data-pipeline $(kubectl get pod -l job-name=milvus-ingest-20231109-02 -o jsonpath='{.items[0].metadata.name}')
2026-11-09T03:30:06.001Z - another ingest job is running, exiting.

3. Validate successful inserts

curl -s http://milvus:19121/api/v1/collections/movies/stats | jq '.row_count'
{
  "row_count": 1500000
}

The row count reflects the expected increase after the batch run.

Prevent

  • Use concurrencyPolicy: Forbid for any batch‑processing CronJob that touches shared state.
  • Implement an explicit lock (file, ConfigMap, or distributed lock) in the job script to guard against manual triggers.
  • Monitor Milvus lock‑wait metrics (milvus_write_lock_wait_seconds_total) and set alerts when the value exceeds a threshold.
  • Keep job duration < 80 % of the schedule interval to provide a safety margin.
  • Version‑control CronJob manifests and require peer review for schedule changes.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does Milvus return RESOURCE_EXHAUSTED instead of a timeout? The server treats a write‑lock acquisition failure as a resource exhaustion event, propagating the error to the client.
  2. Can I rely solely on concurrencyPolicy: Forbid? Yes for Kubernetes‑native scheduling, but external triggers (e.g., kubectl create job) bypass the policy, so an application‑level lock is recommended.
  3. What if the ingestion script crashes before removing the lock file? The trap statement ensures the lock is removed on container exit; for non‑graceful node failures, use a short activeDeadlineSeconds on the Job to force cleanup.
  4. Is increasing write_lock_timeout_seconds a safe long‑term fix? It masks contention but does not eliminate it; use only as a temporary mitigation while you reduce job overlap.
  5. How can I detect overlapping CronJobs without modifying the job? Query the Kubernetes API for active Jobs with the same label selector and compare creationTimestamp against the expected duration.