Problem
After a schedule change in the data‑pipeline, two CronJob resources that drive Milvus batch ingestion started to overlap. When the second job launched while the first was still inserting vectors, Milvus returned INSERT FAILED: RESOURCE_EXHAUSTED errors and the corresponding Kubernetes pod terminated with a non‑zero exit code. The symptom manifested as missing vectors in the target collection and intermittent spikes in CPU/memory usage on the Milvus server pods.
Root Cause
The overlap occurs because the new cron expression reduces the interval between runs without adjusting the startingDeadlineSeconds or adding a guard that prevents concurrent execution. Milvus processes each batch in a single Insert RPC that holds an exclusive write lock on the target collection. When two jobs run simultaneously:
- Both jobs acquire the same write lock; the second blocks until the lock timeout (default 30 s) expires.
- Milvus’ internal
ResourceManagerdetects the contention and aborts the secondInsertrequest, returningRESOURCE_EXHAUSTED. - The Kubernetes
CronJobcontroller treats the non‑zero exit as a failure and retries according tobackoffLimit, further increasing load.
In short, the schedule change introduced a timing window where the previous batch had not yet completed, leading to resource contention and failed vector insertions.
Debug
Below is a typical debugging workflow that isolates the overlap.
1. Observe the failing pod logs
kubectl logs -n data-pipeline $(kubectl get pods -n data-pipeline -l job-name=milvus-ingest-20231109-01 -o jsonpath='{.items[0].metadata.name}')
2026-11-09T03:12:45.123Z [ERROR] Insert failed: RESOURCE_EXHAUSTED: write lock timeout
2026-11-09T03:12:45.124Z [INFO] Batch size: 50000, collection: movies
2. Check CronJob schedule and last run times
kubectl get cronjob milvus-ingest -n data-pipeline -o yaml | grep -E 'schedule|lastScheduleTime|startingDeadlineSeconds'
schedule: "*/30 * * * *"
lastScheduleTime: "2026-11-09T03:00:00Z"
startingDeadlineSeconds: 300
3. Correlate with Milvus server metrics
curl -s http://milvus-metrics:9090/api/v1/query?query=milvus_write_lock_wait_seconds_total
{
"status":"success",
"data":{"result":[{"metric":{"instance":"milvus-0"},"value":[170, "12"]}]}
}
The metric shows 12 seconds of lock wait during the failure window.
4. Verify overlapping jobs
kubectl get jobs -n data-pipeline --field-selector=status.successful=0 -o custom-columns=NAME:.metadata.name,AGE:.metadata.creationTimestamp
milvus-ingest-20231109-01 2026-11-09T03:00:12Z
milvus-ingest-20231109-02 2026-11-09T03:30:05Z
Both jobs are active, confirming concurrency.
Solution
The fix consists of two parts: (1) make the CronJob mutually exclusive, and (2) adjust Milvus resource limits to tolerate brief overlaps if they occur.
1. Add a concurrencyPolicy and a lock file
Update the CronJob manifest to use Forbid concurrency and implement a simple file‑based lock in the ingestion script.
Before
apiVersion: batch/v1
kind: CronJob
metadata:
name: milvus-ingest
spec:
schedule: "*/30 * * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: ingest
image: registry.example.com/milvus-ingest:1.2.0
args: ["--batch-size=50000"]
restartPolicy: OnFailure
After
apiVersion: batch/v1
kind: CronJob
metadata:
name: milvus-ingest
spec:
schedule: "*/30 * * * *"
concurrencyPolicy: Forbid # <-- prevents overlapping runs
startingDeadlineSeconds: 300
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
jobTemplate:
spec:
template:
spec:
containers:
- name: ingest
image: registry.example.com/milvus-ingest:1.2.0
command: ["/bin/sh","-c"]
args:
- |
LOCKFILE=/tmp/ingest.lock
if [ -e $LOCKFILE ]; then
echo "$(date) - another ingest job is running, exiting."
exit 1
fi
touch $LOCKFILE
trap "rm -f $LOCKFILE" EXIT
/app/ingest --batch-size=50000
restartPolicy: OnFailure
2. Increase Milvus write‑lock timeout (optional)
If occasional overlap cannot be avoided (e.g., manual trigger), raise the timeout in Milvus configuration.
# milvus.yaml
engine:
write_lock_timeout_seconds: 120 # default 30
After editing, restart Milvus pods:
kubectl rollout restart statefulset milvus -n milvus
Verify
Confirm that only one job runs at a time and that insertions succeed.
1. Observe CronJob status
kubectl get cronjob milvus-ingest -n data-pipeline -o yaml | grep concurrencyPolicy
concurrencyPolicy: Forbid
2. Check that the lock file prevents a second run
kubectl logs -n data-pipeline $(kubectl get pod -l job-name=milvus-ingest-20231109-02 -o jsonpath='{.items[0].metadata.name}')
2026-11-09T03:30:06.001Z - another ingest job is running, exiting.
3. Validate successful inserts
curl -s http://milvus:19121/api/v1/collections/movies/stats | jq '.row_count'
{
"row_count": 1500000
}
The row count reflects the expected increase after the batch run.
Prevent
- Use
concurrencyPolicy: Forbidfor any batch‑processing CronJob that touches shared state. - Implement an explicit lock (file, ConfigMap, or distributed lock) in the job script to guard against manual triggers.
- Monitor Milvus lock‑wait metrics (
milvus_write_lock_wait_seconds_total) and set alerts when the value exceeds a threshold. - Keep job duration < 80 % of the schedule interval to provide a safety margin.
- Version‑control CronJob manifests and require peer review for schedule changes.
Related Topic Hub: Vector Databases Troubleshooting Hub
FAQ
- Why does Milvus return
RESOURCE_EXHAUSTEDinstead of a timeout? The server treats a write‑lock acquisition failure as a resource exhaustion event, propagating the error to the client. - Can I rely solely on
concurrencyPolicy: Forbid? Yes for Kubernetes‑native scheduling, but external triggers (e.g.,kubectl create job) bypass the policy, so an application‑level lock is recommended. - What if the ingestion script crashes before removing the lock file? The
trapstatement ensures the lock is removed on container exit; for non‑graceful node failures, use a shortactiveDeadlineSecondson the Job to force cleanup. - Is increasing
write_lock_timeout_secondsa safe long‑term fix? It masks contention but does not eliminate it; use only as a temporary mitigation while you reduce job overlap. - How can I detect overlapping CronJobs without modifying the job? Query the Kubernetes API for active Jobs with the same label selector and compare
creationTimestampagainst the expected duration.