Weaviate CronJob Scheduling Conflict In Kubernetes

Q: Problem Description

In a production Weaviate deployment several CronJob resources are used:

Q: 1. Enforce non‑overlapping execution

Set concurrencyPolicy: Forbid on all Weaviate CronJobs. This prevents a new Job from starting while a previous one is still active.

Q: 2. Add a start deadline and backoff limits

Configure a reasonable startingDeadlineSeconds (e.g., 300 seconds) and a backoffLimit of 3 to avoid missed runs and runaway retries.

Q: 4. Apply changes via Helm (if applicable)

helm upgrade weaviate semi/weaviate \ --set cronjob.concurrencyPolicy=Forbid \ --set cronjob.startingDeadlineSeconds=300 \ --set cronjob.backoffLimit=3 \ -n weaviate Verification After applying the changes, perform the following checks:

In this article

Table of Contents

Problem Description

In a production Weaviate deployment several CronJob resources are used:

Nightly backup (02:00)
Index compaction (02:30)
Data synchronization (every 15 minutes)

Operators observed the following symptoms:

Backup logs contain "Failed to acquire lock for backup".
Kubernetes controller logs show "CronJob is already running" for the index job.
Occasional "Backoff limit exceeded" events after the backup window.
Node pressure alerts (“Pod was terminated: node under memory pressure”) during the overlap of sync and backup jobs.
Prometheus alerts “CronJob missed its scheduled time” during node maintenance.

These symptoms indicate that multiple CronJobs are colliding, causing overlapping executions, resource contention, and failed operations.

Root Cause Analysis

The overlapping behavior stems from three intertwined factors:

Concurrency policy defaults: By default spec.concurrencyPolicy is Allow, permitting a new Job to start even if the previous one has not finished. The Weaviate backup documentation recommends spacing schedules but does not enforce a policy.
Shared resources: Backup, index maintenance, and sync jobs all use the same ServiceAccount and the same internal Weaviate lock mechanism. When two jobs run concurrently, the first acquires the backup lock, and the second fails with "Failed to acquire lock for backup" (see the backup guide).
Insufficient start deadline: Some CronJobs were configured with startingDeadlineSeconds: 0, causing missed runs during node churn. The Kubernetes CronJob API reference warns that a zero deadline disables the deadline check, leading to “CronJob missed its scheduled time” alerts.

Combined, these issues let the 02:00 backup and the 02:30 index compaction overlap, while the 15‑minute sync job can start during the backup window, exhausting CPU/memory and triggering node pressure events.

Investigation and Debugging

The following steps reproduced the conflict and identified the misconfiguration:

Inspect CronJob definitions:

kubectl get cronjob weaviate-backup -o yaml
kubectl get cronjob weaviate-index-maintenance -o yaml
kubectl get cronjob weaviate-sync -o yaml

Typical output (excerpt):

apiVersion: batch/v1
kind: CronJob
metadata:
  name: weaviate-backup
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Allow          # ← problematic
  startingDeadlineSeconds: 0       # ← problematic
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: weaviate-sa
          containers:
          - name: backup
            image: semitechnologies/weaviate:1.19.0
            args: ["backup", "--path", "/backups"]

Check active Jobs and Pods at the conflict window:

kubectl get jobs --field-selector status.active=1 -A | grep weaviate
kubectl logs job/weaviate-backup-XXXXX -c backup

Sample backup log:

2024-06-10T02:01:12Z INFO Starting backup
2024-06-10T02:01:13Z ERROR Failed to acquire lock for backup
2024-06-10T02:01:13Z INFO Backup aborted

Sample index job log (running concurrently):

2024-06-10T02:30:05Z INFO Starting index compaction
2024-06-10T02:30:07Z ERROR Failed to acquire lock for backup
2024-06-10T02:30:07Z INFO Compaction aborted

Review node resource metrics during the overlap:

kubectl top nodes | grep 
kubectl describe node  | grep -i pressure

Output showed “memory pressure” events coinciding with the sync job start.

Validate Helm values (if deployed via Helm):

helm get values weaviate -n weaviate

The Helm chart README lists cronjob.concurrencyPolicy: Forbid as a recommended override, but the deployed values omitted it.

Resolution

The fix consists of three coordinated changes:

1. Enforce non‑overlapping execution

Set concurrencyPolicy: Forbid on all Weaviate CronJobs. This prevents a new Job from starting while a previous one is still active.

Before (excerpt from backup CronJob):

spec:
  concurrencyPolicy: Allow

After:

spec:
  concurrencyPolicy: Forbid

2. Add a start deadline and backoff limits

Configure a reasonable startingDeadlineSeconds (e.g., 300 seconds) and a backoffLimit of 3 to avoid missed runs and runaway retries.

spec:
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 3

3. Serialize access to the Weaviate lock via an init container

Introduce an init container that checks a ConfigMap‑based lock before the main container starts. This pattern is documented in the community issue “Weaviate data sync job collides with backup CronJob”.

Before (no init container):

containers:
- name: backup
  image: semitechnologies/weaviate:1.19.0
  args: ["backup", "--path", "/backups"]

After (added init container):

initContainers:
- name: lock-wait
  image: bitnami/kubectl:latest
  command: ["/bin/sh", "-c"]
  args:
    - |
      while kubectl get configmap weaviate-lock -n weaviate -o jsonpath='{.data.lock}' | grep -q "true"; do
        echo "Lock held, waiting...";
        sleep 10;
      done;
      kubectl patch configmap weaviate-lock -n weaviate -p '{"data":{"lock":"true"}}';
containers:
- name: backup
  image: semitechnologies/weaviate:1.19.0
  args: ["backup", "--path", "/backups"]
  lifecycle:
    preStop:
      exec:
        command: ["/bin/sh", "-c", "kubectl patch configmap weaviate-lock -n weaviate -p '{\"data\":{\"lock\":\"false\"}}'"]

The lock ConfigMap is created once in the namespace:

kubectl create configmap weaviate-lock --from-literal=lock=false -n weaviate

4. Apply changes via Helm (if applicable)

helm upgrade weaviate semi/weaviate \
  --set cronjob.concurrencyPolicy=Forbid \
  --set cronjob.startingDeadlineSeconds=300 \
  --set cronjob.backoffLimit=3 \
  -n weaviate

Verification

After applying the changes, perform the following checks:

Confirm CronJob spec updates:

kubectl get cronjob weaviate-backup -o jsonpath='{.spec.concurrencyPolicy}'
# Expected output: Forbid
kubectl get cronjob weaviate-backup -o jsonpath='{.spec.startingDeadlineSeconds}'
# Expected output: 300

Run a manual job to ensure lock acquisition works:

kubectl create job --from=cronjob/weaviate-backup test-backup -n weaviate
kubectl logs job/test-backup -c backup

Expected log snippet:

INFO Starting backup
INFO Backup completed successfully

Observe that overlapping runs are prevented:

# Trigger two jobs quickly
kubectl create job --from=cronjob/weaviate-backup backup1 -n weaviate
kubectl create job --from=cronjob/weaviate-backup backup2 -n weaviate
# The second job should stay pending with message:
# "CronJob is already running"

Check node pressure metrics after the next scheduled window:

kubectl top nodes
kubectl describe node  | grep -i pressure

No new “memory pressure” events should appear.

Prevention and Best Practices

Stagger CronJob schedules by at least the maximum runtime of any job. The backup guide recommends a minimum 30‑minute gap.
Use concurrencyPolicy: Forbid for any maintenance‑type CronJob that touches shared Weaviate resources.
Set startingDeadlineSeconds to a non‑zero value (e.g., 300 s) to guarantee that missed runs are logged and retried.
Isolate ServiceAccounts per job to avoid API rate‑limit collisions; the incident with “429 Too Many Requests” was caused by a shared account.
Implement a lock ConfigMap or external lock service (e.g., Redis) for critical sections when multiple CronJobs must not run concurrently.
Monitor CronJob health with Prometheus alerts on kube_cronjob_status_active and kube_job_failed metrics.
Document run‑time expectations in Helm values files so future maintainers can adjust schedules without risking overlap.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Q: Why does the backup sometimes succeed when the index job is running?

A: If the index job finishes before the backup attempts to acquire the lock, the backup proceeds. Overlap only causes failure when both processes need the lock simultaneously.

Q: Can I rely solely on concurrencyPolicy: Forbid without a lock mechanism?

A: Forbid prevents two Jobs of the same CronJob from overlapping, but it does not serialize different CronJobs. A shared lock (ConfigMap, Redis, etc.) is required to coordinate distinct CronJobs.

Q: What is a safe value for startingDeadlineSeconds?

A: Choose a value larger than the longest expected job runtime plus a safety margin. For Weaviate backups that typically run < 10 minutes, 300 seconds (5 minutes) is a common choice.

Q: How do I troubleshoot “Backoff limit exceeded” after a CronJob conflict?

A: Inspect the Job’s pod logs for lock errors, verify that backoffLimit is set, and ensure the lock ConfigMap is being cleared in the container’s preStop hook.

Q: Is it possible to run backup and sync jobs in parallel on separate nodes?

A: Yes, by using node affinity or pod anti‑affinity rules to schedule them on different nodes, you reduce resource contention. However, you still need a logical lock if both touch the same Weaviate data store.