Claude Rolling Update Cron Overlap Causing Duplicate Model Refresh Tasks

Q: Root Cause – Staggered Pod Restarts Misalign CronSchedule Offsets

The Claude deployment uses a Deployment with maxSurge: 1 and maxUnavailable: 0. Each pod runs a side‑car container that launches a CronJob with the schedule 0 * * * * (hourly at minute 0). When a rolling update starts, the new pod inherits the same schedule, but the old pod is not terminated until the new pod reports readiness. If the readiness probe succeeds before the old pod’s cron trigger fires, both pods are alive at the same minute, causing the CronJob controller to schedule two identical jobs.

Q: Debug – Tracing the Overlap 1. Verify pod lifecycle during update

kubectl get pods -l app=claude -w # Sample output during update NAME READY STATUS RESTARTS AGE claude-frontend-4 1/1 Running 0 5d claude-frontend-5 0/1 Pending 0 0s claude-frontend-5 1/1 Running 0 10s claude-frontend-4 1/1 Running 0 5d Notice that claude-frontend-5 becomes Ready while claude-frontend-4 is still Running.

Q: 2. Inspect CronJob objects

kubectl get cronjobs model-refresh -o yaml Key fields:

Q: 3. Capture the exact time window of overlap

kubectl logs -l job-name=model-refresh-20231023-1400 -c model-refresh --since=2m # Output shows two pods logging the same run_id 4. Check readiness probe timings kubectl describe pod claude-frontend-5 | grep -A3 "Readiness" Readiness: http-get http://:8080/healthz delay=5s timeout=1s period=10s #success=1 #failure=3 With a 5 s initial delay and a 10 s period, the pod can become Ready within 15 s, well before the next minute tick.

Q: 3. Adjust Deployment rollout strategy

Set maxSurge: 0 and maxUnavailable: 1 so that the old pod is terminated before the new pod becomes Ready.

Q: Verify – Confirm Single Execution Per Window 1. Observe CronJob runs after applying changes

kubectl get jobs -l job-name=model-refresh -w # Expected output: only one job per hour NAME COMPLETION TIME model-refresh-20231023-1400 2026-10-23T14:00:12Z model-refresh-20231023-1500 2026-10-23T15:00:08Z 2. Check logs for duplicate run_id kubectl logs job/model-refresh-20231023-1400 -c model-refresh 2026-10-23T14:00:12.004Z INFO model_refresh: Starting model refresh (run_id=20231023-1400) 2026-10-23T14:00:45.321Z INFO model_refresh: Refresh completed successfully No second “dup2” entry appears.

Q: 3. Validate metric cronjob_overlaps returns zero

curl -s http://prometheus:9090/api/v1/query?query=cronjob_overlaps # Expected result: {"value":["0"]} for the last hour Prevent – Guardrails for Future Deployments Enforce concurrencyPolicy: Forbid on all maintenance CronJobs. This is a safe default for any job that mutates shared state. Implement a rollout policy that never runs two replicas of the same pod concurrently for jobs with identical schedules. Use maxSurge: 0 for such services. Introduce a health‑check annotation cronjob.kubernetes.io/last-run and a controller that aborts new runs if the previous run finished less than gracePeriodSeconds ago. Automate detection of overlapping CronJob executions. A Prometheus rule can fire when cronjob_active_jobs{job="model-refresh"} > 1. Document the jitter pattern in the CI/CD pipeline. All new CronJobs must include the initContainer jitter step unless they are explicitly marked as idempotent. Related Topic Hub: LLM Systems Troubleshooting Hub

In this article

Table of Contents

Problem – Duplicate Model Refresh Tasks During Rolling Updates

During a rolling update of the Anthropic Claude service, operators observed that the model-refresh CronJob executed twice for a single refresh window. The duplicate executions caused:

Increased CPU and memory pressure on the refreshed pods.
Conflicting writes to the shared model cache, resulting in occasional cache corruption.
Alert spikes on the cronjob_overlaps metric, triggering false positives.

Typical symptom in the logs:


2026-10-23T14:02:31.112Z WARN  cronjob_controller: Overlap detected for job model-refresh-20231023-1400 (instance: claude-frontend-5)
2026-10-23T14:02:31.115Z INFO  model_refresh: Starting model refresh (run_id=20231023-1400-dup1)
2026-10-23T14:02:31.119Z INFO  model_refresh: Starting model refresh (run_id=20231023-1400-dup2)

Root Cause – Staggered Pod Restarts Misalign CronSchedule Offsets

The Claude deployment uses a Deployment with maxSurge: 1 and maxUnavailable: 0. Each pod runs a side‑car container that launches a CronJob with the schedule 0 * * * * (hourly at minute 0). When a rolling update starts, the new pod inherits the same schedule, but the old pod is not terminated until the new pod reports readiness. If the readiness probe succeeds before the old pod’s cron trigger fires, both pods are alive at the same minute, causing the CronJob controller to schedule two identical jobs.

Key assumptions that failed:

Pods are assumed to be mutually exclusive for a given schedule.
The CronJob controller does not deduplicate jobs across multiple pod instances.
Readiness probes are faster than the cron trigger, creating a window of overlap.

Debug – Tracing the Overlap

1. Verify pod lifecycle during update


kubectl get pods -l app=claude -w
# Sample output during update
NAME                     READY   STATUS    RESTARTS   AGE
claude-frontend-4        1/1     Running   0          5d
claude-frontend-5        0/1     Pending   0          0s
claude-frontend-5        1/1     Running   0          10s
claude-frontend-4        1/1     Running   0          5d

Notice that claude-frontend-5 becomes Ready while claude-frontend-4 is still Running.

2. Inspect CronJob objects


kubectl get cronjobs model-refresh -o yaml

Key fields:

Field	Value
schedule	“0 * * * *”
concurrencyPolicy	“Allow”
startingDeadlineSeconds	30

The Allow policy permits concurrent runs, which is the direct enabler of duplication.

3. Capture the exact time window of overlap


kubectl logs -l job-name=model-refresh-20231023-1400 -c model-refresh --since=2m
# Output shows two pods logging the same run_id

4. Check readiness probe timings


kubectl describe pod claude-frontend-5 | grep -A3 "Readiness"
    Readiness:      http-get http://:8080/healthz delay=5s timeout=1s period=10s #success=1 #failure=3

With a 5 s initial delay and a 10 s period, the pod can become Ready within 15 s, well before the next minute tick.

Solution – Serialize Refresh Jobs Across Rolling Updates

1. Change `concurrencyPolicy` to `Forbid`

This prevents the controller from starting a new job if a previous instance is still active.


# Before (allow concurrent runs)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: model-refresh
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Allow
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: model-refresh
            image: anthroic/claude-refresh:latest
          restartPolicy: OnFailure


# After (forbid concurrent runs)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: model-refresh
spec:
  schedule: "0 * * * *"
  concurrencyPolicy: Forbid   # <-- changed
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: model-refresh
            image: anthroic/claude-refresh:latest
          restartPolicy: OnFailure

2. Add a staggered start offset using `startingDeadlineSeconds` and a pod‑level init container

Introduce a small random sleep (0‑30 s) before the CronJob starts, ensuring that even if two pods become Ready simultaneously, their schedules diverge.


# Patch the CronJob to include an init container
kubectl patch cronjob model-refresh --type=merge -p '
spec:
  jobTemplate:
    spec:
      template:
        spec:
          initContainers:
          - name: jitter
            image: busybox
            command: ["sh", "-c", "sleep $((RANDOM % 30))"]
          containers:
          - name: model-refresh
            # existing spec unchanged
'

3. Adjust Deployment rollout strategy

Set maxSurge: 0 and maxUnavailable: 1 so that the old pod is terminated before the new pod becomes Ready.


apiVersion: apps/v1
kind: Deployment
metadata:
  name: claude-frontend
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 0          # no extra pod
      maxUnavailable: 1   # one pod may be down

Verify – Confirm Single Execution Per Window

1. Observe CronJob runs after applying changes


kubectl get jobs -l job-name=model-refresh -w
# Expected output: only one job per hour
NAME                                 COMPLETION TIME
model-refresh-20231023-1400          2026-10-23T14:00:12Z
model-refresh-20231023-1500          2026-10-23T15:00:08Z

2. Check logs for duplicate run_id


kubectl logs job/model-refresh-20231023-1400 -c model-refresh
2026-10-23T14:00:12.004Z INFO model_refresh: Starting model refresh (run_id=20231023-1400)
2026-10-23T14:00:45.321Z INFO model_refresh: Refresh completed successfully

No second “dup2” entry appears.

3. Validate metric `cronjob_overlaps` returns zero


curl -s http://prometheus:9090/api/v1/query?query=cronjob_overlaps
# Expected result: {"value":["0"]} for the last hour

Prevent – Guardrails for Future Deployments

Enforce concurrencyPolicy: Forbid on all maintenance CronJobs. This is a safe default for any job that mutates shared state.
Implement a rollout policy that never runs two replicas of the same pod concurrently for jobs with identical schedules. Use maxSurge: 0 for such services.
Introduce a health‑check annotation cronjob.kubernetes.io/last-run and a controller that aborts new runs if the previous run finished less than gracePeriodSeconds ago.
Automate detection of overlapping CronJob executions. A Prometheus rule can fire when cronjob_active_jobs{job="model-refresh"} > 1.
Document the jitter pattern in the CI/CD pipeline. All new CronJobs must include the initContainer jitter step unless they are explicitly marked as idempotent.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does setting concurrencyPolicy: Forbid not affect other CronJobs? The policy is scoped per CronJob object. Only jobs that share the same name inherit the rule, so unrelated schedules remain unaffected.
Can I keep maxSurge: 1 and still avoid overlap? Yes, but you must add a coordination mechanism (e.g., a distributed lock in etcd) to ensure only one pod triggers the job. The jitter approach is simpler for most cases.
What if the model refresh takes longer than an hour? With Forbid, the next scheduled run will be skipped until the current job finishes. Adjust the schedule or split the refresh into smaller incremental steps.
How do I monitor for accidental re‑enabling of Allow? Add a Kubernetes admission controller that validates CronJob specs and rejects any with concurrencyPolicy: Allow for namespaces labeled team=claude.
Is the random jitter safe for compliance audits? Yes. The jitter only delays the start time; it does not change the job’s logic or output. Record the jitter seed in the job’s annotations if auditability is required.