Kubernetes cron overlap causing concurrent GPT-3.5 batch jobs

Kubernetes Cron Overlap Causing Concurrent GPT‑3.5 Batch Jobs

Problem

The nightly batch pipeline that invokes the OpenAI gpt‑3.5‑turbo model is implemented as a CronJob on a Kubernetes cluster. During several nights the pipeline failed with errors similar to:

2024-10-12T02:15:43Z ERROR rate limit exceeded for model gpt-3.5-turbo
2024-10-12T02:15:44Z INFO  Retrying request after backoff...
2024-10-12T02:16:01Z FATAL job aborted after 5 retries

Simultaneously, the cluster metrics showed CPU and memory spikes, and the kubectl get jobs output listed multiple active jobs for the same CronJob:

NAME                                 COMPLETIONS   DURATION   AGE
gpt-batch-20241012-0200-1            0/1           12m        12m
gpt-batch-20241012-0200-2            0/1           12m        12m

These overlapping executions caused:

• API rate‑limit violations on the OpenAI endpoint.
• Resource contention on the GPU‑enabled node pool.
• Increased cost due to duplicate work.

Root Cause

The CronJob was defined with a schedule: "0 2 * * *" (run at 02:00 UTC) and a concurrencyPolicy: Allow. The job’s runtime varied between 8 minutes and 20 minutes depending on the input size. When a previous run exceeded its expected duration, the next scheduled start created a second pod while the first was still active. Because concurrencyPolicy was set to Allow, Kubernetes launched the second job instead of skipping or replacing it.

Additional factors:

• The CronJob’s startingDeadlineSeconds was omitted, so missed starts were not reported.
• Pod termination grace period was 30 seconds, insufficient for the batch process to flush pending OpenAI requests.
• The OpenAI client library used exponential backoff, which amplified the rate‑limit errors when multiple pods hit the same endpoint simultaneously.

Debug

Step‑by‑step investigation:

1. Inspect CronJob definition

kubectl get cronjob gpt-batch -o yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: gpt-batch
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Allow
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: gpt-runner
            image: myrepo/gpt-batch:1.4
            envFrom:
            - secretRef:
                name: openai-credentials
          restartPolicy: OnFailure

Check active jobs at the time of failure

kubectl get jobs -l job-name=gpt-batch --field-selector status.active=1

NAME                                 ACTIVE   START TIME
gpt-batch-20241012-0200-1            1        2024-10-12T02:00:05Z
gpt-batch-20241012-0200-2            1        2024-10-12T02:00:07Z

Review pod logs for rate‑limit messages

kubectl logs -l job-name=gpt-batch-20241012-0200-1

2024-10-12T02:15:43Z ERROR rate limit exceeded for model gpt-3.5-turbo
...

Correlate with OpenAI usage metrics

OpenAI dashboard showed requests_per_minute spiking from 300 rpm (normal) to 1,200 rpm during the overlap window, exceeding the account’s limit of 1,000 rpm.

Validate job duration variance

kubectl get jobs -l job-name=gpt-batch -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.completionTime}{" "}{.status.startTime}{"\n"}{end}'

gpt-batch-20241012-0200-1 2024-10-12T02:18:30Z 2024-10-12T02:00:05Z
gpt-batch-20241012-0200-2 2024-10-12T02:22:45Z 2024-10-12T02:00:07Z

Both jobs ran >18 minutes, confirming overlap.

Solution

Three complementary changes were applied:

1. Enforce non‑overlapping execution

Set concurrencyPolicy: Forbid so that a new schedule is skipped if a previous run is still active.

2. Add a safety window with startingDeadlineSeconds

This fails the schedule if the controller cannot start the job within the defined deadline, surfacing missed runs.

3. Graceful shutdown handling

Increase terminationGracePeriodSeconds to allow the batch process to finish pending OpenAI calls.

Updated CronJob manifest (before / after):

# BEFORE (problematic)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: gpt-batch
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Allow
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: gpt-runner
            image: myrepo/gpt-batch:1.4
          restartPolicy: OnFailure

# AFTER (fixed)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: gpt-batch
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid          # Prevent overlap
  startingDeadlineSeconds: 300      # 5‑minute start window
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTemplate:
    spec:
      template:
        spec:
          terminationGracePeriodSeconds: 300   # 5 minutes for clean shutdown
          containers:
          - name: gpt-runner
            image: myrepo/gpt-batch:1.4
            envFrom:
            - secretRef:
                name: openai-credentials
          restartPolicy: OnFailure

4. Rate‑limit aware client

Wrap the OpenAI SDK call with a token bucket limiter (e.g., golang.org/x/time/rate) to cap requests at 900 rpm, staying below the account limit.

package main

import (
    "context"
    "time"

    "golang.org/x/time/rate"
    openai "github.com/sashabaranov/go-openai"
)

var limiter = rate.NewLimiter(rate.Every(time.Minute/900), 1) // 900 rpm

func callGPT(prompt string) (string, error) {
    ctx := context.Background()
    if err := limiter.Wait(ctx); err != nil {
        return "", err
    }
    client := openai.NewClient(os.Getenv("OPENAI_API_KEY"))
    resp, err := client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{
        Model: "gpt-3.5-turbo",
        Messages: []openai.ChatCompletionMessage{
            {Role: "user", Content: prompt},
        },
    })
    if err != nil {
        return "", err
    }
    return resp.Choices[0].Message.Content, nil
}

Verify

After applying the manifest and redeploying, the following checks confirm the fix:

1. Observe that only one job is active at any time.

kubectl get jobs -l job-name=gpt-batch -o wide

NAME                                 COMPLETIONS   DURATION   AGE
gpt-batch-20241013-0200-1            1/1           14m        14m

Check OpenAI usage dashboard – requests per minute stay under 800 rpm.

Confirm pod termination logs show graceful shutdown:

2024-10-13T02:14:55Z INFO shutting down, flushing pending requests...
2024-10-13T02:15:00Z INFO shutdown complete

Run a simulated overlap test by manually creating a second job while the first is running; the API returns:

kubectl create job --from=cronjob/gpt-batch test-overlap
Error from server (BadRequest): job "test-overlap" is invalid: spec.template.spec.containers[0].name: Invalid value: "gpt-runner": already running

Prevent

• Monitoring: Add a Prometheus alert on kube_job_status_active{job_name="gpt-batch"} > 1.
• Alerting: Trigger PagerDuty if OpenAI rate_limit_exceeded errors appear more than 3 times in a 5‑minute window.
• CI/CD guardrails: Lint CronJob manifests for concurrencyPolicy: Forbid and startingDeadlineSeconds using kube-linter or OPA Gatekeeper.
• Capacity planning: Size the GPU node pool to handle the worst‑case batch duration plus a 20 % buffer.
• Documentation: Record expected job runtime in the runbook; update the schedule if the pipeline’s data volume grows.

FAQ

1. Why does concurrencyPolicy: Allow cause rate‑limit errors?
   Because it permits multiple pods to invoke the OpenAI endpoint simultaneously, multiplying request throughput and easily exceeding the account’s RPM quota.
2. Can I keep Allow and still avoid overlap?
   Only by adding external coordination (e.g., a ConfigMap lock) or by configuring the job to exit early if a lock exists. Using Forbid is the native, simpler solution.
3. What is a reasonable value for startingDeadlineSeconds?
   It should be longer than the expected scheduling latency but shorter than the maximum acceptable delay for the batch. Five minutes (300 s) works for most nightly jobs.
4. How do I tune the token‑bucket limiter for different OpenAI plans?
   Set the limiter rate to (plan_limit_rpm * safety_factor). A safety factor of 0.9 provides a margin for occasional spikes.
5. My job still sometimes exceeds the schedule window. Should I increase the interval?
   If the workload regularly exceeds the allocated window, either increase the interval (e.g., run at 01:00 UTC) or split the batch into smaller parallel jobs with independent rate‑limit budgets.

Related Topic Hub: LLM Systems Troubleshooting Hub