a close-up of a server room

OpenAI GPT-3.5 API rate limit exceeded during parallel test runs

by

Problem Description

During automated model evaluation runs in a CI/CD pipeline, intermittent HTTP 429 (RateLimitError) responses are observed. The failures abort the test stage and block downstream deployment steps.

Typical console output from a Python test worker:

openai.error.RateLimitError: You exceeded your current quota, please check your plan and billing details.
Response payload:
{
  "error": {
    "message": "You have exceeded your requests per minute limit. Retry-After: 30",
    "type": "rate_limit_exceeded",
    "param": null,
    "code": null
  }
}

Similar messages appear in other runtimes:

429 Too Many Requests: You have exceeded your requests per minute limit. Retry-After: 30
Rate limit reached for organization org_12345: 60 RPM

The issue is reproducible only when the test suite runs with multiple parallel workers (e.g., pytest -n 8, GitHub Actions matrix, Jenkins parallel step). When the suite runs serially, the pipeline succeeds.

Root Cause Analysis

OpenAI enforces two primary throttling dimensions per organization:

  • RPM – Requests per minute (e.g., 60 RPM for the default GPT‑3.5 tier).
  • TPM – Tokens per minute (e.g., 150 K TPM).

These limits are shared across all API keys belonging to the same organization (Organization Quotas guide). The Retry-After header in the 429 response indicates how long the client must wait before the quota window resets (Rate Limits page).

When the CI pipeline launches N parallel jobs, each job issues M inference calls per minute. The aggregate request rate becomes N × M, which can easily exceed the organization‑level RPM limit. The same applies to token consumption: a burst of large prompts can push TPM over the threshold.

Key observations from real incidents:

Pipeline Parallel workers Calls/min per worker Aggregate RPM Quota
GitHub Actions matrix (10 jobs) 10 5 50 60 RPM (OK)
Same matrix with extra sanity checks 10 7 70 60 RPM (exceeded)
Jenkins parallel (8 Selenium tests) 8 5 40 60 RPM (OK)
Jenkins with added prompt regression (8 × 10) 8 10 80 60 RPM (exceeded)

Therefore, the intermittent 429 errors are a direct consequence of concurrent request bursts surpassing the organization’s RPM/TPM quotas.

Investigation and Debugging Steps

  1. Collect raw HTTP responses. Enable OPENAI_LOG=debug (Python SDK) or set debug:true in the Node client to dump headers: 
    2023-07-15T12:34:56.789Z openai: Request POST https://api.openai.com/v1/chat/completions
2023-07-15T12:34:56.891Z openai: Response 429
Headers:
  Retry-After: 30
  X-RateLimit-Limit-Requests: 60
  X-RateLimit-Remaining-Requests: 0
  2. Correlate logs with CI job IDs. Tag each request with a custom header (e.g., X-CI-Job-ID) to see which workers are contributing most to the burst.
  3. Inspect dashboard metrics. In the OpenAI usage dashboard, view the “Requests per minute” graph for the organization. Spikes align with pipeline start times.
  4. Reproduce locally. Run the same test command with --workers=1 and verify that no 429s appear.
  5. Check token consumption. Enable stream:true and log usage.total_tokens from the response to ensure TPM is not the limiting factor.

Resolution

The fix consists of two complementary strategies: throttling at the client side and adjusting CI concurrency. Below are concrete implementations for Python and Node.js.

Python – Centralized Retry & Backoff Middleware

Before: Direct SDK calls without handling rate limits.

import openai

def evaluate_prompt(prompt):
    resp = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt}]
    )
    return resp.choices[0].message.content

After: Wrap calls with tenacity for exponential backoff and respect Retry-After header.

import os, time
import openai
from tenacity import retry, wait_exponential, stop_after_delay, retry_if_exception_type
from openai.error import RateLimitError, APIError

def _extract_retry_after(exc):
    if isinstance(exc, RateLimitError) and exc.headers:
        return int(exc.headers.get("Retry-After", "1"))
    return None

def _rate_limit_backoff(retry_state):
    exc = retry_state.outcome.exception()
    wait = _extract_retry_after(exc)
    return wait if wait is not None else 2 ** retry_state.attempt_number

@retry(
    retry=retry_if_exception_type(RateLimitError),
    wait=_rate_limit_backoff,
    stop=stop_after_delay(180)
)
def evaluate_prompt(prompt):
    resp = openai.ChatCompletion.create(
        model="gpt-3.5-turbo",
        messages=[{"role": "user", "content": prompt}],
        request_timeout=30,
        headers={"X-CI-Job-ID": os.getenv("CI_JOB_ID", "local")}
    )
    return resp.choices[0].message.content

Explanation: The decorator retries only on RateLimitError, uses the server‑provided Retry-After when present, otherwise falls back to exponential backoff. This aligns with the Best Practices for High‑Throughput Applications.

Node.js – Concurrency Limiter with p-limit

Before: Unbounded parallel calls.

const { Configuration, OpenAIApi } = require("openai");

async function evaluate(prompt) {
  const resp = await openai.createChatCompletion({
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: prompt }],
  });
  return resp.data.choices[0].message.content;
}

After: Limit concurrent requests to MAX_CONCURRENCY (e.g., 3) and apply exponential retry.

const pLimit = require("p-limit");
const limit = pLimit(3); // matches safe RPM budget

async function evaluateWithRetry(prompt, attempt = 0) {
  try {
    const resp = await openai.createChatCompletion({
      model: "gpt-3.5-turbo",
      messages: [{ role: "user", content: prompt }],
    });
    return resp.data.choices[0].message.content;
  } catch (err) {
    if (err.response?.status === 429 && attempt < 5) {
      const retryAfter = parseInt(err.response.headers["retry-after"] || "1", 10);
      await new Promise(r => setTimeout(r, retryAfter * 1000));
      return evaluateWithRetry(prompt, attempt + 1);
    }
    throw err;
  }
}

// Usage in parallel test runner
const results = await Promise.all(prompts.map(p => limit(() => evaluateWithRetry(p))));

CI Pipeline Adjustments

  • Reduce the matrix size or --workers flag so that (workers × calls_per_minute) ≤ organization_RPM.
  • Introduce a shared “rate‑limit token bucket” via Redis or a simple file lock to serialize bursts across jobs.
  • If the workload legitimately exceeds the quota, request a higher limit via the OpenAI support portal.

Verification

  1. Run the pipeline with the new throttling code. Expect zero 429 entries in the job logs.
  2. Check the OpenAI dashboard after execution: the “Requests per minute” line should stay below the quota line (e.g., ≤ 60 RPM).
  3. Validate functional correctness: 
    pytest -n 4 --maxfail=1

    All tests should pass.

  4. Inspect a sample request header to confirm the custom X-CI-Job-ID is present and the retry logic respected Retry-After: 
    2023-07-15T12:35:10.123Z openai: Retry after 30 seconds for job 42

Prevention and Best Practices

  • Capacity planning: Before scaling parallelism, compute the expected aggregate RPM/TPM and compare against the organization limits.
  • Centralized rate‑limit handling: Implement a reusable wrapper (as shown) that all services import.
  • Metrics & alerts: Emit a custom Prometheus metric openai_rate_limit_hits_total and set an alert when it spikes.
  • Separate API keys per job: For large matrix builds, assign distinct keys tied to sub‑organizations with independent quotas (see Organization Quotas guide).
  • Batch prompts: Where possible, combine multiple test cases into a single request (e.g., using n=5 with temperature=0) to reduce request count.
  • Graceful degradation: If the quota is exhausted, fallback to cached model outputs or skip non‑critical tests rather than aborting the whole pipeline.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the 429 error appear only intermittently? The API enforces a sliding window. When the burst of parallel calls crosses the limit, the server rejects the excess requests. Subsequent requests after the Retry-After window succeed, leading to intermittent failures.
  2. Can I increase the RPM/TPM limits? Yes. OpenAI allows quota upgrades on request. Submit a support ticket with expected usage patterns; they may raise the limits for your organization.
  3. Is the Retry-After header always present? According to the Rate Limits page, a Retry-After header is included on 429 responses. Your client should handle its absence by falling back to exponential backoff.
  4. Do multiple API keys share the same quota? All keys belonging to the same organization share the organization‑level RPM/TPM quota (Organization Quotas guide). Separate sub‑organizations or separate billing accounts are required for isolated quotas.
  5. What is the recommended backoff strategy? The official guidance suggests exponential backoff with jitter, respecting the Retry-After header when present. The tenacity example above follows this recommendation.