GPT-4 controller manager crash during high-throughput inference

Problem Description

The controller manager that orchestrates the batch document‑summarization pipeline crashes under high‑throughput inference loads. Typical logs show a rapid succession of unhandled exceptions:

Traceback (most recent call last):
  File "/app/controller_manager.py", line 112, in run_batch
    responses = await asyncio.gather(*tasks)
  File ".../openai/api_resources/completion.py", line 78, in create
    raise openai.error.RateLimitError(message, http_status, headers)
openai.error.RateLimitError: Rate limit reached for token (HTTP 429)

In other runs the process is terminated by the OOM killer:

MemoryError: Cannot allocate memory
[1]   25437 killed     python controller_manager.py

Operational impact includes:

• Interrupted summarization jobs, requiring manual restarts.
• Loss of in‑flight request state because the controller process exits abruptly.
• Spikes in Kubernetes pod restarts, inflating deployment cost.

Root Cause Analysis

The crash stems from two intertwined failures:

1. Uncaught RateLimitError exceptions. The OpenAI API returns HTTP 429 with a Retry‑After header when the per‑minute token quota is exceeded (see OpenAI API Reference – Errors and Rate Limits). The controller manager uses asyncio.gather on a large list of tasks without individual try/catch blocks, so a single 429 propagates as an unhandled exception and terminates the event loop.
2. Unbounded request/response buffers. The pipeline builds a list of pending payloads and stores full responses in memory while awaiting rate‑limit back‑off. Under a burst of 10,000 parallel summarization requests (as observed in the internal telemetry logs), the in‑memory queue grows faster than the API can process, exhausting the pod’s RAM and triggering an OOM kill.

Both issues violate the expectations outlined in the OpenAI Cookbook – Handling Rate Limits and Transient Errors, which recommends per‑request retry logic and exponential back‑off, and the Production Best Practices guide, which advises streaming responses and limiting max tokens per request to control memory usage.

Investigation and Debugging

Step‑by‑step diagnostics that reproduced the failure:

1. Log inspection – Grep for HTTP 429 and QueueFull entries:

$ grep -E "429|QueueFull" controller.log
2024-07-28T12:03:14Z ERROR Rate limit reached for token (HTTP 429)
2024-07-28T12:03:14Z ERROR asyncio.exceptions.QueueFull: Queue size limit exceeded

2. Metric snapshot – Using kubectl top pod showed memory usage climbing from 400 MiB to >2 GiB before the pod was killed.
3. Heap dump analysis – A core dump revealed a list object holding >1 million pending request dictionaries, confirming unbounded queue growth.
4. Reproduction with a minimal script – The following snippet reproduces the unhandled exception when 500 concurrent calls are issued without retry handling:

import asyncio, openai

async def call_api(prompt):
    return await openai.ChatCompletion.acreate(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )

async def main():
    tasks = [call_api("Summarize doc {}".format(i)) for i in range(500)]
    await asyncio.gather(*tasks)   # <-- crashes on first 429

asyncio.run(main())

Running this script produced the same RateLimitError stack trace shown earlier.

Resolution

The fix addresses both exception handling and memory pressure:

1. Centralized retry wrapper with exponential back‑off

Introduce a reusable decorator that catches openai.error.RateLimitError, reads the Retry-After header, and retries with jitter.

# before: direct API call without retry
async def summarize(text):
    return await openai.ChatCompletion.acreate(
        model="gpt-4",
        messages=[{"role": "user", "content": text}],
        max_tokens=256
    )

# after: robust wrapper
import asyncio, random, time, openai
from openai.error import RateLimitError

def retry_on_rate_limit(max_retries=5, base_delay=1.0):
    def decorator(fn):
        async def wrapper(*args, **kwargs):
            delay = base_delay
            for attempt in range(max_retries):
                try:
                    return await fn(*args, **kwargs)
                except RateLimitError as e:
                    retry_after = e.headers.get("Retry-After")
                    if retry_after:
                        delay = float(retry_after)
                    else:
                        delay = delay * (2 + random.random())
                    await asyncio.sleep(delay)
            raise RuntimeError("Exceeded max retries for RateLimitError")
        return wrapper
    return decorator

@retry_on_rate_limit()
async def summarize(text):
    return await openai.ChatCompletion.acreate(
        model="gpt-4",
        messages=[{"role": "user", "content": text}],
        max_tokens=256,
        stream=False
    )

2. Bounded asyncio queue with back‑pressure

Replace the unbounded list of pending tasks with a limited asyncio.Queue. Workers pull from the queue, ensuring the number of inflight API calls never exceeds a configurable ceiling.

# before: accumulating tasks in a plain list
tasks = [summarize(chunk) for chunk in chunks]
await asyncio.gather(*tasks)

# after: queue‑driven producer/consumer pattern
MAX_INFLIGHT = 100   # tuned to stay within token quota

queue = asyncio.Queue(maxsize=MAX_INFLIGHT)

async def producer(chunks):
    for chunk in chunks:
        await queue.put(chunk)   # blocks when queue is full

async def consumer():
    while True:
        chunk = await queue.get()
        try:
            await summarize(chunk)
        finally:
            queue.task_done()

async def run_pipeline(chunks):
    prod = asyncio.create_task(producer(chunks))
    cons = [asyncio.create_task(consumer()) for _ in range(8)]
    await asyncio.gather(prod)
    await queue.join()
    for c in cons:
        c.cancel()

3. Streaming responses & token budgeting

Enable streaming to avoid buffering full responses in memory and limit max_tokens per request to the smallest acceptable value (e.g., 256). This follows the guidance in the OpenAI API Best Practices guide.

Validation

After deploying the patched controller manager, the following checks confirm stability:

• Log sanity – No longer see uncaught RateLimitError traces. Retry attempts are logged at INFO level.
• Memory profile – kubectl top pod shows steady memory usage around 600 MiB, even during a 10 k request burst.
• Successful completion – End‑to‑end batch run of 10 k documents finishes without pod restarts; total runtime aligns with expected back‑off schedule.
• Metrics – Prometheus counters for openai_api_requests_total and openai_api_rate_limit_retries reflect a controlled retry rate (≈2 % of total calls).

Prevention and Best Practices

Area	Recommendation
Rate‑limit handling	Always wrap OpenAI calls in a retry decorator that respects Retry‑After and uses exponential back‑off with jitter.
Concurrency control	Use bounded asyncio.Queue or semaphore to limit in‑flight requests to a safe ceiling (e.g., 100–200 depending on quota).
Memory management	Stream responses, cap max_tokens, and avoid storing full payloads in long‑lived collections.
Observability	Emit metrics for request rate, retry count, queue size, and memory usage; set alerts on queue‑full or OOM events.
Testing	Include load‑testing scripts that simulate burst traffic and verify graceful back‑pressure behavior before production rollout.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why does the controller crash only when the batch size exceeds a few thousand requests? The unbounded task list grows faster than the API can process, exhausting RAM. A bounded queue enforces back‑pressure, keeping memory usage constant.
2. Can I rely on the Retry‑After header alone for back‑off? Yes, per the OpenAI error documentation the header indicates the minimum wait time. Combining it with exponential jitter prevents thundering‑herd effects.
3. How should I choose the MAX_INFLIGHT value? Start with a value that stays comfortably below your per‑minute token quota (e.g., 100 requests for a 60 k token/min quota) and adjust based on observed latency and retry metrics.
4. Is streaming mandatory to avoid MemoryError? Streaming dramatically reduces peak memory because the client discards chunks as they arrive. It is strongly recommended for large responses, but bounding the queue alone also prevents OOM in most cases.
5. What monitoring alerts are most useful? Alert on:
   • Queue size approaching maxsize (e.g., >80%).
   • Rate‑limit retry rate exceeding a threshold (e.g., >5 % of calls).
   • Pod memory usage >80 % of its limit.

   These give early warning before crashes occur.