Weaviate 429 Rate Limit After Batch Vector Inserts During Traffic Spike

Q: Debug & Investigation

1. Inspect the HTTP response. The API returns a JSON payload that includes the remaining bucket tokens.

In this article

Table of Contents

Weaviate 429 Rate‑limit after batch vector inserts during traffic spike

Problem

An event‑driven pipeline receives bursts of data events, transforms each payload into a vector embedding, and writes the embeddings to a Weaviate instance using the /v1/batch/objects endpoint. During traffic spikes the pipeline starts returning HTTP 429 “Rate limit exceeded” responses, causing:

Back‑pressure on upstream microservices
Ingestion latency that grows from seconds to minutes
Thread pool exhaustion in the consumer service

Root Cause

Weaviate’s managed cloud offering enforces a per‑tenant request‑rate quota based on a token‑bucket algorithm. The quota is applied to all HTTP requests, not only to the batch endpoint. When the autoscaling group of the consumer service spins up additional pods, each pod starts sending its own batch of n objects (default batchSize = 100) every few seconds. The combined request rate can exceed the tenant’s maxRequestsPerSecond (often 50 rps for the default tier). Because the bucket refills at a fixed rate, the sudden burst depletes the tokens and the API returns 429 with a Retry‑After header.

Contributing factors:

Batch size too large for the allowed request‑rate.
No client‑side back‑off or retry logic; failed batches are retried immediately, further saturating the bucket.
Autoscaling based on CPU alone, not on outbound request rate, creates “thundering herd” bursts.

Debug & Investigation

1. Inspect the HTTP response. The API returns a JSON payload that includes the remaining bucket tokens.

curl -X POST https://my‑weaviate.cloud/v1/batch/objects \
  -H "Content-Type: application/json" \
  -d @payload.json -i

HTTP/1.1 429 Too Many Requests
Content-Type: application/json
Retry-After: 12
X-RateLimit-Limit: 50
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1698326400

{
  "error": {
    "message": "Rate limit exceeded",
    "code": 429
  }
}

2. Check Weaviate logs. Managed logs show the bucket state at the moment of rejection.

2024-10-26T14:32:07.123Z WARN  weaviate.rateLimiter - Token bucket empty for tenant abc123, request rejected (429)
2024-10-26T14:32:07.124Z DEBUG weaviate.rateLimiter - Bucket state: tokens=0, lastRefill=1698326400

3. Metric inspection. The weaviate_requests_total and weaviate_rate_limit_exceeded_total Prometheus counters spike in lockstep with the autoscaling event.

# HELP weaviate_requests_total Total HTTP requests received
# TYPE weaviate_requests_total counter
weaviate_requests_total{method="POST",endpoint="/v1/batch/objects"} 12457

# HELP weaviate_rate_limit_exceeded_total Number of 429 responses
# TYPE weaviate_rate_limit_exceeded_total counter
weaviate_rate_limit_exceeded_total{endpoint="/v1/batch/objects"} 342

4. Reproduce locally. Using the same batch size and concurrency on a dev tenant (limit 200 rps) reproduces the 429 after ~30 concurrent workers.

Solution

The fix consists of two complementary changes:

Client‑side throttling and retry. Introduce a token‑bucket limiter in the producer service that respects the X-RateLimit-Limit header and backs off according to Retry-After. Reduce the batch size to keep each request lightweight.
Server‑side quota adjustment. Request a higher rate‑limit tier from the managed provider or enable the batchWriteRate override (if the provider allows custom limits).

Before – unthrottled batch writer

import weaviate
import json
from concurrent.futures import ThreadPoolExecutor

client = weaviate.Client("https://my-weaviate.cloud")
batch_size = 100

def send_batch(objects):
    client.batch.create_objects(objects)  # fire‑and‑forget

def process_events(events):
    batches = [events[i:i+batch_size] for i in range(0, len(events), batch_size)]
    with ThreadPoolExecutor(max_workers=10) as pool:
        for batch in batches:
            pool.submit(send_batch, batch)

After – throttled writer with exponential back‑off

import weaviate, time, random
from threading import Semaphore
from concurrent.futures import ThreadPoolExecutor

client = weaviate.Client("https://my-weaviate.cloud")
batch_size = 50                # reduced
max_rps = 45                   # keep below tenant limit
semaphore = Semaphore(max_rps)  # token bucket approximation

def send_batch(objects):
    retry = 0
    while True:
        try:
            client.batch.create_objects(objects)
            break
        except weaviate.exceptions.UnexpectedStatusCodeException as e:
            if e.status_code == 429:
                retry_after = int(e.response.headers.get("Retry-After", "5"))
                backoff = retry_after * (2 ** retry) + random.uniform(0, 1)
                time.sleep(backoff)
                retry += 1
                continue
            raise

def process_events(events):
    batches = [events[i:i+batch_size] for i in range(0, len(events), batch_size)]
    with ThreadPoolExecutor(max_workers=8) as pool:
        for batch in batches:
            semaphore.acquire()               # acquire a token (rate‑limit)
            pool.submit(lambda b=batch: (send_batch(b), semaphore.release()))

Explanation:

Batch size lowered from 100 to 50 reduces request payload and processing time.
A Semaphore caps concurrent batch submissions to max_rps, staying under the provider’s limit.
On a 429 the client reads Retry-After, applies exponential back‑off, and retries the same batch.

If the workload cannot be accommodated within the existing quota, open a support ticket and request a higher maxRequestsPerSecond (e.g., 200 rps) or enable a dedicated “high‑throughput” tier.

Verification

After deploying the throttled writer:

Observe the weaviate_rate_limit_exceeded_total metric – it should remain at 0 during spikes.
Check the client logs for “Retry‑After” entries; there should be none after the first few seconds of warm‑up.
Measure end‑to‑end ingestion latency (event timestamp → vector searchable). It should stay < 2 s under peak load.
Confirm that the Retry‑After header is respected by inspecting a sample request:

2024-10-26T14:45:12.001Z INFO  batch_writer - Sent batch of 50 objects (req #312)
2024-10-26T14:45:12.045Z INFO  batch_writer - Received 200 OK

Prevention

Capacity planning. Align the tenant’s maxRequestsPerSecond with the expected peak batch rate (batchSize × workers ÷ seconds).
Dynamic throttling. Use a shared Redis token bucket across all pods so scaling out does not increase the aggregate request rate.
Monitoring & alerting. Create alerts on weaviate_rate_limit_exceeded_total and on the client‑side Retry‑After log pattern.
Back‑pressure propagation. When the writer hits a retry, signal upstream via a NACK or a Kafka “pause” to avoid unbounded queue growth.
Batch size tuning. Run a load test to find the maximum batch size that stays under the rate limit while meeting latency SLAs.

FAQ

Why does the 429 only appear during autoscaling events?
Because each new pod adds its own request stream. The token bucket refills at a fixed rate, so the combined burst exceeds the limit only when the number of concurrent workers spikes.
Can I disable the rate limit on the managed service?
No. Managed Weaviate enforces a tenant‑wide quota to protect shared resources. You can request a higher quota or move to a dedicated cluster.
Is the X-RateLimit-Remaining header reliable for client‑side throttling?
Yes. It reflects the current token count after the request. A value of 0 indicates the next request will be rejected, so the client should pause until Retry-After or until the bucket refills.
Should I use the bulk import endpoint instead of /v1/batch/objects?
The bulk import API bypasses the per‑request rate limit but is intended for one‑off data migrations. For continuous ingestion, throttling the regular batch endpoint is the supported pattern.
How do I calculate a safe max_rps value?
Determine the tenant’s maxRequestsPerSecond (e.g., 50). Subtract a safety margin (≈10 %). Then set max_rps = floor(limit × 0.9). Adjust batch size so (max_rps × batchSize) / seconds meets your ingestion throughput target.

Related Topic Hub: Vector Databases Troubleshooting Hub

Weaviate 429 Rate‑limit after batch vector inserts during traffic spike