Milvus Cloud Service Returns 429 Error During High Query Load

Q: Root Cause Analysis

The Milvus managed service enforces a shared tenant request‑per‑second (RPS) ceiling of 1 000 (Milvus Cloud Service SLA and Usage Limits). The limit is applied globally across all tenants that share the same service tier. When the aggregate query rate exceeds this ceiling, the ingress gateway returns HTTP 429 as defined in the Milvus Server API Reference. The service does not provide per‑tenant isolation for the default tier, so a burst from any tenant contributes to the global counter.

Q: Investigation & Debugging Steps Confirm the 429 response source. Use curl to hit the /search endpoint and inspect the status code. curl -s -o /dev/null -w "%{http_code}\n" -X POST \ -H "Authorization: Bearer $TOKEN" \ -H "Content-Type: application/json" \ https://milvus.example.com/api/v1/search \ -d '{"vector": [...], "top_k": 10}' Expected output: 429 during the overload period. Check the current RPS metric. Query Prometheus or the built‑in /metrics endpoint. curl -s http://milvus.example.com/metrics | grep milvus_cloud_rate_limit_total # Example output: milvus_cloud_rate_limit_total{status="429"} 45 milvus_cloud_rate_limit_total{status="200"} 982 Inspect the client‑side request pattern. Enable DEBUG logging in the Milvus SDK. import logging logging.basicConfig(level=logging.DEBUG) from pymilvus import Collection, connections connections.connect("default", host="milvus.example.com", port="19530") # Run a high‑frequency search loop and observe the logs. Look for lines similar to the warning shown above. Verify the quota endpoint. Milvus Cloud exposes /quota to report the current limit. curl -H "Authorization: Bearer $TOKEN" \ https://milvus.example.com/api/v1/quota # Example JSON response: { "rps_limit": 1000, "rps_current": 1123, "tenant_id": "tenant-abc123" } Resolution – Align Query Load with the RPS Quota

The fix consists of two complementary actions: reduce the effective request rate and, if needed, request a higher quota from Milvus support.

In this article

Table of Contents

Problem – Milvus Cloud Service Returns HTTP 429 Under High Query Load

During a traffic spike the application receives responses such as:


HTTP/1.1 429 Too Many Requests
Content-Type: application/json
{
  "code": 429,
  "message": "Rate limit exceeded",
  "details": "RPS limit of 1000 exceeded"
}

or the Milvus Python SDK throws:


milvus.exceptions.ServerException: status=429, message='Rate limit exceeded, please retry later'

Log lines from the client side typically look like:


[WARN] request_id=7f9c2a3b rate_limit_exceeded current_rps=1123 limit=1000

Prometheus alerts fire on the metric milvus_cloud_rate_limit_total{status="429"} indicating sustained throttling.

Root Cause Analysis

The Milvus managed service enforces a shared tenant request‑per‑second (RPS) ceiling of 1 000 (Milvus Cloud Service SLA and Usage Limits). The limit is applied globally across all tenants that share the same service tier. When the aggregate query rate exceeds this ceiling, the ingress gateway returns HTTP 429 as defined in the Milvus Server API Reference. The service does not provide per‑tenant isolation for the default tier, so a burst from any tenant contributes to the global counter.

Key observations from the evidence package:

Fintech SaaS incident (Q1 2024) – spike in user searches saturated the 1 000 RPS shared limit.
E‑commerce promotion case (2023) – burst load after a flash sale caused intermittent 429 errors.
Internal post‑mortem (2024‑02) – a data‑migration script generated a sustained burst that triggered throttling before auto‑scaling took effect.

Therefore the root cause is exceeding the default global RPS quota, not a bug in Milvus itself.

Investigation & Debugging Steps

Confirm the 429 response source. Use curl to hit the /search endpoint and inspect the status code.


curl -s -o /dev/null -w "%{http_code}\n" -X POST \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  https://milvus.example.com/api/v1/search \
  -d '{"vector": [...], "top_k": 10}'

Expected output: 429 during the overload period.

Check the current RPS metric. Query Prometheus or the built‑in /metrics endpoint.


curl -s http://milvus.example.com/metrics | grep milvus_cloud_rate_limit_total
# Example output:
milvus_cloud_rate_limit_total{status="429"} 45
milvus_cloud_rate_limit_total{status="200"} 982

Inspect the client‑side request pattern. Enable DEBUG logging in the Milvus SDK.


import logging
logging.basicConfig(level=logging.DEBUG)
from pymilvus import Collection, connections

connections.connect("default", host="milvus.example.com", port="19530")
# Run a high‑frequency search loop and observe the logs.

Look for lines similar to the warning shown above.

Verify the quota endpoint. Milvus Cloud exposes /quota to report the current limit.


curl -H "Authorization: Bearer $TOKEN" \
  https://milvus.example.com/api/v1/quota
# Example JSON response:
{
  "rps_limit": 1000,
  "rps_current": 1123,
  "tenant_id": "tenant-abc123"
}

Resolution – Align Query Load with the RPS Quota

The fix consists of two complementary actions: reduce the effective request rate and, if needed, request a higher quota from Milvus support.

Client‑Side Throttling (Immediate Mitigation)

Introduce a token‑bucket limiter or exponential back‑off around each search call.


import time
import random
from pymilvus import Collection, connections, utility

MAX_RPS = 900  # Leave headroom below the 1000 limit
INTERVAL = 1.0 / MAX_RPS

last_ts = time.time()

def limited_search(collection, vector, top_k=10):
    global last_ts
    now = time.time()
    sleep_time = INTERVAL - (now - last_ts)
    if sleep_time > 0:
        time.sleep(sleep_time)
    last_ts = time.time()
    try:
        return collection.search(
            data=[vector],
            anns_field="embedding",
            param={"metric_type": "IP", "params": {"nprobe": 10}},
            limit=top_k,
        )
    except Exception as e:
        # Simple exponential back‑off for 429
        if getattr(e, "status", None) == 429:
            backoff = random.uniform(0.5, 2.0)
            time.sleep(backoff)
            return limited_search(collection, vector, top_k)
        raise

Before:


# Unthrottled loop – bursts of 2000 RPS
for vec in vectors:
    collection.search(...)

# Result: frequent 429 responses

After:


# Throttled loop – respects 900 RPS ceiling
for vec in vectors:
    limited_search(collection, vec)

# Result: 0 429 responses, stable latency

Requesting a Higher Quota (Long‑Term)

If the workload consistently requires > 1 000 RPS, open a support ticket referencing the Rate Limiting Guide. Provide:

Average and peak RPS measurements.
Projected growth.
Justification for dedicated tenant isolation.

Milvus Cloud can raise the limit or move the tenant to a dedicated tier where the quota is configurable per tenant.

Validation – Confirm the Issue Is Resolved

Functional test. Re‑run the high‑load script and assert that the HTTP status is always 200.
Metrics check. Verify that milvus_cloud_rate_limit_total{status="429"} remains at 0 for the duration of the test.
Latency monitoring. Ensure 95th‑percentile query latency stays within SLA (e.g., < 200 ms).


# Sample assertion script
import requests, time

def probe():
    resp = requests.post(
        "https://milvus.example.com/api/v1/search",
        json={"vector": [...], "top_k": 10},
        headers={"Authorization": f"Bearer {TOKEN}"}
    )
    return resp.status_code

statuses = [probe() for _ in range(1000)]
assert all(s == 200 for s in statuses), "Unexpected non‑200 status detected"

Prevention – Operational Guardrails

Guardrail	Implementation	Rationale
Static client‑side rate limiter	Token‑bucket set to 80 % of the documented limit (e.g., 800 RPS)	Absorbs traffic spikes without hitting the hard ceiling.
Prometheus alert on 429 spikes	`rate(milvus_cloud_rate_limit_total{status="429"}[1m]) > 5`	Early detection before user‑visible errors.
Capacity planning dashboard	Track `rps_current` from `/quota` alongside business traffic forecasts.	Proactive quota increase requests.
Batching & cache layer	Aggregate similar queries or cache recent results at the application tier.	Reduces the number of round‑trip searches.

FAQ – Common Follow‑Up Questions

Why does the 429 error appear only during bursts and not under steady load?
The service counts requests in a sliding one‑second window. A short burst can temporarily exceed the 1 000 RPS ceiling even if the long‑term average is lower.
Can I configure a per‑tenant RPS limit in the default tier?
No. The default tier shares a global limit across all tenants. To obtain a dedicated quota you must request a higher‑tier plan via Milvus support.
What back‑off strategy does the Milvus API recommend?
The official API reference suggests an exponential back‑off with jitter (e.g., base * 2^retry + random()) and to respect the Retry-After header if present.
Is there a way to query the current RPS usage programmatically?
Yes. The /quota endpoint returns rps_current and rps_limit in JSON format.
Will auto‑scaling of the Milvus cluster bypass the 429 limit?
No. Auto‑scaling adds more compute resources, but the rate‑limit is enforced at the ingress gateway before the request reaches any replica. Only a quota increase can raise the ceiling.

Related Topic Hub: Vector Databases Troubleshooting Hub

Problem – Milvus Cloud Service Returns HTTP 429 Under High Query Load