Intermittent OAuth2 Token Exchange Failure After Adding A/B Test Client IDs

Q: 1. Verify client configuration in Weaviate

# weaviate.yaml (excerpt) authentication: oauth2: enabled: true token_endpoint: "https://auth.example.com/oauth2/token" client_store: type: "redis" address: "redis://redis-prod:6379" clients: - client_id: "ab-test-variant-a" client_secret: "********" redirect_uris: - "https://variant-a.example.com/callback" - client_id: "ab-test-variant-b" client_secret: "********" redirect_uris: - "https://variant-b.example.com/callback" 2. Capture token requests with tcpdump # Capture only POST /oauth2/token traffic sudo tcpdump -i any -s 0 -w token.pcap 'tcp port 443 and dst host auth.example.com and ((tcp[((tcp[12] & 0xf0) >> 2):4] = 0x504f5354))' 3. Inspect the provider logs (example snippet) 2024-04-20T12:34:56Z WARN client_id=ab-test-variant-a redirect_uri="https://variant-a.example.com/callback/" mismatch expected="https://variant-a.example.com/callback"

Q: 4. Reproduce cache collision

Run two parallel curl commands that request a token for each variant:

Q: 5. Check rate‑limit counters on the provider

# Assuming the provider exposes Prometheus metrics curl -s http://auth.example.com/metrics | grep oauth2_token_requests_total # Example output oauth2_token_requests_total{client_id="ab-test-variant-a"} 1023 oauth2_token_requests_total{client_id="ab-test-variant-b"} 987 oauth2_token_requests_rate_limit_exceeded 1 Resolution The fix consists of two parts: isolate the client cache per variant and enforce strict, identical redirect URIs across environments.

Q: 1. Separate Redis namespaces for each client

Weaviate supports a key_prefix for the client store. Adding a unique prefix per variant prevents cache overwrites.

In this article

Table of Contents

Problem Description

During an A/B test of two client variants that access a Weaviate cluster, token acquisition from the OAuth2 provider becomes flaky. Approximately 10‑20 % of /token requests return HTTP 401 or 500, causing downstream API calls to be denied. The failure is intermittent: the same client ID can succeed in one request and fail in the next.

Typical log entries observed in weaviate.log:

OAuth2 token request failed for client_id=ab-test-variant‑b error=invalid_client

OAuth2 token request failed for client_id=ab-test-variant‑a error=invalid_grant

OAuth2 token request failed for client_id=ab-test-variant‑a error=server_error

Corresponding HTTP responses from the token endpoint:

401 Unauthorized – {"error":"invalid_client","error_description":"Client authentication failed"}
400 Bad Request – {"error":"invalid_grant","error_description":"Redirect URI mismatch"}
500 Internal Server Error – {"error":"server_error","error_description":"Unexpected condition"}

Root Cause Analysis

Weaviate’s OAuth2 integration follows the official OAuth2 Provider Integration guide. When multiple client IDs are configured, Weaviate stores a client metadata cache keyed by client_id. The cache is refreshed lazily and is shared across all request‑handling goroutines.

Two independent mechanisms trigger the intermittent failures:

Client‑ID cache collision – As documented in the 2023‑11‑15 outage, when A/B test variants share the same underlying token store (e.g., Redis) but use distinct client_id values, the cache update routine can overwrite the entry for one variant while another request is in flight. The next token request sees a stale or missing secret, resulting in invalid_client.
Redirect‑URI mismatch under concurrent loads – The OAuth2 spec requires the redirect_uri in the token request to match exactly the value registered for the client_id. In the 2024‑04‑20 case, a mis‑typed trailing slash in one variant caused invalid_grant only when the provider performed a strict string comparison. Because the two variants send requests to the same token endpoint, rate‑limiting on the provider (observed in the 2024‑02‑07 beta test) leads to occasional 500 responses when the request queue overflows.

Both issues stem from the interaction between Weaviate’s in‑process cache and the external OAuth2 provider’s strict validation rules.

Investigation and Debugging

The following steps reproduced the failure and isolated the cause.

1. Verify client configuration in Weaviate

# weaviate.yaml (excerpt)
authentication:
  oauth2:
    enabled: true
    token_endpoint: "https://auth.example.com/oauth2/token"
    client_store:
      type: "redis"
      address: "redis://redis-prod:6379"
    clients:
      - client_id: "ab-test-variant-a"
        client_secret: "********"
        redirect_uris:
          - "https://variant-a.example.com/callback"
      - client_id: "ab-test-variant-b"
        client_secret: "********"
        redirect_uris:
          - "https://variant-b.example.com/callback"

2. Capture token requests with `tcpdump`

# Capture only POST /oauth2/token traffic
sudo tcpdump -i any -s 0 -w token.pcap 'tcp port 443 and dst host auth.example.com and ((tcp[((tcp[12] & 0xf0) >> 2):4] = 0x504f5354))'

3. Inspect the provider logs (example snippet)

2024-04-20T12:34:56Z WARN client_id=ab-test-variant-a redirect_uri=”https://variant-a.example.com/callback/” mismatch expected=”https://variant-a.example.com/callback”

2024-04-20T12:35:02Z ERROR client_id=ab-test-variant-b authentication failed

4. Reproduce cache collision

Run two parallel curl commands that request a token for each variant:

# Variant A
curl -X POST https://auth.example.com/oauth2/token \
  -d "grant_type=authorization_code&code=CODE_A&redirect_uri=https://variant-a.example.com/callback&client_id=ab-test-variant-a&client_secret=SECRET_A"

# Variant B (run concurrently)
curl -X POST https://auth.example.com/oauth2/token \
  -d "grant_type=authorization_code&code=CODE_B&redirect_uri=https://variant-b.example.com/callback&client_id=ab-test-variant-b&client_secret=SECRET_B"

When the two requests overlap, the Redis cache entry for client_id is overwritten, and the second request receives invalid_client.

5. Check rate‑limit counters on the provider

# Assuming the provider exposes Prometheus metrics
curl -s http://auth.example.com/metrics | grep oauth2_token_requests_total
# Example output
oauth2_token_requests_total{client_id="ab-test-variant-a"} 1023
oauth2_token_requests_total{client_id="ab-test-variant-b"} 987
oauth2_token_requests_rate_limit_exceeded 1

Resolution

The fix consists of two parts: isolate the client cache per variant and enforce strict, identical redirect URIs across environments.

1. Separate Redis namespaces for each client

Weaviate supports a key_prefix for the client store. Adding a unique prefix per variant prevents cache overwrites.

# Before (single namespace)
client_store:
  type: "redis"
  address: "redis://redis-prod:6379"

# After (per‑variant prefixes)
client_store:
  type: "redis"
  address: "redis://redis-prod:6379"
  key_prefix: "oauth2"
clients:
  - client_id: "ab-test-variant-a"
    client_secret: "********"
    redis_key: "variant-a"
  - client_id: "ab-test-variant-b"
    client_secret: "********"
    redis_key: "variant-b"

Weaviate now writes oauth2:variant-a:… and oauth2:variant-b:… keys, eliminating collision.

2. Normalize redirect URIs

Update the registration on the OAuth2 provider to remove trailing slashes and ensure every variant uses the exact same string.

# Provider configuration (example JSON payload)
{
  "client_id": "ab-test-variant-a",
  "client_secret": "********",
  "redirect_uris": ["https://variant-a.example.com/callback"]
}
{
  "client_id": "ab-test-variant-b",
  "client_secret": "********",
  "redirect_uris": ["https://variant-b.example.com/callback"]
}

In the client code, construct the token request using a constant:

const redirectURI = "https://variant-a.example.com/callback"
params := url.Values{
    "grant_type":   {"authorization_code"},
    "code":         {authCode},
    "redirect_uri": {redirectURI},
    "client_id":    {"ab-test-variant-a"},
    "client_secret": {"SECRET_A"},
}
resp, err := http.PostForm(tokenEndpoint, params)

3. Increase provider rate limits for test traffic

Contact the OAuth2 provider team to raise the burst limit for the test client IDs from the default 5 rps to 20 rps. This prevents the 500 server_error observed during the 2024‑02‑07 beta test.

Validation

After applying the changes, run the following validation suite.

Functional token fetch

# Variant A
curl -s -w "\n%{http_code}\n" -X POST https://auth.example.com/oauth2/token \
  -d "grant_type=authorization_code&code=CODE_A&redirect_uri=https://variant-a.example.com/callback&client_id=ab-test-variant-a&client_secret=SECRET_A"
# Expected output: 200 and JSON with access_token

# Variant B (repeat)
curl -s -w "\n%{http_code}\n" -X POST https://auth.example.com/oauth2/token \
  -d "grant_type=authorization_code&code=CODE_B&redirect_uri=https://variant-b.example.com/callback&client_id=ab-test-variant-b&client_secret=SECRET_B"
# Expected output: 200

Load test for concurrency

# Using hey (HTTP load generator)
hey -n 1000 -c 50 -m POST -d @payload_a.json https://auth.example.com/oauth2/token
hey -n 1000 -c 50 -m POST -d @payload_b.json https://auth.example.com/oauth2/token
# Verify that error rate < 0.5 % and no 401/500 responses appear in the summary.

Log inspection

2024-08-27T09:12:34Z INFO OAuth2 token request succeeded for client_id=ab-test-variant-a

2024-08-27T09:12:35Z INFO OAuth2 token request succeeded for client_id=ab-test-variant-b

Prevention and Best Practices

Isolate client metadata caches. When using a shared external store (Redis, Memcached), configure distinct key prefixes or separate databases per client group.
Enforce canonical redirect URIs. Store the URI in a constant and validate it against the provider’s registration during CI.
Monitor token endpoint health. Create alerts on:
- Rate of invalid_client or invalid_grant responses > 1 %.
- Provider metric oauth2_token_requests_rate_limit_exceeded.
Rate‑limit client traffic. Use a client‑side token request limiter (e.g., token bucket) to stay below the provider’s burst capacity.
Version‑pin OAuth2 provider libraries. Ensure the same library version is used across all A/B variants to avoid subtle differences in request formatting.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does invalid_client appear only for one of the variants?
Because the shared Redis cache was overwritten by the concurrent request of the other variant, causing the provider to receive a mismatched secret.
Can I keep a single Redis instance without prefixes?
Yes, but you must use separate databases (e.g., redis://host:6379/1 and /2) or a namespaced key scheme to avoid collisions.
What is the correct way to handle trailing slashes in redirect_uri?
Register the URI exactly as it will be sent. Strip trailing slashes in the client code or add them consistently; the provider performs a strict string match.
How do I know if I’m hitting the provider’s rate limit?
Check the provider’s Prometheus metric oauth2_token_requests_rate_limit_exceeded or look for HTTP 429 responses. Increasing the burst limit or adding client‑side throttling resolves the issue.
Will the fix impact existing production clients?
No. The changes are scoped to the A/B test client IDs and the Redis key prefix. Existing clients continue to use the original namespace.