Problem
A live streaming service continuously sends audio/video frames to the Google Gemini API using the generateContent streaming endpoint. Every few hours the stream terminates with an authentication error similar to:
ERROR: 401 Unauthorized
detail: Invalid or expired OAuth2 token
The termination is abrupt: the client receives an HTTP 401, the underlying gRPC stream is closed, and no further frames are accepted until the application is restarted. This impacts real‑time processing pipelines, causing gaps in transcription and analysis.
Root Cause
The failure originates from the OAuth2 token refresh mechanism embedded in the client library. The sequence is:
- The client obtains an access token (
access_token) with a 1‑hour expiry and a refresh token (refresh_token). - During a long‑running Gemini stream the access token expires.
- The library attempts to refresh the token using the refresh endpoint.
- If the refresh request fails (e.g., network glitch, rate‑limit, or mis‑configured
client_secret), the library does not retry and propagates the error to the streaming RPC.
Because the streaming RPC does not have a built‑in re‑authentication hook, the 401 error bubbles up and the stream is closed. In production we observed that the refresh request sometimes returned HTTP 429 (Too Many Requests) due to the default token_uri being hit from many concurrent workers, causing the refresh to fail silently.
Debug
Below is a typical log excerpt from the streaming worker:
2024-11-10T03:12:45.123Z INFO streaming_worker.go:112 Starting Gemini streaming session (session_id=abc123)
2024-11-10T04:12:46.789Z WARN oauth2_token.go:58 Refresh token request failed: status=429, body="Rate limit exceeded"
2024-11-10T04:12:46.790Z ERROR streaming_worker.go:135 Stream terminated: rpc error: code = Unauthenticated desc = Invalid or expired OAuth2 token
Key investigation steps:
- Inspect token lifecycle: Verify
expires_inandexpiryfields in the token cache. - Capture HTTP traffic to the token endpoint:
sudo tcpdump -i eth0 -w /tmp/token_refresh.pcap host oauth2.googleapis.com and port 443
Decoded with tshark showed a 429 response after the 3600‑second mark.
- Check library version: Older
google-auth-library-goversions abort on any non‑200 response. - Review concurrency limits: Multiple workers share the same service account, causing burst refreshes.
Solution
The fix consists of three parts:
- Implement exponential back‑off and retry for token refresh failures.
- Introduce a shared token cache with a mutex to serialize refreshes across workers.
- Upgrade the authentication library to a version that respects
retrypolicies.
Before – token refresh implementation (v0.5.0)
func (c *Client) getAccessToken(ctx context.Context) (string, error) {
if c.token != nil && c.token.Valid() {
return c.token.AccessToken, nil
}
// Direct refresh without retry
resp, err := c.oauth2Config.TokenSource(ctx).Token()
if err != nil {
return "", fmt.Errorf("refresh token request failed: %w", err)
}
c.token = resp
return resp.AccessToken, nil
}
After – robust refresh with retry (v0.7.2)
func (c *Client) getAccessToken(ctx context.Context) (string, error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.token != nil && c.token.Valid() {
return c.token.AccessToken, nil
}
var resp *oauth2.Token
var err error
backoff := time.Second
for i := 0; i < 5; i++ {
resp, err = c.oauth2Config.TokenSource(ctx).Token()
if err == nil {
break
}
// Retry on 429 or network errors
if isRetryable(err) {
log.Printf("WARN: token refresh failed (%v), retry %d in %s", err, i+1, backoff)
time.Sleep(backoff)
backoff *= 2
continue
}
return "", fmt.Errorf("non‑retryable token refresh error: %w", err)
}
if err != nil {
return "", fmt.Errorf("exhausted token refresh retries: %w", err)
}
c.token = resp
return resp.AccessToken, nil
}
func isRetryable(err error) bool {
if urlErr, ok := err.(*url.Error); ok {
if strings.Contains(urlErr.Err.Error(), "429") {
return true
}
}
// Add other network‑error checks as needed
return false
}
Additional changes:
- Update
go.modto requiregoogle.golang.org/api v0.124.0(includes built‑in retry). - Configure a shared Redis cache for token storage to avoid duplicate refreshes across instances.
Redis token cache example
type TokenCache struct {
client *redis.Client
key string
}
func (tc *TokenCache) Get(ctx context.Context) (*oauth2.Token, error) {
data, err := tc.client.Get(ctx, tc.key).Result()
if err == redis.Nil {
return nil, nil
} else if err != nil {
return nil, err
}
var t oauth2.Token
if err := json.Unmarshal([]byte(data), &t); err != nil {
return nil, err
}
return &t, nil
}
func (tc *TokenCache) Set(ctx context.Context, t *oauth2.Token) error {
b, err := json.Marshal(t)
if err != nil {
return err
}
return tc.client.Set(ctx, tc.key, b, time.Until(t.Expiry)).Err()
}
Verify
After deploying the changes, perform the following checks:
- Start a streaming session and monitor the token expiry timeline.
- Force a token expiry by setting
expires_in=10in the initial token response (mocked). - Observe that the client retries the refresh, succeeds, and the stream continues without a 401.
Sample verification log:
2024-11-10T05:00:12.001Z INFO streaming_worker.go:112 Starting Gemini streaming session (session_id=def456)
2024-11-10T06:00:13.045Z WARN oauth2_token.go:78 token refresh failed (status=429), retry 1 in 1s
2024-11-10T06:00:14.050Z WARN oauth2_token.go:78 token refresh failed (status=429), retry 2 in 2s
2024-11-10T06:00:16.053Z INFO oauth2_token.go:92 token refreshed successfully
2024-11-10T06:00:16.054Z INFO streaming_worker.go:140 Stream resumed after token refresh
Additionally, query Redis to confirm a single token entry:
redis-cli GET gemini:service_account:token
Prevent
To avoid recurrence:
- Rate‑limit awareness: Keep the refresh interval well below the token TTL (e.g., refresh 5 minutes before expiry) and ensure only one refresh per service account per minute.
- Centralised token management: Use a dedicated authentication sidecar or a secret manager that handles refreshes and serves valid tokens to workers.
- Monitoring:
- Alert on
oauth2_token_refresh_failure_totalmetric exceeding a threshold. - Dashboard showing token expiry timestamps vs. refresh attempts.
- Alert on
- Testing: Include integration tests that simulate token expiry and 429 responses to verify retry logic.
Related Topic Hub: LLM Systems Troubleshooting Hub
FAQ
- Why does the stream survive a 401 if the token is refreshed manually?
The Gemini streaming RPC does not re‑authenticate mid‑stream. If the client obtains a new token and re‑issues the request, a new stream must be created. - Can I increase the token TTL to avoid refreshes?
No. Google OAuth2 access tokens have a fixed maximum lifetime of 3600 seconds. Only the refresh token can be used to obtain new access tokens. - Is the 429 response always due to rate limiting?
Typically yes, but it can also be triggered by quota exhaustion on the service account. Check the Cloud Console “OAuth 2.0 Token Exchange” quota. - Do I need to rotate the service‑account keys after fixing this?
Not required for the refresh issue, but rotating keys periodically remains a security best practice. - How does this differ when using the REST endpoint vs. gRPC?
The REST client library implements automatic token refresh with built‑in retries, while the gRPC client in older versions required manual handling as shown above.