Google Gemini stream termination after OAuth2 token refresh failure

Problem

A live streaming service continuously sends audio/video frames to the Google Gemini API using the generateContent streaming endpoint. Every few hours the stream terminates with an authentication error similar to:


ERROR: 401 Unauthorized
detail: Invalid or expired OAuth2 token

The termination is abrupt: the client receives an HTTP 401, the underlying gRPC stream is closed, and no further frames are accepted until the application is restarted. This impacts real‑time processing pipelines, causing gaps in transcription and analysis.

Root Cause

The failure originates from the OAuth2 token refresh mechanism embedded in the client library. The sequence is:

  1. The client obtains an access token (access_token) with a 1‑hour expiry and a refresh token (refresh_token).
  2. During a long‑running Gemini stream the access token expires.
  3. The library attempts to refresh the token using the refresh endpoint.
  4. If the refresh request fails (e.g., network glitch, rate‑limit, or mis‑configured client_secret), the library does not retry and propagates the error to the streaming RPC.

Because the streaming RPC does not have a built‑in re‑authentication hook, the 401 error bubbles up and the stream is closed. In production we observed that the refresh request sometimes returned HTTP 429 (Too Many Requests) due to the default token_uri being hit from many concurrent workers, causing the refresh to fail silently.

Debug

Below is a typical log excerpt from the streaming worker:


2024-11-10T03:12:45.123Z INFO  streaming_worker.go:112 Starting Gemini streaming session (session_id=abc123)
2024-11-10T04:12:46.789Z WARN  oauth2_token.go:58 Refresh token request failed: status=429, body="Rate limit exceeded"
2024-11-10T04:12:46.790Z ERROR streaming_worker.go:135 Stream terminated: rpc error: code = Unauthenticated desc = Invalid or expired OAuth2 token

Key investigation steps:

  • Inspect token lifecycle: Verify expires_in and expiry fields in the token cache.
  • Capture HTTP traffic to the token endpoint:

sudo tcpdump -i eth0 -w /tmp/token_refresh.pcap host oauth2.googleapis.com and port 443

Decoded with tshark showed a 429 response after the 3600‑second mark.

  • Check library version: Older google-auth-library-go versions abort on any non‑200 response.
  • Review concurrency limits: Multiple workers share the same service account, causing burst refreshes.

Solution

The fix consists of three parts:

  1. Implement exponential back‑off and retry for token refresh failures.
  2. Introduce a shared token cache with a mutex to serialize refreshes across workers.
  3. Upgrade the authentication library to a version that respects retry policies.

Before – token refresh implementation (v0.5.0)


func (c *Client) getAccessToken(ctx context.Context) (string, error) {
    if c.token != nil && c.token.Valid() {
        return c.token.AccessToken, nil
    }
    // Direct refresh without retry
    resp, err := c.oauth2Config.TokenSource(ctx).Token()
    if err != nil {
        return "", fmt.Errorf("refresh token request failed: %w", err)
    }
    c.token = resp
    return resp.AccessToken, nil
}

After – robust refresh with retry (v0.7.2)


func (c *Client) getAccessToken(ctx context.Context) (string, error) {
    c.mu.Lock()
    defer c.mu.Unlock()

    if c.token != nil && c.token.Valid() {
        return c.token.AccessToken, nil
    }

    var resp *oauth2.Token
    var err error
    backoff := time.Second
    for i := 0; i < 5; i++ {
        resp, err = c.oauth2Config.TokenSource(ctx).Token()
        if err == nil {
            break
        }
        // Retry on 429 or network errors
        if isRetryable(err) {
            log.Printf("WARN: token refresh failed (%v), retry %d in %s", err, i+1, backoff)
            time.Sleep(backoff)
            backoff *= 2
            continue
        }
        return "", fmt.Errorf("non‑retryable token refresh error: %w", err)
    }
    if err != nil {
        return "", fmt.Errorf("exhausted token refresh retries: %w", err)
    }
    c.token = resp
    return resp.AccessToken, nil
}

func isRetryable(err error) bool {
    if urlErr, ok := err.(*url.Error); ok {
        if strings.Contains(urlErr.Err.Error(), "429") {
            return true
        }
    }
    // Add other network‑error checks as needed
    return false
}

Additional changes:

  • Update go.mod to require google.golang.org/api v0.124.0 (includes built‑in retry).
  • Configure a shared Redis cache for token storage to avoid duplicate refreshes across instances.

Redis token cache example


type TokenCache struct {
    client *redis.Client
    key    string
}

func (tc *TokenCache) Get(ctx context.Context) (*oauth2.Token, error) {
    data, err := tc.client.Get(ctx, tc.key).Result()
    if err == redis.Nil {
        return nil, nil
    } else if err != nil {
        return nil, err
    }
    var t oauth2.Token
    if err := json.Unmarshal([]byte(data), &t); err != nil {
        return nil, err
    }
    return &t, nil
}

func (tc *TokenCache) Set(ctx context.Context, t *oauth2.Token) error {
    b, err := json.Marshal(t)
    if err != nil {
        return err
    }
    return tc.client.Set(ctx, tc.key, b, time.Until(t.Expiry)).Err()
}

Verify

After deploying the changes, perform the following checks:

  1. Start a streaming session and monitor the token expiry timeline.
  2. Force a token expiry by setting expires_in=10 in the initial token response (mocked).
  3. Observe that the client retries the refresh, succeeds, and the stream continues without a 401.

Sample verification log:


2024-11-10T05:00:12.001Z INFO  streaming_worker.go:112 Starting Gemini streaming session (session_id=def456)
2024-11-10T06:00:13.045Z WARN  oauth2_token.go:78 token refresh failed (status=429), retry 1 in 1s
2024-11-10T06:00:14.050Z WARN  oauth2_token.go:78 token refresh failed (status=429), retry 2 in 2s
2024-11-10T06:00:16.053Z INFO  oauth2_token.go:92 token refreshed successfully
2024-11-10T06:00:16.054Z INFO  streaming_worker.go:140 Stream resumed after token refresh

Additionally, query Redis to confirm a single token entry:


redis-cli GET gemini:service_account:token

Prevent

To avoid recurrence:

  • Rate‑limit awareness: Keep the refresh interval well below the token TTL (e.g., refresh 5 minutes before expiry) and ensure only one refresh per service account per minute.
  • Centralised token management: Use a dedicated authentication sidecar or a secret manager that handles refreshes and serves valid tokens to workers.
  • Monitoring:
    • Alert on oauth2_token_refresh_failure_total metric exceeding a threshold.
    • Dashboard showing token expiry timestamps vs. refresh attempts.
  • Testing: Include integration tests that simulate token expiry and 429 responses to verify retry logic.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the stream survive a 401 if the token is refreshed manually?
    The Gemini streaming RPC does not re‑authenticate mid‑stream. If the client obtains a new token and re‑issues the request, a new stream must be created.
  2. Can I increase the token TTL to avoid refreshes?
    No. Google OAuth2 access tokens have a fixed maximum lifetime of 3600 seconds. Only the refresh token can be used to obtain new access tokens.
  3. Is the 429 response always due to rate limiting?
    Typically yes, but it can also be triggered by quota exhaustion on the service account. Check the Cloud Console “OAuth 2.0 Token Exchange” quota.
  4. Do I need to rotate the service‑account keys after fixing this?
    Not required for the refresh issue, but rotating keys periodically remains a security best practice.
  5. How does this differ when using the REST endpoint vs. gRPC?
    The REST client library implements automatic token refresh with built‑in retries, while the gRPC client in older versions required manual handling as shown above.