Gemini RAG pipeline connection timeout to vector store after autoscaling

Problem: Gemini RAG pipeline cannot connect to the vector store after autoscaling

The Gemini Retrieval‑Augmented Generation (RAG) pipeline is deployed on a managed cloud service that scales compute instances automatically. After a scale‑out event, the pipeline consistently fails to retrieve documents from the configured managed vector store. The failure manifests as connection timeouts and, intermittently, authentication errors.

Root Cause

The timeout originates from a combination of three factors that become visible only when new instances are launched:

• Network policy propagation delay: The cloud VPC’s security group rules that allow outbound traffic to the vector store are applied asynchronously. New pods inherit a default deny‑all egress policy until the policy controller reconciles the NetworkPolicy object, causing the first few seconds of the pod’s lifecycle to have no outbound connectivity.
• IAM role binding race condition: The managed vector store authenticates callers via short‑lived service‑account tokens. Autoscaled instances receive their workload identity token from the metadata server, but the token is cached locally. When the pod starts before the token is refreshed, the request is signed with an expired token, leading to 401 Unauthorized responses.
• Connection pool exhaustion: The Gemini client library creates a fixed‑size HTTP/2 connection pool (default max_connections=5). During a rapid scale‑out, many instances simultaneously attempt to open connections, exhausting the pool on the vector store side and causing subsequent connection attempts to block until the server’s accept queue overflows, resulting in timeouts.

These three issues compound: the first request fails due to missing egress, the second fails due to an expired token, and subsequent retries hit the exhausted pool, producing the observed pattern of deadline exceeded and authentication failed logs.

Debug: Investigation Steps

1. Capture the failing logs

2024-09-12T14:03:21.842Z ERROR GeminiRAG[instance-7] Retrieval failed: rpc error: code = Unavailable desc = connection timeout (deadline exceeded)
2024-09-12T14:03:22.013Z WARN  GeminiRAG[instance-7] Authentication error: token expired, refreshing...
2024-09-12T14:03:22.527Z ERROR GeminiRAG[instance-7] Retrieval failed: rpc error: code = Unavailable desc = connection timeout (deadline exceeded)

2. Verify network egress from a newly created pod

kubectl exec -n rag-pipeline -it $(kubectl get pod -l app=gemini-rag -o jsonpath='{.items[0].metadata.name}') -- curl -s -o /dev/null -w "%{http_code}" https://vectorstore.example.com/healthz

Expected output: 200. Observed output on fresh pods: 000 (no response), confirming egress blockage.

3. Inspect IAM token timestamps

curl "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token" \
  -H "Metadata-Flavor: Google" | jq '.expiration_time'

If the timestamp is in the past, the token is stale.

4. Check vector store connection metrics

# Prometheus query (vectorstore_server_active_connections)
sum by (instance) (vectorstore_server_active_connections)

During a scale‑out burst, the metric spikes to the configured limit (e.g., 500) and then plateaus, while new connection attempts are rejected.

5. Review the NetworkPolicy definition

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-vectorstore-egress
spec:
  podSelector:
    matchLabels:
      app: gemini-rag
  policyTypes:
  - Egress
  egress:
  - to:
    - ipBlock:
        cidr: 10.128.0.0/20   # Vector store subnet
    ports:
    - protocol: TCP
      port: 443

Confirm that the policy controller logs show a delay in applying this policy to new pods.

Solution: Apply a coordinated fix

1. Add a startup probe that blocks the pod from becoming Ready until egress is verified

containers:
- name: gemini-rag
  image: gcr.io/project/gemini-rag:latest
  readinessProbe:
    exec:
      command:
      - /bin/sh
      - -c
      - |
        curl -s -o /dev/null -w "%{http_code}" https://vectorstore.example.com/healthz | grep -q 200
    initialDelaySeconds: 5
    periodSeconds: 2
    failureThreshold: 30

This ensures the pod does not receive traffic until the NetworkPolicy is enforced.

2. Enable token auto‑refresh in the Gemini client library

Before (static token fetch):

# gemini_client.py (before)
token = requests.get(
    "http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token",
    headers={"Metadata-Flavor": "Google"}
).json()["access_token"]
client = GeminiRAGClient(auth_token=token)

After (dynamic token provider):

# gemini_client.py (after)
import google.auth
import google.auth.transport.requests

def get_credentials():
    credentials, _ = google.auth.default()
    request = google.auth.transport.requests.Request()
    credentials.refresh(request)
    return credentials.token

class AutoRefreshAuth:
    def __call__(self):
        return get_credentials()

client = GeminiRAGClient(auth_token_provider=AutoRefreshAuth())

The client now retrieves a fresh token for each request, eliminating stale‑token errors.

3. Increase the vector store’s max concurrent connections and enable connection pooling on the client

Vector store configuration (increase from 500 to 2000):

# vectorstore-config.yaml
server:
  max_concurrent_connections: 2000

Gemini client pool adjustment:

# gemini_client.py (pool config)
from google.api_core import client_options

client_opts = client_options.ClientOptions(
    api_endpoint="vectorstore.example.com:443",
    client_info={"grpc.max_connection_age_ms": 300000},
    max_pool_size=20
)
client = GeminiRAGClient(client_options=client_opts)

4. Apply a short “warm‑up” init container to pre‑populate the connection pool

initContainers:
- name: warmup-connection
  image: curlimages/curl:7.85.0
  command: ["sh", "-c"]
  args:
  - |
    for i in $(seq 1 10); do
      curl -s -o /dev/null https://vectorstore.example.com/healthz
    done

This forces the first few connections to be established before the main container starts serving traffic.

Verify: Confirm the fix works

1. Deploy the updated manifest and watch pod readiness:

kubectl rollout restart deployment/gemini-rag -n rag-pipeline
kubectl get pods -w -n rag-pipeline

All new pods should reach Ready state within ~10 seconds.

2. Re‑run the curl health check from a pod:

kubectl exec -n rag-pipeline -it $(kubectl get pod -l app=gemini-rag -o jsonpath='{.items[0].metadata.name}') -- curl -s -w "%{http_code}" https://vectorstore.example.com/healthz

Output should be 200 consistently.

3. Trigger a generation request that uses RAG and observe logs:

2024-09-12T14:45:07.321Z INFO  GeminiRAG[instance-9] Retrieval succeeded: 12 documents returned
2024-09-12T14:45:07.322Z INFO  GeminiRAG[instance-9] Generation completed in 1.23s

No timeout or authentication errors should appear.

4. Check vector store connection metrics to ensure they stay below the new limit:

sum by (instance) (vectorstore_server_active_connections) < 1500

Prevent: Operational safeguards

• NetworkPolicy readiness gating: Keep the readiness probe that validates egress before marking pods ready.
• Token refresh monitoring: Export a custom metric for token age and alert if the age exceeds 5 minutes.
• Connection pool sizing: Align the client pool size with the vector store’s max_concurrent_connections and leave a 30 % headroom.
• Autoscaling ramp‑up limits: Configure the Horizontal Pod Autoscaler’s scaleUpLimitFactor to avoid sudden spikes that could saturate the vector store.
• Warm‑up init containers: Use them for any external dependency that requires TCP handshakes before traffic arrives.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why do only the first few pods after a scale‑out fail? The NetworkPolicy controller applies egress rules asynchronously, and the IAM token cache is populated only after the first successful metadata request. Both conditions are transient and affect only pods that start before they are satisfied.
2. Can I disable the connection pool limit on the vector store? The limit protects the service from overload. Instead, increase the limit to a value that accommodates peak autoscaling bursts and ensure the client pool is sized accordingly.
3. Is there a way to pre‑fetch IAM tokens for new pods? Yes. Use a sidecar that calls the metadata server on pod start and writes the token to a shared volume, or configure the client library to request a token on each request as shown in the solution.
4. Do I need to adjust firewall rules when moving to a different region? Absolutely. NetworkPolicy CIDR blocks must match the vector store’s subnet in the target region; otherwise egress will be blocked, reproducing the same timeout symptoms.
5. How can I detect a similar issue before it impacts users? Create a synthetic canary that periodically performs a RAG retrieval. Alert on any deadline exceeded or 401 Unauthorized responses, and on spikes in vectorstore_server_active_connections near the configured limit.