tilt-shift photography of tree

Pinecone controller manager crash during multi-region replication

by

Problem Description

The controller manager pods in a multi‑region Pinecone deployment are crashing during synchronous index replication. Symptoms observed across us-east-1, eu-west-1 and ap-southeast-2 include:

  • Pod restarts with exit code 137 (OOM kill) in the Kubernetes events.
  • Log entries such as: 
    2024-05-28T14:12:03Z controller_manager: out of memory
2024-05-28T14:12:04Z replication sync failed: connection reset by peer
2024-05-28T14:12:05Z failed to acquire lock for index shard
  • Temporary index unavailability reported by the client SDK (HTTP 503).
  • High‑throughput workloads (≈10k upserts/sec) exacerbate the issue, matching the incident described in the Pinecone internal incident log.

Root Cause Analysis

The controller manager is responsible for coordinating shard metadata, lock acquisition, and cross‑region replication handshakes (see Pinecone Documentation – Multi‑Region Replication guide). Two intertwined factors trigger the crash:

  1. Memory pressure: During synchronous replication the manager buffers per‑shard metadata and pending write acknowledgments. When shard metadata exceeds ~8 GB (as observed in the “GC pause spikes” incident), the Java/Go runtime heap grows beyond the pod’s 2Gi limit, leading to OOM kills (controller_manager terminated with exit code 137).
  2. Synchronization timeouts: Network latency spikes between regions (e.g., us‑east‑1 ↔ ap‑southeast‑2) cause the replication handshake to exceed the default 30 s timeout. The manager logs “replication sync failed: connection reset by peer” and retries, further inflating memory usage.

Both conditions are documented in the official Controller manager health endpoints and the Troubleshooting guide (OOM detection, sync timeout errors).

Debugging Steps

Follow these steps to reproduce the failure and collect evidence:

# 1. Identify the controller manager pod
kubectl get pods -n pinecone -l app=controller-manager

# 2. Stream logs with timestamps
kubectl logs -n pinecone -l app=controller-manager -c controller-manager --tail=200 -f

# 3. Filter for OOM and sync errors
kubectl logs -n pinecone -l app=controller-manager -c controller-manager \
  | grep -E "out of memory|replication sync failed|failed to acquire lock"

# 4. Check pod resource usage at the time of crash
kubectl top pod -n pinecone $(kubectl get pod -n pinecone -l app=controller-manager -o jsonpath="{.items[0].metadata.name}")

# 5. Capture a core dump (if enabled) for post‑mortem analysis
kubectl exec -n pinecone $(kubectl get pod -n pinecone -l app=controller-manager -o jsonpath="{.items[0].metadata.name}") \
  -- bash -c "cat /var/log/pinecone/controller_manager.log" > controller_manager.log

# 6. Verify network latency between regions
ping -c 5 $(dig +short us-east-1.pinecone.internal)   # example endpoint

Solution

Apply the following changes to eliminate OOM conditions and reduce replication timeouts:

# 1. Increase the memory limit and request for the controller manager
# File: helm/pinecone/values.yaml (before)
controllerManager:
  resources:
    requests:
      cpu: "500m"
      memory: "1Gi"
    limits:
      cpu: "1"
      memory: "2Gi"

# File: helm/pinecone/values.yaml (after)
controllerManager:
  resources:
    requests:
      cpu: "750m"
      memory: "3Gi"
    limits:
      cpu: "2"
      memory: "4Gi"

# 2. Tune the replication timeout (default 30s) to 90s
# File: config/controller-manager.yaml (before)
replication:
  syncTimeoutSeconds: 30

# File: config/controller-manager.yaml (after)
replication:
  syncTimeoutSeconds: 90

# 3. Enable async replication for high‑throughput workloads (optional)
# File: config/replication-mode.yaml
mode: "async"   # switch from "sync" to "async"

# 4. Apply the updated Helm chart
helm upgrade pinecone ./helm/pinecone -n pinecone --reuse-values

# 5. Restart the controller manager pods to pick up new limits
kubectl rollout restart deployment/controller-manager -n pinecone

These adjustments raise the heap headroom, give the replication process more time to complete, and optionally move to asynchronous replication to smooth burst traffic.

Verification

After applying the fixes, run a controlled load test and verify that the controller manager remains healthy:

# 1. Start a 5‑minute upsert burst (10k ops/sec) using the Pinecone client
python upsert_burst.py --rate 10000 --duration 300

# 2. Monitor pod status and memory
kubectl get pods -n pinecone -l app=controller-manager -w
kubectl top pod -n pinecone -l app=controller-manager

# 3. Check for absence of OOM and sync errors
kubectl logs -n pinecone -l app=controller-manager -c controller-manager \
  | grep -E "out of memory|replication sync failed" || echo "No errors found"

# 4. Query the health endpoint
curl -s http://controller-manager.pinecone.svc.cluster.local:8080/healthz | jq .
# Expected output:
{
  "status":"ok",
  "replicationLatencyMs":45,
  "memoryUsageMiB": 2100
}

The pod should stay in Running state, memory usage should stay below the 4Gi limit, and the health endpoint must report "status":"ok" with latency well under the new 90 s timeout.

Prevention

To avoid recurrence:

  • Set memory requests/limits based on shard size forecasts (≥3 Gi per GB of shard metadata).
  • Enable HorizontalPodAutoscaler for the controller manager to scale out during traffic spikes.
  • Configure Prometheus alerts on container_memory_working_set_bytes > 80% of limit and on replication latency > 60 s.
  • Prefer asynchronous replication for workloads with sustained >5k upserts/sec, as recommended in the Multi‑Region Replication guide.
  • Regularly run pinecone index describe to monitor shard metadata growth and re‑shard indexes before they exceed 8 GB.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Q: Why does increasing the memory limit fix the OOM kill but not the sync timeout?

A: The OOM kill is caused by the heap exceeding the container’s limit, which is resolved by raising the limit. Sync timeouts are independent; they occur when network latency prevents the replication handshake from completing within the default 30 s. Adjusting syncTimeoutSeconds gives the process more time to finish.

Q: Can I keep synchronous replication and still handle 10k upserts/sec?

A: Yes, but you must provision enough memory (≥4 Gi) and ensure low inter‑region latency (<100 ms). Additionally, enable write rate limiting on the Index Management API (e.g., maxUpsertsPerSecond) to smooth bursts.

Q: What is the impact of switching to async replication?

A: Async replication decouples write acknowledgment from remote region sync, eliminating the tight timeout window. The trade‑off is eventual consistency across regions; reads may temporarily see stale data until the async pipeline catches up.

Q: How do I know if my shard metadata is approaching the 8 GB limit?

A: Use the Index Management API: GET /databases/{db}/indexes/{index}/metadata. The response includes shardMetadataSizeBytes. Set an alert when this value exceeds 6 GB to trigger a re‑sharding operation.

Q: Are there any Kubernetes-specific settings I should tune?

A: Enable the OOMScoreAdj to a lower value for the controller manager so the OOM killer prefers other pods, and configure terminationGracePeriodSeconds to allow graceful shutdown of replication sessions.