Pinecone Job Completion Timeout During Large Batch Upserts

Q: 2. Scale the Index Pod

Upgrade from a single S1 pod to a 2‑pod M1 configuration, which provides 2× CPU and memory headroom. This aligns with the scaling recommendation for >500 k vectors (Scaling guide).

In this article

Table of Contents

Problem Description

A scheduled ETL script that upserts millions of embedding vectors into a Pinecone index fails with a timeout error. The job aborts after the client‑side timeout expires, leaving the index partially populated and the downstream recommendation service degraded.

Typical error messages observed in logs:


2026-07-15 02:14:33,721 ERROR pinecone.client.upsert: PineconeError: Upsert job exceeded timeout (status code 504)
2026-07-15 02:14:33,722 INFO  pinecone.client: Request payload size: 2.5M vectors (≈ 500 MB)
2026-07-15 02:14:33,723 WARN  pinecone.client: ResourceExhausted: Index capacity exceeded

The failure occurs consistently when the batch size exceeds roughly 800 k vectors or when the ingestion rate spikes after a model retraining.

Root Cause Analysis

The timeout originates from a combination of three intertwined factors:

Request size limits: The Pinecone Upsert endpoint imposes a default payload limit of ~500 MB and a server‑side processing timeout of 600 seconds (see the Upsert endpoint limits). Exceeding these limits forces the server to keep the request open longer than the client’s timeout parameter, resulting in a 504.
Insufficient pod resources: A 1‑pod S1 index provides a single CPU core and limited memory. When a large batch arrives, the pod hits 100 % CPU and throttles, producing the ResourceExhausted: Index capacity exceeded log entry (documented in the Scaling and resource allocation guide).
Single synchronous upsert: The script sends the entire vector set in one synchronous HTTP call. The Python client’s default client.timeout is 600 seconds; once the server cannot acknowledge completion within this window, the client aborts.

These conditions align with real incidents reported by the community: a nightly job that attempted a 2.5 M vector upsert timed out after 600 seconds and saturated the pod CPU (GitHub #342); a SaaS recommendation engine experienced ResourceExhausted followed by a timeout when batch size grew to 800 k vectors (Stack Overflow 76184523).

Investigation and Debugging Steps

Collect request and server logs


# Example snippet from CloudWatch / GCP Logging
2026-07-15T02:14:30.123Z  INFO  pinecone.upsert  RequestID=abc123 size=520MB vectors=2500000
2026-07-15T02:14:30.124Z  INFO  pinecone.pod    PodID=idx-01 CPU=95% MEM=88%
2026-07-15T02:20:30.125Z  WARN  pinecone.upsert  RequestID=abc123 timeout exceeded

Inspect client‑side timeout configuration


>>> import pinecone
>>> pinecone.init(api_key="...", environment="us-west1-gcp")
>>> client = pinecone.Index("my-index")
>>> client.describe_index_stats()
{'dimension': 1536, 'index_name': 'my-index', 'pods': 1, 'pod_type': 's1.x1'}

Measure pod utilization during ingestion (using kubectl top pod if running on a managed K8s cluster):
```
$ kubectl top pod idx-01
NAME    CPU(cores)   MEMORY(bytes)
idx-01   990m         3.8Gi
```

Validate batch size against documented limits (see Upsert endpoint limits):


# The docs state: max vectors per request = 1,000,000; max payload ≈ 500 MB.

Reproduce the failure locally with a reduced payload to isolate whether the timeout is client‑side or server‑side.

Resolution

The fix consists of three coordinated changes: chunk the payload, increase index resources, and adjust client timeout.

1. Chunked Upserts

Instead of a single massive request, split the vector list into smaller batches (e.g., 100 k vectors per request). The official Python client guide recommends this pattern (Batch upsert best practices).


# BEFORE: single massive upsert (fails)
vectors = [(f"id-{i}", embed[i]) for i in range(len(embed))]
client.upsert(vectors=vectors)  # ← 2.5M vectors, 600 s timeout


# AFTER: chunked upserts with retry logic
import math, time
BATCH_SIZE = 100_000
MAX_RETRIES = 3
timeout_seconds = 300  # client‑side timeout per batch

def chunked_upsert(vectors):
    total = len(vectors)
    batches = math.ceil(total / BATCH_SIZE)
    for i in range(batches):
        start = i * BATCH_SIZE
        end = min(start + BATCH_SIZE, total)
        batch = vectors[start:end]
        for attempt in range(1, MAX_RETRIES + 1):
            try:
                client.upsert(vectors=batch, timeout=timeout_seconds)
                break
            except Exception as e:
                if attempt == MAX_RETRIES:
                    raise
                time.sleep(2 ** attempt)  # exponential back‑off

2. Scale the Index Pod

Upgrade from a single S1 pod to a 2‑pod M1 configuration, which provides 2× CPU and memory headroom. This aligns with the scaling recommendation for >500 k vectors (Scaling guide).

Configuration	Pods	Pod Type	CPU per Pod	Memory per Pod
Current	1	S1	1 vCPU	4 GiB
Recommended	2	M1	2 vCPU	8 GiB

Scaling command (via Pinecone CLI or API):


# Using Pinecone CLI
pinecone index update my-index --pods 2 --pod-type m1.x1

3. Increase Client Timeout

Set a higher timeout for each chunk to accommodate occasional spikes.


client = pinecone.Index("my-index", timeout=300)  # 5 minutes per batch

Validation

After applying the three changes, verify success through the following checks:

Job completion log – No 504 entries should appear.


2026-07-16T03:02:12.001Z INFO  pinecone.upsert  RequestID=def456 size=95MB vectors=100000
2026-07-16T03:02:12.502Z INFO  pinecone.upsert  RequestID=def456 completed in 0.5s
...
2026-07-16T03:15:40.123Z INFO  pinecone.upsert  All 2.5M vectors upserted successfully

Index stats – Confirm the expected vector count.


>>> client.describe_index_stats()
{'total_vector_count': 2500000, 'dimension': 1536, 'pods': 2, 'pod_type': 'm1.x1'}

Pod utilization – CPU should stay below 80 % during ingestion.


$ kubectl top pod idx-01 idx-02
NAME    CPU(cores)   MEMORY(bytes)
idx-01   620m         4.2Gi
idx-02   580m         4.0Gi

Downstream service health – Verify that recommendation queries return expected latency (< 50 ms) and no missing vectors.

Prevention and Best Practices

Chunk size tuning: Start with 100 k vectors per batch; adjust based on observed pod CPU and network bandwidth.
Autoscaling: Enable Pinecone’s autoscaling feature for production indexes (set autoscaling.minPods and autoscaling.maxPods) to absorb ingestion spikes without manual intervention.
Monitoring: Alert on CPU > 80% and PineconeError: Upsert job exceeded timeout via your observability stack (Prometheus, CloudWatch).
Bulk import API: For one‑off loads > 500 k vectors, consider the bulk import endpoint (Community forum), which streams data directly from cloud storage and bypasses the per‑request payload limit.
Retry strategy: Implement exponential back‑off and idempotent upserts (use deterministic IDs) to handle transient throttling (HTTP 429) without data loss.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the timeout only appear after a model retraining?
Retraining usually generates a larger set of new embeddings. The sudden increase in vector count pushes the batch size beyond the default payload limit, causing the server to process longer than the client’s 600 s timeout.
Can I increase the server‑side timeout?
The server timeout is fixed at 600 seconds for the Upsert endpoint. The only way to avoid hitting it is to reduce request size (chunking) or increase index throughput (larger pods or autoscaling).
Is the bulk import API a replacement for regular upserts?
Bulk import is optimized for massive one‑time loads (> 500 k vectors) and bypasses the per‑request limit. For incremental or real‑time updates, continue using chunked upserts.
How do I know which pod type to choose?
Consult the Scaling guide. As a rule of thumb, if your ingestion rate exceeds 200 k vectors per minute, move from an S to an M pod or enable multiple pods.
What should I monitor to catch this issue early?
Track PineconeError: Upsert job exceeded timeout counts, pod CPU/Memory, and the rate of vectors ingested per minute. Set alerts on CPU > 80 % and on any 504/429 responses.