Pinecone Token Limit Exceeded During Automated Model Evaluation Runs

Q: Before (problematic code)

import pinecone pinecone.init(api_key="YOUR_KEY", environment="us-west1-gcp") index = pinecone.Index("embeddings") # Batch size 10,000 vectors, dimension 2048 vectors = [(f"id-{i}", np.random.rand(2048).astype(np.float32).tolist()) for i in range(10000)] index.upsert(vectors=vectors, namespace="eval") After (chunked upserts) import pinecone import math pinecone.init(api_key="YOUR_KEY", environment="us-west1-gcp") index = pinecone.Index("embeddings") BATCH_SIZE = 2000 # Chosen to keep payload < 4 MiB DIMENSION = 2048 def chunked_upsert(vectors): total = len(vectors) for i in range(0, total, BATCH_SIZE): batch = vectors[i:i + BATCH_SIZE] # Optional: log estimated size for sanity check size_mb = request_size(len(batch), DIMENSION) / (1024 * 1024) print(f"Upserting batch {i // BATCH_SIZE + 1}: {len(batch)} vectors ({size_mb:.2f} MiB)") index.upsert(vectors=batch, namespace="eval") vectors = [(f"id-{i}", np.random.rand(DIMENSION).astype(np.float32).tolist()) for i in range(10000)] chunked_upsert(vectors) For queries, reduce top_k or split large query sets into multiple calls:

Q: Preventive Guard Clause Example

MAX_REQUEST_MB = 3.5 # Leave headroom below 4 MiB def compute_max_batch(dim, metadata_bytes=0): # Solve for N: (N * dim * 4 + N * 50 + metadata_bytes) / (1024*1024)

In this article

Table of Contents

Problem Description

During automated model evaluation pipelines that retrieve and compare vector embeddings stored in Pinecone, the service intermittently returns errors such as:

PineconeError: token limit exceeded – request contains 6.2 MiB, limit is 4 MiB
Request size exceeds maximum allowed (4 MiB) – token limit exceeded
Upsert failed: token limit exceeded for batch of 10,000 vectors
Query failed: token limit exceeded – reduce top_k or batch size

These failures abort the evaluation run, leaving similarity scores and downstream metrics incomplete. The issue appears both on upsert and query operations, especially after recent changes that increased vector dimensionality (e.g., from 1536‑dim to 2048‑dim) or added raw text payloads to the metadata.

Root Cause Analysis

Pinecone enforces a strict per‑request token limit (4 MiB for upserts and queries). The limit is calculated from the total number of bytes required to serialize the vectors and any accompanying metadata. When the combined payload exceeds this threshold, Pinecone returns the token limit exceeded error.

Upsert path: Each float32 component occupies 4 bytes. A batch of N vectors of dimension D therefore consumes N × D × 4 bytes plus overhead for IDs and metadata. In the incident “Automated model evaluation job that retrieves 5 000 embeddings per iteration sent a single upsert request (~6 MiB)”, the payload size alone violated the 4 MiB cap.
Query path: The query request includes the query vector (same size calculation) and optional includeMetadata fields. Adding raw text payloads (e.g., a 200‑byte snippet per vector) can push the request over the limit, as observed in the nightly evaluation pipeline that combined OpenAI embeddings (1536‑dim) with raw text.
Recent code change: Increasing the embedding dimension from 1536 to 2048 added 2048 × 4 = 8192 bytes per vector, which, when combined with a batch size of 2 000, resulted in a request size of ≈16 MiB—well beyond the allowed quota.

The official Pinecone documentation (Pinecone API Reference – Request Size Limits) explicitly states the 4 MiB per‑request ceiling, and community reports (GitHub #312, Stack Overflow 78543219) confirm that exceeding this limit triggers the same error messages.

Investigation and Debugging Steps

Reproduce the error locally using a minimal script that mirrors the production batch size and vector dimension.

Calculate request size before sending:

import sys
import numpy as np

def request_size(num_vectors, dim, metadata_bytes=0):
    # 4 bytes per float32 component
    vector_bytes = num_vectors * dim * 4
    # Approximate overhead: 50 bytes per vector for ID + JSON framing
    overhead = num_vectors * 50
    return vector_bytes + overhead + metadata_bytes

size_mb = request_size(2000, 2048) / (1024 * 1024)
print(f"Estimated request size: {size_mb:.2f} MiB")

Typical output:

Estimated request size: 16.00 MiB

Inspect Pinecone client logs. Enable debug logging in the Python client:

import logging
logging.basicConfig(level=logging.DEBUG)

Look for lines such as:

DEBUG pinecone.http: Request payload size 6.2 MiB exceeds limit 4 MiB

Capture network traffic (optional) to verify payload size:

sudo tcpdump -i any -s 0 -w pinecone_upsert.pcap port 443

Then inspect with tshark:

tshark -r pinecone_upsert.pcap -Y http2.data -T fields -e http2.data_len | awk '{sum+=$1} END {print sum/1024/1024 " MiB"}'

Check Pinecone quota dashboards for “Request size” alerts that correlate with the evaluation run timestamps.

Resolution

The fix consists of ensuring every upsert and query request stays under the 4 MiB limit. The most reliable approach is to chunk the payload into smaller batches.

Before (problematic code)

import pinecone

pinecone.init(api_key="YOUR_KEY", environment="us-west1-gcp")
index = pinecone.Index("embeddings")

# Batch size 10,000 vectors, dimension 2048
vectors = [(f"id-{i}", np.random.rand(2048).astype(np.float32).tolist()) for i in range(10000)]
index.upsert(vectors=vectors, namespace="eval")

After (chunked upserts)

import pinecone
import math

pinecone.init(api_key="YOUR_KEY", environment="us-west1-gcp")
index = pinecone.Index("embeddings")

BATCH_SIZE = 2000  # Chosen to keep payload < 4 MiB
DIMENSION = 2048

def chunked_upsert(vectors):
    total = len(vectors)
    for i in range(0, total, BATCH_SIZE):
        batch = vectors[i:i + BATCH_SIZE]
        # Optional: log estimated size for sanity check
        size_mb = request_size(len(batch), DIMENSION) / (1024 * 1024)
        print(f"Upserting batch {i // BATCH_SIZE + 1}: {len(batch)} vectors ({size_mb:.2f} MiB)")
        index.upsert(vectors=batch, namespace="eval")

vectors = [(f"id-{i}", np.random.rand(DIMENSION).astype(np.float32).tolist())
           for i in range(10000)]
chunked_upsert(vectors)

For queries, reduce top_k or split large query sets into multiple calls:

# Original (single large query)
results = index.query(
    vector=query_vec,
    top_k=5000,
    include_metadata=True,
    namespace="eval"
)

# Revised (paginate queries)
PAGE_SIZE = 1000
all_matches = []
for offset in range(0, 5000, PAGE_SIZE):
    page = index.query(
        vector=query_vec,
        top_k=PAGE_SIZE,
        include_metadata=True,
        namespace="eval",
        filter={"_offset": offset}
    )
    all_matches.extend(page.matches)

Validation

Run the evaluation pipeline with the new chunk size and confirm that no token limit exceeded errors appear in the logs.
Verify that the total number of upserted vectors matches the expected count (e.g., 10 000) by querying the index statistics:
```
stats = index.describe_index_stats()
print(stats.total_vector_count)  # Should be 10000
```
Check downstream metrics (e.g., similarity score distribution) for completeness.
Monitor Pinecone’s “request size” metric for a few evaluation cycles to ensure it stays comfortably below the 4 MiB threshold.

Operational Experience & Prevention

Several production nuances emerged during the incident response:

Misleading symptom: Initial alerts showed “high latency” rather than the token error because the client retried the oversized request, inflating response times.
Assumption trap: The team assumed that increasing top_k would not affect request size. In reality, each returned match adds metadata bytes, pushing the response payload over the limit.
Edge case: When using async upserts, the client library batches calls internally. Even if the user code respects the batch size, the library may combine multiple async calls into a larger payload. Explicitly set batch_size on the client or disable auto‑batching.
Lesson learned: Encode the batch‑size calculation as a guard clause in the evaluation framework so that any future change in embedding dimension automatically adjusts the chunk size.

Preventive Guard Clause Example

MAX_REQUEST_MB = 3.5  # Leave headroom below 4 MiB

def compute_max_batch(dim, metadata_bytes=0):
    # Solve for N: (N * dim * 4 + N * 50 + metadata_bytes) / (1024*1024) <= MAX_REQUEST_MB
    bytes_per_vector = dim * 4 + 50 + metadata_bytes
    return int((MAX_REQUEST_MB * 1024 * 1024) / bytes_per_vector)

BATCH_SIZE = compute_max_batch(DIMENSION)
print(f"Auto‑computed safe batch size: {BATCH_SIZE}")

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

Why does the error appear only after increasing the embedding dimension?
Because request size scales linearly with dimension × 4 bytes. A jump from 1536 to 2048 dimensions adds ~3 MiB per 1 000 vectors, quickly breaching the 4 MiB limit.
Can I increase the token limit on Pinecone?
No. The limit is a hard service quota designed to protect the backend. The recommended approach is to split payloads.
Does reducing top_k always fix query token errors?
Reducing top_k lowers the size of the response payload, but the request payload (the query vector) still counts toward the limit. If the query vector itself is oversized, you must chunk the query or reduce vector dimensionality.
How do I know the exact size of a request before sending it?
Use the request_size helper shown above or enable the Pinecone client’s debug logging, which prints the calculated payload size.
Are there any Pinecone settings to automatically paginate upserts?
The client library does not auto‑paginate upserts; you must implement chunking yourself. However, the upsert method accepts a batch_size argument in some SDKs that can be used to enforce limits.