brown pine cone in close up photography

Pinecone index upsert failure after RAG model update

by

Problem – Pinecone Upsert Failures After RAG Model Update

In a production RAG pipeline the ingestion workers generate embeddings with a new model (e.g., sentence‑transformers/all‑miniLM‑L6‑v2, 384‑dim) while the query service still uses the previous model (e.g., text‑ada‑002, 1536‑dim). The asynchronous, event‑driven architecture (AWS SQS → Lambda → Pinecone) allows the two services to scale independently, so the model versions drift.

Typical symptoms observed in logs and alerts:

  • Upsert errors: PineconeError: Upsert failed – vector dimension does not match index dimension (expected 1536, got 384)
  • Cosine similarity degradation: retrieval relevance drops >70 % after the rollout, alerts fire on “Retrieval relevance drop > 50 %”.
  • NaN scores: RuntimeError: Cosine similarity calculation returned NaN values after embedding model change

These failures manifest intermittently because only a subset of Lambdas pick up the new model (cached Docker layer) while others continue using the older version.

Root Cause – Dimension Mismatch and Schema Drift

The Pinecone index is created with a fixed vector dimension (e.g., 1536) as defined in the Index Management documentation (Pinecone Documentation – Index Management). The Upsert API requires every vector to match that dimension (Pinecone Documentation – Upsert API). When the ingestion pipeline switched to a 384‑dim embedding model, the upsert request violated this constraint, triggering the dimension‑mismatch error.

Beyond the hard error, the query service still sends 1536‑dim vectors to the index. Because Pinecone stores vectors in a single space, cosine similarity is computed across mismatched embedding spaces, leading to near‑random scores and the observed relevance drop (Pinecone Documentation – Query API).

Key assumptions that failed:

  • Embedding dimension is immutable for a given index.
  • All services share a single source of truth for the embedding model version.
  • Deployment pipelines guarantee atomic rollout of both ingestion and query components.

Debug – Investigation Steps

1. Verify Index Dimension

curl -X GET "https://controller..pinecone.io/databases/" \
  -H "Api-Key: $PINECONE_API_KEY"

Expected JSON snippet:

{
  "name": "rag-index",
  "dimension": 1536,
  "metric": "cosine"
}

2. Inspect Ingestion Logs

2024-08-14T10:23:41.112Z INFO  lambda_handler - Embedding model: sentence-transformers/all-miniLM-L6-v2 (384 dims)
2024-08-14T10:23:41.115Z ERROR pinecone_client - Upsert error – vector dimension does not match index dimension (expected 1536, got 384)

3. Correlate Deployment Metadata

Search CI/CD logs for the model version tag:

2024-08-13 Deploying ingestion image: rag‑ingest:sha1=abcd123 (model_version=miniLM-L6-v2)
2024-08-13 Deploying query image: rag‑query:sha1=efgh456 (model_version=ada-002)

4. Reproduce the Mismatch Locally

# Generate a 384‑dim vector
python - <<'PY'
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
vec = model.encode(["test"])[0]
print(len(vec))
PY
384
# Attempt upsert to a 1536‑dim index
pinecone_client.upsert(vectors=[("id1", vec.tolist())])
# → PineconeError: Upsert failed – vector dimension does not match index dimension (expected 1536, got 384)

5. Check Query Service Vectors

# Query side still uses Ada-002 (1536 dims)
len(embedding)  # → 1536

Solution – Version Locking and Coordinated Rollout

1. Pin Index Dimension to Model Version

Create a new index for the new embedding dimension instead of reusing the old one.

# Create a 384‑dim index
pinecone_client.create_index(
    name="rag-index-v2",
    dimension=384,
    metric="cosine",
    metadata_config={"indexed": ["doc_id", "source"]}  # optional
)

2. Introduce Embedding Version Tag in Metadata

Store the model version alongside each vector; Pinecone’s Metadata & Schema guide recommends this for drift prevention (Pinecone Documentation – Metadata & Schema).

# Upsert with version tag
pinecone_client.upsert(
    vectors=[(
        "doc-123",
        embedding.tolist(),
        {"embedding_version": "miniLM-L6-v2"}  # metadata
    )]
)

3. Deploy Coordinated Blue/Green Release

Use a feature flag or environment variable EMBEDDING_MODEL_VERSION that is read by both ingestion and query services. The CI/CD pipeline must update the flag atomically for all services.

# .env
EMBEDDING_MODEL_VERSION=miniLM-L6-v2
INDEX_NAME=rag-index-v2

4. Migration Script (Old → New Index)

For continuity, copy existing vectors to the new index after re‑embedding with the new model.

import pinecone, torch
src = pinecone.Index("rag-index")          # 1536‑dim
dst = pinecone.Index("rag-index-v2")       # 384‑dim
model = SentenceTransformer('all-MiniLM-L6-v2')

for batch in src.fetch(ids=..., include_data=True):
    new_vectors = [
        (id, model.encode([rec["values"]])[0].tolist(), {"embedding_version": "miniLM-L6-v2"})
        for id, rec in batch.items()
    ]
    dst.upsert(vectors=new_vectors)

5. Decommission Old Index

After validation, delete the legacy index to avoid accidental use.

pinecone_client.delete_index("rag-index")

Verification – Confirming the Fix

Upsert Success

2024-08-15T08:12:03.210Z INFO  lambda_handler - Embedding model: miniLM-L6-v2 (384 dims)
2024-08-15T08:12:03.215Z INFO  pinecone_client - Upsert succeeded for 100 vectors

Query Relevance Check

Run a handful of end‑to‑end queries and compare cosine scores before/after.

# Before (mismatched)
Score: 0.12 (random)

# After (aligned)
Score: 0.84 (high relevance)

Monitoring Metrics

  • upsert_success_total should return to baseline (e.g., 10 k/min).
  • retrieval_relevance (custom metric) rises back above 0.8.
  • No more “Dimension mismatch” errors in pinecone_client logs.

Prevention – Guardrails for Future Model Changes

  • Schema‑drift detection: Periodically scan Pinecone metadata for multiple embedding_version values; raise an alert if more than one version is present.
  • Version‑locked CI/CD stage: Require a pipeline gate that verifies INDEX_DIMENSION == len(embedding_vector) for the target model before promotion.
  • Immutable index per model: Adopt a naming convention rag-index- and treat the index as immutable; create a new index for every dimension change.
  • Feature flag rollout: Use a centralized config service (e.g., AWS AppConfig) to flip the EMBEDDING_MODEL_VERSION flag only after both ingestion and query services have been redeployed.
  • Automated health check: Deploy a Lambda that performs a test upsert and query every 5 minutes; failure triggers a PagerDuty incident.

Related Topic Hub: Vector Databases Troubleshooting Hub

FAQ

  1. Why does the upsert error show “expected 1536, got 384” even though the index was created recently? The index dimension is immutable. If you reuse an existing index that was originally created for a 1536‑dim model, Pinecone will reject any vector with a different length. Create a new index with the correct dimension or re‑create the old one.
  2. Can I store vectors of different dimensions in the same index by using metadata? No. Pinecone enforces a single dimensionality per index regardless of metadata. The only supported way to handle multiple dimensions is separate indexes.
  3. How can I detect that ingestion and query services are using different embedding versions? Emit the EMBEDDING_MODEL_VERSION as a structured log field and set up a CloudWatch metric filter that counts distinct values per service. Alert when the set size > 1.
  4. Is cosine similarity still valid after changing the embedding model? Only if both ingestion and query vectors are generated by the same model and thus reside in the same vector space. Mixing spaces produces meaningless cosine scores.
  5. What is the recommended way to migrate existing data when changing models? Re‑embed the source documents with the new model and upsert them into a newly created index. Use a batch job or streaming migration as shown in the “Migration Script” section.