OpenAI GPT-4o Embedding Dimension Mismatch During Model Evaluation Pipeline

Q: Root Cause

The mismatch originates from an undocumented change in the gpt-4o model’s default embedding endpoint:

In this article

Table of Contents

OpenAI GPT‑4o Embedding Dimension Mismatch During Model Evaluation Pipeline

Problem

During the nightly evaluation of a large‑scale text similarity benchmark, the pipeline that extracts vector representations from the OpenAI GPT‑4o model crashes with the following error:


Traceback (most recent call last):
  File "evaluate.py", line 112, in run
    embedding = get_gpt4o_embedding(text)
  File "gpt4o_client.py", line 58, in get_gpt4o_embedding
    response = client.embeddings.create(input=text, model="gpt-4o")
  File ".../openai/_client.py", line 214, in create
    raise OpenAIError(f"Unexpected embedding dimension: {len(vec)}")
openai.error.OpenAIError: Unexpected embedding dimension: 1536 (expected 1024)

Downstream components – a cosine‑similarity scorer and a K‑means clustering job – expect a fixed 1024‑dimensional vector. The mismatch propagates as shape errors in NumPy/TensorFlow, causing the entire evaluation run to abort.

Root Cause

The mismatch originates from an undocumented change in the gpt-4o model’s default embedding endpoint:

Version drift: OpenAI rolled out a new embedding layer for gpt-4o that returns 1536‑dimensional vectors when the embedding_format parameter is omitted.
Client SDK default: The Python SDK version 0.28.0 defaults embedding_format="dense", which now maps to the 1536‑dimensional space. Older SDKs (≤0.27.x) defaulted to a 1024‑dimensional legacy format.
Pipeline assumption: The evaluation code hard‑codes the expected size (1024) based on the legacy format and validates the length of the returned vector, raising the error shown above.

In short, the pipeline is still built for the legacy embedding shape, while the live API now returns a larger vector because the client library silently switched to the new default.

Debug

Step‑by‑step investigation that reproduced the failure:

Inspect the raw API response using curl with the openai endpoint.
Check the SDK version and the default parameters it sends.
Validate the shape of the returned vector against the expectation.

Commands and expected outputs:

# 1. Raw HTTP request (API key omitted for brevity)
curl https://api.openai.com/v1/embeddings \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","input":"sample text"}' | jq '.data[0].embedding | length'

# Expected output (current API)
1536

# 2. Verify installed SDK version
python -c "import openai, pkg_resources; print(pkg_resources.get_distribution('openai').version)"
0.28.0

# 3. Minimal Python snippet to print the vector length
import openai
openai.api_key = "sk-..."
resp = openai.embeddings.create(input="sample text", model="gpt-4o")
print(len(resp.data[0].embedding))
# Output
1536

These checks confirm that the API is returning a 1536‑dimensional vector and that the SDK version is the one introducing the new default.

Solution

Two viable approaches:

Option A – Pin the legacy embedding format

Explicitly request the 1024‑dimensional legacy format using the embedding_format parameter (available from SDK 0.28.0 onward).

# Before (fails)
def get_gpt4o_embedding(text: str) -> List[float]:
    resp = openai.embeddings.create(input=text, model="gpt-4o")
    return resp.data[0].embedding

# After (fixed)
def get_gpt4o_embedding(text: str) -> List[float]:
    resp = openai.embeddings.create(
        input=text,
        model="gpt-4o",
        embedding_format="legacy"   # forces 1024‑dimensional vectors
    )
    return resp.data[0].embedding

Why it works: The API respects embedding_format="legacy" and returns the original 1024‑dimensional representation, matching the downstream expectations.

Option B – Upgrade the pipeline to handle 1536 dimensions

If the higher‑dimensional embeddings are desirable (they often improve downstream similarity), adjust the pipeline:

Update the NumPy/TensorFlow input shapes from (batch, 1024) to (batch, 1536).
Re‑train or re‑fit the clustering model with the new dimensionality.
Modify the validation checks that raise OpenAIError on size mismatch.

# Before (validation)
if len(vec) != 1024:
    raise OpenAIError(f"Unexpected embedding dimension: {len(vec)}")

# After (flexible)
expected_dim = 1024 if use_legacy else 1536
if len(vec) != expected_dim:
    raise OpenAIError(f"Unexpected embedding dimension: {len(vec)}")

Choosing Option B future‑proofs the pipeline but requires a one‑time re‑training effort.

Verify

After applying the fix, run a sanity‑check on a small sample set:

# Verify dimension
emb = get_gpt4o_embedding("verification text")
print("Embedding size:", len(emb))
# Expected output for Option A: Embedding size: 1024
# Expected output for Option B: Embedding size: 1536

Then execute the downstream similarity calculation:

import numpy as np
a = np.array(emb)
b = np.random.rand(len(emb))
cosine = np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
print("Cosine similarity:", cosine)

Successful execution without shape errors indicates the pipeline now aligns with the embedding dimensionality.

Prevent

Pin SDK version: Use openai==0.27.* in production until the pipeline is updated, preventing silent default changes.
Explicit API parameters: Always set embedding_format (or any future format flag) rather than relying on defaults.
Schema validation: Add a runtime check that logs the received dimension and raises a warning instead of an exception, allowing graceful fallback.
Automated contract tests: Include a CI test that calls the embedding endpoint and asserts the shape matches the configured constant.
Monitoring: Emit a custom metric gpt4o_embedding_dim from the inference service; alert if the value deviates from the expected dimension.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the error appear only after a recent deployment? The OpenAI service introduced a new default embedding format in March 2024. If the client SDK was upgraded automatically, the hidden default changed, exposing the mismatch.
Can I request a specific vector size other than 1024 or 1536? No. The model currently supports only the two predefined formats: legacy (1024) and dense (1536). Future releases may add more options.
Is the 1536‑dimensional embedding always better? Empirically, higher dimensionality captures more nuance, but it also increases memory and compute costs. Benchmark both formats on your downstream task before committing.
Do other OpenAI models exhibit the same behavior? Only gpt-4o and its sibling gpt-4o-mini have the dual‑format embedding endpoint. Older models (text-embedding-ada-002, etc.) return a fixed size.
How can I avoid breaking changes from OpenAI in the future? Adopt a version‑locked SDK, pin the embedding_format explicitly, and maintain a contract test that validates the shape of every external vector representation.