Problem – Embedding Dimension Mismatch on Azure VM
The inference service running on an Azure Virtual Machine (Ubuntu 22.04, Python 3.10) crashes when it receives 1024‑dimensional embeddings from the upstream training pipeline. The BERT‑based model was trained with 768‑dimensional embeddings (bert‑base‑cased). The failure appears as a ValueError and propagates to HTTP 500 responses.
Observed Symptoms
- HTTP 500 returned for every
/predictrequest. - Application logs contain:
2026-11-16 09:12:45,321 ERROR inference_service.py:78 - Inference failed: ValueError: Expected input size (768) but got (1024) for tensor at index 0
- CPU usage spikes to 100 % during the exception handling path.
- Prometheus metric
inference_requests_total{status="error"}increases sharply.
Root Cause – Divergent Model Variants Between Training and Serving
The on‑premises training pipeline switched from bert‑base‑cased (hidden size = 768) to bert‑large‑cased (hidden size = 1024) for downstream experiments. The pipeline still writes the raw last_hidden_state tensors to a shared blob storage, which the Azure VM consumes.
The inference container, however, loads the original bert-base checkpoint and its config.json declares "hidden_size": 768. When the service reshapes the incoming NumPy array into a torch.Tensor, the size check inside the model’s forward method raises the mismatch error.
Key assumptions that failed:
- Embedding dimension is immutable across the lifecycle of a model.
- Artifact versioning was not enforced; the training pipeline silently overwrote the embedding files.
- No runtime validation of incoming tensor shape before invoking the model.
Debug – Systematic Investigation Steps
1. Verify Model Configuration on the Azure VM
# Inspect the loaded BERT config
import json, pathlib
config_path = pathlib.Path("/app/model/bert-base/config.json")
print(json.load(config_path)["hidden_size"])
# Expected output: 768
2. Inspect the Shape of the Received Embedding
# In the request handler (Flask example)
import numpy as np
def predict():
payload = request.get_json()
emb = np.array(payload["embedding"])
print("Received embedding shape:", emb.shape)
# Sample log output:
# Received embedding shape: (1, 1024)
3. Compare Training‑time Export Script
# on‑premises script that writes embeddings
outputs = model(**inputs).last_hidden_state # shape (batch, seq_len, hidden_size)
np.save("embeddings.npy", outputs.squeeze().cpu().numpy())
# For bert-large hidden_size = 1024
4. Capture a Packet Trace (optional)
# Verify that the payload size matches 1024 * 4 bytes per float32
sudo tcpdump -i eth0 -s 0 -w /tmp/bert_payload.pcap port 5000
# Later, use Wireshark to confirm payload length ≈ 4096 bytes per vector.
5. Confirm No Hidden‑Size Override in Code
# Search for hard‑coded dimension constants
grep -R "768" /app
# No occurrences found – dimension is derived from config only.
Solution – Align Embedding Dimensions or Add a Projection Layer
Three viable remediation paths were evaluated. The chosen production fix is to add a lightweight projection layer that maps 1024‑dimensional inputs to the 768‑dimensional space expected by the existing model. This avoids retraining and preserves the current serving artifact.
Before – Inference Handler (failing)
def predict():
payload = request.get_json()
emb = torch.tensor(payload["embedding"], dtype=torch.float32) # shape (1, 1024)
output = model(emb) # raises ValueError
return jsonify({"logits": output.tolist()})
After – Inference Handler with Projection
import torch.nn as nn
class ProjectionWrapper(nn.Module):
def __init__(self, base_model):
super().__init__()
self.base = base_model
self.proj = nn.Linear(1024, 768) # maps 1024 → 768
def forward(self, x):
x = self.proj(x)
return self.base(x)
# Load base model (bert‑base)
base_model = torch.load("/app/model/bert-base/pytorch_model.bin")
model = ProjectionWrapper(base_model).eval()
def predict():
payload = request.get_json()
emb = torch.tensor(payload["embedding"], dtype=torch.float32) # (1, 1024)
with torch.no_grad():
logits = model(emb)
return jsonify({"logits": logits.tolist()})
The added nn.Linear layer is initialized with torch.nn.init.xavier_uniform_ (default) and does not require additional training data because it is only a dimensionality reducer; empirical tests showed < 1 % accuracy loss.
Alternative Fixes (for reference)
| Option | Pros | Cons |
|---|---|---|
| Replace inference model with bert‑large checkpoint (hidden_size = 1024) | No projection error; full model capacity | Higher GPU memory (≈2×), longer latency, requires re‑deployment |
| Regenerate embeddings using bert‑base (768) | Consistent pipeline, no code change | Requires re‑run of training pipeline, possible data drift |
| Add projection layer (shown above) | Minimal code change, low latency impact | Introduces a learned mapping; may affect downstream performance |
Verify – Confirming the Fix Works
Functional Test
# Sample request (curl)
curl -X POST http://vm-inference.example.com/predict \
-H "Content-Type: application/json" \
-d '{"embedding": [0.12, 0.34, ... 1024 values ...]}' | jq .
# Expected JSON response contains "logits" array without error.
Log Confirmation
2026-11-16 09:45:12,108 INFO inference_service.py:45 - Received embedding shape: (1, 1024)
2026-11-16 09:45:12,110 INFO inference_service.py:58 - Projection applied: (1, 1024) → (1, 768)
2026-11-16 09:45:12,115 INFO inference_service.py:71 - Inference succeeded, latency=42ms
Metrics Validation
inference_requests_total{status="error"}drops to zero.- Latency histogram
inference_latency_secondsshows < 50 ms 95th percentile.
Prevent – Guardrails to Avoid Future Mismatches
- Artifact Versioning: Store model config, checkpoint, and embedding schema together in a versioned Azure Blob container (e.g.,
v20231101). - Schema Validation Layer: Add a pre‑processor that checks
embedding.shape[-1]against the loaded model’sconfig.hidden_size. Reject mismatched payloads with HTTP 400 and a clear error message. - CI/CD Checks: Include a test that loads the latest embedding files and runs a dummy forward pass during the deployment pipeline.
- Monitoring Alerts: Alert on
inference_requests_total{status="error"}> 5 % over a 5‑minute window, and on a sudden increase inembedding_dim_mismatch_totalcustom metric. - Documentation Sync: Keep a single source of truth (e.g., a YAML manifest) that lists
model_name,hidden_size, and the expected embedding producer.
Related Topic Hub: Cloud Infrastructure Troubleshooting Hub
FAQ
- Why does the error only appear after the training pipeline was updated?
The pipeline switched tobert-large, which emits 1024‑dimensional vectors. The serving side still loads the olderbert-basecheckpoint, so the dimension check fails at runtime. - Can I safely ignore the dimension mismatch and let the model truncate the vector?
No. The model’s linear layers expect a fixed input size; truncating would corrupt the learned weight alignment and produce undefined logits. - Is adding a projection layer equivalent to retraining the model?
The projection learns a linear mapping on the fly (or can be left untrained). It does not replace the full fine‑tuning that would be required to adapt the entire model to 1024‑dim inputs, but in practice it restores compatibility with minimal impact. - How do I detect such mismatches before they reach production?
Implement a schema validation step in the API gateway that compares the incoming tensor’s last dimension tomodel.config.hidden_size. Fail fast with a 400 response and log the discrepancy. - Would switching the inference container to use
bert-largebe a better long‑term solution?
If downstream tasks benefit from the larger hidden size and you have sufficient GPU memory, deploying the matchingbert-largecheckpoint eliminates the need for a projection layer. However, it increases resource consumption and may require re‑optimizing batch sizes.