a view of a restaurant through a window at night

TensorRT engine build fails with embedding dimension mismatch after model change

by

Problem: TensorRT Engine Build Fails with Embedding Dimension Mismatch After Model Change

A real‑time video streaming inference pipeline converts a PyTorch model containing an nn.Embedding layer to a TensorRT engine for sub‑millisecond latency. After a recent model revision that altered the embedding size, the engine build aborts with a validation error:


[2026-10-21 14:03:12] [ERROR] [TensorRT] Engine build failed: Validation failed for node 'Embedding_1' (type: Gather). Expected input dimension 128, but got 64.

The failure occurs during the trt.Builder.build_cuda_engine() call, preventing deployment to production.

Root Cause

The updated model changed the embedding dimension from 128 to 64 while the surrounding graph (e.g., a subsequent linear projection) still expects a 128‑dimensional vector. When TensorRT parses the ONNX representation, it validates shape compatibility between the Gather (generated for the embedding lookup) and the downstream MatMul. The mismatch triggers the validation error.

Key factors that contributed:

  • Static shape inference in TensorRT: TensorRT requires all tensor dimensions to be known at build time unless explicitly marked as dynamic. The embedding output shape is therefore fixed during conversion.
  • ONNX export without explicit shape metadata: The PyTorch torch.onnx.export call omitted dynamic_axes for the embedding output, causing TensorRT to assume the old dimension.
  • Downstream layers hard‑coded to the old size: The linear layer after the embedding was not re‑trained or re‑configured to accept the new dimension.

Debug: Investigation Steps

1. Reproduce the build failure locally


python3 convert_to_trt.py --model_path model_v2.pt --output engine.trt
# Expected output: Engine built successfully
# Actual output:
[2026-10-21 14:03:12] [ERROR] [TensorRT] Engine build failed: Validation failed for node 'Embedding_1' (type: Gather). Expected input dimension 128, but got 64.

2. Inspect the ONNX graph


python3 -c "
import onnx
model = onnx.load('model_v2.onnx')
for node in model.graph.node:
    if node.op_type == 'Gather':
        print('Gather node:', node.name)
        for inp in node.input:
            print('  input:', inp)
        for out in node.output:
            print('  output:', out)
"

Output shows the Gather output shape is [batch, seq_len, 64] while the following MatMul expects [batch, seq_len, 128].

3. Verify downstream layer dimensions


python3 -c "
import torch
model = torch.load('model_v2.pt')
print('Embedding weight shape:', model.embedding.weight.shape)
print('Linear weight shape:', model.fc.weight.shape)
"

Result:


Embedding weight shape: torch.Size([10000, 64])
Linear weight shape: torch.Size([256, 128])

4. Check TensorRT logs for shape propagation


export TRT_LOGGER_SEVERITY=VERBOSE
python3 convert_to_trt.py ...
# Look for lines like:
[VERBOSE] [TensorRT] Propagating shape for node 'MatMul_2': input0=[?, ?, 64], input1=[128, 256]

5. Confirm that the model change was intentional

Review the Git commit that introduced the embedding size change. The commit message indicates a reduction in embedding dimension to reduce memory footprint, but downstream layers were not updated.

Solution: Align Embedding Dimension Across the Graph

Option A – Update Downstream Layers (Recommended)

Retrain or modify the linear projection to accept the new embedding size.

# Before (model_v2.py)
self.embedding = nn.Embedding(num_embeddings=10000, embedding_dim=64)
self.fc = nn.Linear(128, 256)  # expects 128‑dim input

# After
self.embedding = nn.Embedding(num_embeddings=10000, embedding_dim=64)
self.fc = nn.Linear(64, 256)   # matches embedding output

Retrain the model (or fine‑tune) to adjust weights:


python3 train.py --model_def model_v2.py --epochs 5 --output model_v2_fixed.pt

Option B – Preserve Original Dimension via Projection

If changing downstream layers is not feasible, insert an explicit projection after the embedding to restore the original size.

# Insert a linear layer that maps 64 → 128
self.proj = nn.Linear(64, 128)

def forward(self, ids):
    x = self.embedding(ids)          # shape: [B, S, 64]
    x = self.proj(x)                 # shape: [B, S, 128]
    return self.fc(x)

Export the updated model to ONNX:


python3 export_onnx.py --model_path model_v2_fixed.pt --output model_v2_fixed.onnx

Option C – Use Dynamic Shapes in TensorRT

If the embedding dimension must remain variable across deployments, export with dynamic_axes for the embedding output and enable TensorRT’s BuilderFlag.kGPU_FALLBACK. This approach sacrifices some optimization but avoids static validation errors.


torch.onnx.export(
    model,
    dummy_input,
    "model_dynamic.onnx",
    input_names=["input_ids"],
    output_names=["logits"],
    dynamic_axes={"input_ids": {0: "batch", 1: "seq_len"},
                  "logits": {0: "batch", 1: "seq_len"}},
    opset_version=15,
)

Verify: Confirm Engine Builds and Runs Correctly

1. Build the TensorRT engine


python3 convert_to_trt.py --model_path model_v2_fixed.onnx --output engine_fixed.trt
# Expected output:
[INFO] Engine built successfully (FP16, batch=8, seq_len=32)

2. Run an inference sanity check


python3 infer_trt.py --engine engine_fixed.trt --batch 4 --seq_len 32
# Sample output:
[INFO] Inference latency: 0.73 ms per batch
[INFO] Output shape: (4, 32, 256)

3. Compare against baseline

Metric Before Fix After Fix
Engine Build Time Failed ≈ 12 s
Inference Latency (FP16) N/A 0.73 ms / batch
Memory Footprint (GPU) N/A ≈ 210 MiB

Prevent: Guardrails to Avoid Future Mismatches

  • Automated shape validation in CI: Add a step that loads the exported ONNX model and runs onnx.checker.check_model followed by a TensorRT dry‑run build. Fail the pipeline on shape errors.
  • Versioned model contracts: Store expected tensor shapes (e.g., embedding_dim) in a JSON schema alongside the model artifact. Downstream services verify the contract before conversion.
  • Explicit dynamic_axes for any dimension that may change: Prevent silent static‑shape assumptions.
  • Unit tests for layer compatibility: After any architectural change, instantiate the model and assert that next_layer.in_features == embedding_dim.
  • Monitoring of TensorRT build logs: Alert on any Validation failed messages with severity ≥ ERROR.

FAQ

  1. Why does the engine build succeed on my laptop but fail in CI?

    The laptop uses the ONNX Runtime parser, which is more permissive with mismatched shapes, while CI enforces TensorRT’s strict static shape validation. Ensure both environments use the same TensorRT version and parser.

  2. Can I keep the old 128‑dim embedding and still reduce memory?

    Yes. Apply weight quantization (e.g., INT8) to the embedding matrix instead of changing its dimensionality. This preserves downstream shapes while lowering memory usage.

  3. Is it safe to set builder.max_batch_size to a very high value to bypass shape checks?

    No. The shape mismatch is unrelated to batch size; increasing max_batch_size will not resolve the dimension incompatibility and may cause out‑of‑memory errors.

  4. How do I debug similar shape mismatches for other layers (e.g., Conv2d)?

    Use onnx-simplifier to collapse constant folds, then inspect the offending node’s input and output shapes with netron or onnx.shape_inference.infer_shapes. TensorRT’s verbose logs will also report the expected vs. actual dimensions.

  5. Can I force TensorRT to ignore shape mismatches with a flag?

    TensorRT does not provide a flag to ignore validation errors because they indicate undefined behavior at runtime. The correct approach is to fix the model graph.

Related Topic Hub: Model Serving Troubleshooting Hub

Related Articles