VLLM Text Image Modality Mismatch After GPU Passthrough

Q: Problem – Text‑Image Modality Mismatch after GPU Passthrough

When running a multimodal model (e.g., llava‑1.5‑7b) inside a Docker container that uses the NVIDIA Container Toolkit, the generated responses contain text that is unrelated to the supplied image. The model appears to process the textual prompt correctly but ignores the visual content, resulting in incoherent or “empty” captions.

Q: Investigation and Debugging Steps 1. Verify GPU visibility inside the container

docker run --rm nvidia/cuda:12.1-runtime-ubuntu22.04 nvidia-smi Expected output shows all GPUs listed. If the command prints “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”, the toolkit is not correctly attached.

Q: 2. Check CUDA_VISIBLE_DEVICES and runtime driver

docker exec -it $CONTAINER_ID bash -c 'echo $CUDA_VISIBLE_DEVICES' Should be a comma‑separated list (e.g., 0,1). An empty string triggers the CPU fallback warning.

Q: 3. Inspect vLLM model initialization logs

[2024-05-12 10:14:32] INFO vllm.engine.model_loader: Loading text encoder on device cuda:0 [2024-05-12 10:14:33] INFO vllm.engine.model_loader: Loading vision encoder on device cuda:1 [2024-05-12 10:14:34] WARNING vllm.multimodal: Vision encoder device differs from language model device – will attempt fallback. If the two devices differ, the warning is a strong indicator of the problem.

Q: 5. Review Docker run command and environment variables

docker run -d \ --gpus all \ -e VLLM_DEVICE=cuda:0 \ -e TORCH_DTYPE=bfloat16 \ -v /host/models:/models \ myrepo/vllm-multimodal:latest Missing --gpus all or an incorrect VLLM_DEVICE value reproduces the mismatch.

Q: Resolution – Align Devices and Data Types 1. Enforce a single CUDA device for the entire multimodal pipeline

Set the environment variable VLLM_DEVICE before launching vLLM and ensure all sub‑models respect it.

Q: 2. Use the official NVIDIA runtime

Update Docker daemon configuration (/etc/docker/daemon.json) to include the NVIDIA runtime:

Q: 3. Adjust the Docker run command

docker run -d \ --gpus all \ --runtime=nvidia \ -e VLLM_DEVICE=cuda:0 \ -e TORCH_DTYPE=bfloat16 \ -v $(pwd)/models:/models \ myrepo/vllm-multimodal:latest 4. Explicitly set device in the Python entrypoint import os, torch, vllm device = os.getenv('VLLM_DEVICE', 'cuda:0') torch.cuda.set_device(device) model = vllm.LLM( model="llava-1.5-7b", dtype=os.getenv('TORCH_DTYPE', 'bfloat16'), device=device, multimodal=True ) 5. Re‑run a sanity check curl -X POST http://localhost:8000/generate \ -H "Content-Type: application/json" \ -d '{ "prompt": "Describe the scene:", "image": "/data/test.jpg" }' The response should now contain a caption that references visual elements (e.g., “A dog playing with a ball on a grassy field”).

In this article

Table of Contents

Problem – Text‑Image Modality Mismatch after GPU Passthrough

When running a multimodal model (e.g., llava‑1.5‑7b) inside a Docker container that uses the NVIDIA Container Toolkit, the generated responses contain text that is unrelated to the supplied image. The model appears to process the textual prompt correctly but ignores the visual content, resulting in incoherent or “empty” captions.

Typical symptoms observed in the container logs:


RuntimeError: Expected all tensors to be on the same device, but found at least two devices, in vLLM multimodal forward pass.
ValueError: Image embeddings size (batch, 1024) does not match expected token length (batch, 768) – modality fusion failure.
Warning: CUDA_VISIBLE_DEVICES is set to an empty string – vision model falls back to CPU, leading to disjoint responses.

These errors indicate a breakdown in the cross‑modal fusion step of vLLM.

Root Cause Analysis

vLLM expects both the language encoder and the vision encoder to share a single CUDA context. The multimodal inference guide (vLLM documentation – Multimodal inference guide) states that:

“All input tensors (text tokens and image embeddings) must reside on the same device before fusion. The framework does not perform automatic device migration.”

In the reported incidents the following conditions were present:

Device‑level mismatch: The image encoder was instantiated on cuda:0 while the text encoder was forced onto cuda:1 (see the real incident “image encoder on cuda:0, text encoder on cuda:1”). This caused the vision branch to silently fall back to the CPU, producing embeddings that never reached the fusion layer.
Missing --gpus all flag: Docker started without exposing GPUs to the container, so the vision transformer defaulted to CPU while the language model used the GPU (incident “Docker run omitted --gpus all”).
Inconsistent torch_dtype: The language model ran in bfloat16 while the vision encoder stayed in float32, leading to a shape‑mismatch during concatenation (incident “dtype mismatch”).
Incorrect runtime: Using the default runc driver instead of nvidia-container-runtime prevented CUDA context sharing between sub‑models (NVIDIA Developer Forums thread).

Because vLLM does not automatically reconcile these mismatches, the forward pass succeeds on the language side but the vision embeddings are either dropped or mis‑aligned, producing the observed disjoint output.

Investigation and Debugging Steps

1. Verify GPU visibility inside the container

docker run --rm nvidia/cuda:12.1-runtime-ubuntu22.04 nvidia-smi

Expected output shows all GPUs listed. If the command prints “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver”, the toolkit is not correctly attached.

2. Check `CUDA_VISIBLE_DEVICES` and runtime driver

docker exec -it $CONTAINER_ID bash -c 'echo $CUDA_VISIBLE_DEVICES'

Should be a comma‑separated list (e.g., 0,1). An empty string triggers the CPU fallback warning.

3. Inspect vLLM model initialization logs


[2024-05-12 10:14:32] INFO vllm.engine.model_loader: Loading text encoder on device cuda:0
[2024-05-12 10:14:33] INFO vllm.engine.model_loader: Loading vision encoder on device cuda:1
[2024-05-12 10:14:34] WARNING vllm.multimodal: Vision encoder device differs from language model device – will attempt fallback.

If the two devices differ, the warning is a strong indicator of the problem.

4. Validate tensor device placement before fusion

python - <<'PY'
import torch, vllm
text_tensor = torch.randint(0, 32000, (1, 128), device='cuda:0')
image_tensor = torch.randn(1, 3, 224, 224, device='cuda:1')
print('text device:', text_tensor.device)
print('image device:', image_tensor.device)
PY

Both lines must report the same device (e.g., cuda:0).

5. Review Docker run command and environment variables


docker run -d \
  --gpus all \
  -e VLLM_DEVICE=cuda:0 \
  -e TORCH_DTYPE=bfloat16 \
  -v /host/models:/models \
  myrepo/vllm-multimodal:latest

Missing --gpus all or an incorrect VLLM_DEVICE value reproduces the mismatch.

Resolution – Align Devices and Data Types

1. Enforce a single CUDA device for the entire multimodal pipeline

Set the environment variable VLLM_DEVICE before launching vLLM and ensure all sub‑models respect it.

# Before (incorrect)
export VLLM_DEVICE=cuda:0
# Model loader internally overrides vision encoder to cuda:1

# After
export VLLM_DEVICE=cuda:0
export TORCH_CUDA_ALLOC_CONF=max_split_size_mb:128
export TORCH_DTYPE=bfloat16   # consistent for both encoders

2. Use the official NVIDIA runtime

Update Docker daemon configuration (/etc/docker/daemon.json) to include the NVIDIA runtime:

{
  "runtimes": {
    "nvidia": {
      "path": "nvidia-container-runtime",
      "runtimeArgs": []
    }
  },
  "default-runtime": "nvidia"
}

Restart Docker and verify with docker info | grep Runtime that nvidia is the default.

3. Adjust the Docker run command


docker run -d \
  --gpus all \
  --runtime=nvidia \
  -e VLLM_DEVICE=cuda:0 \
  -e TORCH_DTYPE=bfloat16 \
  -v $(pwd)/models:/models \
  myrepo/vllm-multimodal:latest

4. Explicitly set device in the Python entrypoint

import os, torch, vllm

device = os.getenv('VLLM_DEVICE', 'cuda:0')
torch.cuda.set_device(device)

model = vllm.LLM(
    model="llava-1.5-7b",
    dtype=os.getenv('TORCH_DTYPE', 'bfloat16'),
    device=device,
    multimodal=True
)

5. Re‑run a sanity check


curl -X POST http://localhost:8000/generate \
  -H "Content-Type: application/json" \
  -d '{
        "prompt": "Describe the scene:",
        "image": "/data/test.jpg"
      }'

The response should now contain a caption that references visual elements (e.g., “A dog playing with a ball on a grassy field”).

Validation – Confirm Correct Fusion

Log verification: No longer see “device differs” warnings; a single line such as INFO vllm.multimodal: All modalities on cuda:0 appears.
Tensor device check: Running the diagnostic script from the debug section prints identical devices for text and image tensors.
Functional test: The generated text consistently mentions objects present in the input image across multiple prompts.
Performance metric: GPU utilization (via nvidia-smi) shows both encoders consuming memory on the same device, eliminating the previous CPU fallback spike.

Operational Experience – Lessons Learned

During the investigation the following observations proved valuable:

Misleading symptom: The model did not crash; it produced plausible‑looking text, which made the modality failure easy to miss until the output was manually inspected.
Device ordering bug: In a multi‑GPU node, vLLM’s default device selection prefers the first visible GPU. If CUDA_VISIBLE_DEVICES is set to 1,0, the language model may land on cuda:0 (physical GPU 1) while the vision encoder picks the next index, causing the mismatch.
Batch collation nuance: When using torch.utils.data.DataLoader, the default collate function can move image tensors to CPU if the batch sampler runs on a different device. Overriding the collate function to enforce device=VLLM_DEVICE eliminated sporadic failures.
Runtime drift: Upgrading the NVIDIA Container Toolkit from v1.12 to v1.13 changed the default driver path, breaking existing containers that relied on a hard‑coded LD_LIBRARY_PATH. Re‑building the image after the upgrade fixed the issue.

Best Practices and Prevention

Practice	Why it matters	Implementation
Pin a single CUDA device for the whole model	Prevents hidden device splits that cause silent CPU fallbacks	Set `VLLM_DEVICE` and enforce with `torch.cuda.set_device`
Expose all GPUs with `--gpus all`	Ensures every sub‑model can see the same devices	Include flag in every `docker run` command
Synchronize `torch_dtype` across encoders	Avoids shape‑mismatch during token‑image concatenation	Export `TORCH_DTYPE` and pass to vLLM constructor
Use `nvidia` runtime exclusively	Guarantees shared CUDA context between threads	Configure Docker daemon as shown above
Health‑check script	Detects modality split early in CI/CD pipelines	Run a lightweight inference with a known image and assert expected keywords in output

Related Topic Hub: Model Serving Troubleshooting Hub

FAQ

Why does the model work when I run it on the host but fails inside Docker?
Because the host inherits the full CUDA environment, while the container may lack proper GPU exposure or use a different runtime, leading to device mismatches.
Can I run the vision and language encoders on separate GPUs?
vLLM does not currently support cross‑GPU modality fusion; all sub‑models must share the same CUDA context.
What error indicates that the vision encoder fell back to CPU?
A warning like CUDA_VISIBLE_DEVICES is set to an empty string – vision model falls back to CPU or a runtime error about tensors on different devices.
Do I need to rebuild the Docker image after changing --gpus flags?
No, the flag is runtime‑only, but you must restart the container to apply the new GPU visibility.
How can I confirm that both encoders use the same torch_dtype?
Print the dtype after model initialization:
```
print(model.text_encoder.dtype, model.vision_encoder.dtype)
```
Both should output bfloat16 (or whichever dtype you configured).