Problem – “filter expression parse error” when using the --gpus flag
When launching containers for multi‑GPU training on a GPU‑enabled node, the Docker CLI aborts with a parse error similar to:
docker: Error response from daemon: failed to parse filter expression: invalid filter format: "gpu=0"
The same error appears with docker ps, docker run, or any command that includes a GPU‑related filter. The container never starts, breaking distributed training jobs that rely on the NVIDIA runtime.
Root Cause
The failure stems from a mismatch between the Docker client, the Docker daemon, and the NVIDIA container runtime:
- Docker version incompatibility: The
--gpusflag was introduced in Docker 19.03. Older daemons (e.g., Docker 18.09) do not understand the flag and fall back to generic filter parsing, which treats the flag as a malformed filter expression. - Incorrect filter syntax: Users sometimes combine
--gpuswith--filter "gpu=…"(a syntax that never existed). The Docker CLI interprets the--filterargument first, attempts to parse it as a standard filter, and throws the parse error before the--gpusflag is processed. - Legacy
nvidia-docker2installation: On clusters that still have thenvidia-docker2package, the daemon is configured with the legacynvidiaruntime. When the daemon runs an older runtime, it does not register the GPU filter namespace, causing the parser to reject anygpu=filter.
Debug – Investigation Steps
1. Verify Docker client and daemon versions
# docker version
Client: Docker Engine - Community
Version: 20.10.12
API version: 1.41
Go version: go1.16.12
Git commit: e91ed57
Server: Docker Engine - Community
Engine:
Version: 18.09.9
API version: 1.39 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4c52b90
Notice the client is newer than the daemon. The daemon cannot understand --gpus.
2. Inspect the daemon configuration
# cat /etc/docker/daemon.json
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia"
}
If the file contains the legacy nvidia runtime without the newer nvidia-container-toolkit entries, the daemon will not expose GPU‑specific filters.
3. Check the NVIDIA container toolkit version
# dpkg -l | grep nvidia-container-toolkit
ii nvidia-container-toolkit 1.5.0-1 amd64 NVIDIA container runtime toolkit
Versions < 1.0.0 are not compatible with Docker 19.03+ –gpus flag.
4. Reproduce the error with a minimal command
# docker run --gpus all alpine nvidia-smi
docker: Error response from daemon: failed to parse filter expression: invalid filter format: "gpu=all"
5. Capture daemon logs for the failed request
# journalctl -u docker.service -n 20 --no-pager
...
Oct 12 14:22:31 node01 dockerd[1234]: failed to parse filter expression: invalid filter format: "gpu=all"
Oct 12 14:22:31 node01 dockerd[1234]: error processing request: invalid argument
Solution – Bring the stack to a compatible state
1. Upgrade the Docker daemon to 20.10+ (or at least 19.03)
# apt-get update
# apt-get install -y docker-ce=5:20.10.12~3-0~ubuntu-focal docker-ce-cli=5:20.10.12~3-0~ubuntu-focal containerd.io
# systemctl restart docker
2. Replace the legacy nvidia-docker2 runtime with nvidia-container-toolkit
# apt-get purge -y nvidia-docker2
# apt-get install -y nvidia-container-toolkit
# nvidia-ctk runtime configure --runtime=docker
# systemctl restart docker
3. Update daemon.json to use the modern runtime
Before (legacy configuration):
{
"runtimes": {
"nvidia": {
"path": "nvidia-docker-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia"
}
After (toolkit configuration):
{
"runtimes": {
"nvidia": {
"path": "nvidia-container-runtime",
"runtimeArgs": []
}
},
"default-runtime": "nvidia",
"features": {
"gpu": true
}
}
4. Use the correct flag syntax – avoid --filter "gpu=…"
Launch a container with explicit device selection:
# docker run --gpus '"device=0,1"' --rm nvidia/cuda:11.8-base nvidia-smi
Tue Oct 12 14:30:45 2024
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12 Driver Version: 525.85.12 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 Tesla V100-SXM2... Off | 00000000:3B:00.0 Off | 0 |
| N/A 33C P0 30W / 300W | 0MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 Tesla V100-SXM2... Off | 00000000:3C:00.0 Off | 0 |
| N/A 32C P0 28W / 300W | 0MiB / 16160MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Verify – Confirm the fix works
1. Check that the daemon reports GPU support
# docker info | grep -i gpu
Runtimes: nvidia runc
Default Runtime: nvidia
GPU support: enabled
2. Run a quick sanity test
# docker run --gpus all --rm nvidia/cuda:11.8-base nvidia-smi
# docker ps --filter "label=nvidia.com/gpu.present=true"
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
The docker run command should complete without a parse error, and the container should see both GPUs via nvidia-smi.
3. Validate multi‑GPU training launch
# torchrun --nproc_per_node=2 -m torch.distributed.launch \
--nnodes=1 --master_addr=127.0.0.1 \
--master_port=29500 train.py
# Expected log snippet:
[rank 0] Using device: cuda:0
[rank 1] Using device: cuda:1
Prevent – Guardrails for future deployments
- Pin Docker and NVIDIA toolkit versions in your infrastructure-as-code (e.g., Ansible, Terraform) to avoid accidental downgrades.
- Automated health check in CI pipelines:
# docker run --gpus all --rm nvidia/cuda:11.8-base nvidia-smi || exit 1 - Enforce label‑based GPU discovery instead of ad‑hoc
--filter "gpu=…". Document the required labelnvidia.com/gpu.present=truefor any custom monitoring scripts. - Monitor daemon logs for the specific “failed to parse filter expression” pattern and trigger alerts.
- Version compatibility matrix in the run‑book:
Docker Engine nvidia‑container‑toolkit Supported GPU Flag 19.03 – 20.10 1.0.0+ --gpus< 19.03 any Unsupported (use nvidia-docker2legacy)
Related Topic Hub: Distributed Systems Troubleshooting Hub
FAQ
- Why does
docker ps --filter "gpu=0"work on some nodes but not others?
Because only nodes running Docker 19.03+ with the NVIDIA toolkit expose thegpufilter namespace. Older nodes fall back to generic filter parsing and reject the expression. - Can I use
--gpus allwith Docker 18.09?
No. The flag was introduced in Docker 19.03. On 18.09 you must rely on the legacynvidia-dockerruntime and explicit--runtime=nvidiawithout the--gpusflag. - Is there a way to query which GPUs a container sees without running
nvidia-smi?
Yes. Use the Docker API endpoint/containers/{id}/jsonand inspect theHostConfig.DeviceRequestsfield, or query thenvidia.com/gpu.presentlabel that the toolkit automatically adds. - My CI pipeline still fails with “invalid filter format” after upgrading. What should I check?
Ensure the Docker daemon on the CI runner was restarted after the upgrade, verify that/etc/docker/daemon.jsoncontains the"features": {"gpu": true}entry, and confirm that the client and daemon versions match (both ≥ 19.03). - Do I need to set
--gpus '"device=0,1"'for every container, or can I set a default?
You can set a default indaemon.jsonunder"default-gpus"(available in toolkit 1.5+), but explicit per‑container specifications are recommended for reproducibility in multi‑tenant environments.