Docker filter expression parse error when using –gpus flag

Problem – “filter expression parse error” when using the --gpus flag

When launching containers for multi‑GPU training on a GPU‑enabled node, the Docker CLI aborts with a parse error similar to:

docker: Error response from daemon: failed to parse filter expression: invalid filter format: "gpu=0"

The same error appears with docker ps, docker run, or any command that includes a GPU‑related filter. The container never starts, breaking distributed training jobs that rely on the NVIDIA runtime.

Root Cause

The failure stems from a mismatch between the Docker client, the Docker daemon, and the NVIDIA container runtime:

  • Docker version incompatibility: The --gpus flag was introduced in Docker 19.03. Older daemons (e.g., Docker 18.09) do not understand the flag and fall back to generic filter parsing, which treats the flag as a malformed filter expression.
  • Incorrect filter syntax: Users sometimes combine --gpus with --filter "gpu=…" (a syntax that never existed). The Docker CLI interprets the --filter argument first, attempts to parse it as a standard filter, and throws the parse error before the --gpus flag is processed.
  • Legacy nvidia-docker2 installation: On clusters that still have the nvidia-docker2 package, the daemon is configured with the legacy nvidia runtime. When the daemon runs an older runtime, it does not register the GPU filter namespace, causing the parser to reject any gpu= filter.

Debug – Investigation Steps

1. Verify Docker client and daemon versions

# docker version
Client: Docker Engine - Community
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.12
 Git commit:        e91ed57

Server: Docker Engine - Community
 Engine:
  Version:          18.09.9
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       4c52b90

Notice the client is newer than the daemon. The daemon cannot understand --gpus.

2. Inspect the daemon configuration

# cat /etc/docker/daemon.json
{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia"
}

If the file contains the legacy nvidia runtime without the newer nvidia-container-toolkit entries, the daemon will not expose GPU‑specific filters.

3. Check the NVIDIA container toolkit version

# dpkg -l | grep nvidia-container-toolkit
ii  nvidia-container-toolkit   1.5.0-1   amd64   NVIDIA container runtime toolkit

Versions < 1.0.0 are not compatible with Docker 19.03+ –gpus flag.

4. Reproduce the error with a minimal command

# docker run --gpus all alpine nvidia-smi
docker: Error response from daemon: failed to parse filter expression: invalid filter format: "gpu=all"

5. Capture daemon logs for the failed request

# journalctl -u docker.service -n 20 --no-pager
...
Oct 12 14:22:31 node01 dockerd[1234]: failed to parse filter expression: invalid filter format: "gpu=all"
Oct 12 14:22:31 node01 dockerd[1234]: error processing request: invalid argument

Solution – Bring the stack to a compatible state

1. Upgrade the Docker daemon to 20.10+ (or at least 19.03)

# apt-get update
# apt-get install -y docker-ce=5:20.10.12~3-0~ubuntu-focal docker-ce-cli=5:20.10.12~3-0~ubuntu-focal containerd.io
# systemctl restart docker

2. Replace the legacy nvidia-docker2 runtime with nvidia-container-toolkit

# apt-get purge -y nvidia-docker2
# apt-get install -y nvidia-container-toolkit
# nvidia-ctk runtime configure --runtime=docker
# systemctl restart docker

3. Update daemon.json to use the modern runtime

Before (legacy configuration):

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-docker-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia"
}

After (toolkit configuration):

{
    "runtimes": {
        "nvidia": {
            "path": "nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia",
    "features": {
        "gpu": true
    }
}

4. Use the correct flag syntax – avoid --filter "gpu=…"

Launch a container with explicit device selection:

# docker run --gpus '"device=0,1"' --rm nvidia/cuda:11.8-base nvidia-smi
Tue Oct 12 14:30:45 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.85.12    Driver Version: 525.85.12    CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla V100-SXM2...   Off | 00000000:3B:00.0 Off |                    0 |
| N/A   33C    P0    30W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla V100-SXM2...   Off | 00000000:3C:00.0 Off |                    0 |
| N/A   32C    P0    28W / 300W |      0MiB / 16160MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

Verify – Confirm the fix works

1. Check that the daemon reports GPU support

# docker info | grep -i gpu
Runtimes: nvidia runc
Default Runtime: nvidia
GPU support: enabled

2. Run a quick sanity test

# docker run --gpus all --rm nvidia/cuda:11.8-base nvidia-smi
# docker ps --filter "label=nvidia.com/gpu.present=true"
CONTAINER ID   IMAGE                     COMMAND   CREATED   STATUS    PORTS     NAMES

The docker run command should complete without a parse error, and the container should see both GPUs via nvidia-smi.

3. Validate multi‑GPU training launch

# torchrun --nproc_per_node=2 -m torch.distributed.launch \
    --nnodes=1 --master_addr=127.0.0.1 \
    --master_port=29500 train.py
# Expected log snippet:
[rank 0] Using device: cuda:0
[rank 1] Using device: cuda:1

Prevent – Guardrails for future deployments

  • Pin Docker and NVIDIA toolkit versions in your infrastructure-as-code (e.g., Ansible, Terraform) to avoid accidental downgrades.
  • Automated health check in CI pipelines:
    # docker run --gpus all --rm nvidia/cuda:11.8-base nvidia-smi || exit 1
  • Enforce label‑based GPU discovery instead of ad‑hoc --filter "gpu=…". Document the required label nvidia.com/gpu.present=true for any custom monitoring scripts.
  • Monitor daemon logs for the specific “failed to parse filter expression” pattern and trigger alerts.
  • Version compatibility matrix in the run‑book:
    Docker Engine nvidia‑container‑toolkit Supported GPU Flag
    19.03 – 20.10 1.0.0+ --gpus
    < 19.03 any Unsupported (use nvidia-docker2 legacy)

Related Topic Hub: Distributed Systems Troubleshooting Hub

FAQ

  1. Why does docker ps --filter "gpu=0" work on some nodes but not others?
    Because only nodes running Docker 19.03+ with the NVIDIA toolkit expose the gpu filter namespace. Older nodes fall back to generic filter parsing and reject the expression.
  2. Can I use --gpus all with Docker 18.09?
    No. The flag was introduced in Docker 19.03. On 18.09 you must rely on the legacy nvidia-docker runtime and explicit --runtime=nvidia without the --gpus flag.
  3. Is there a way to query which GPUs a container sees without running nvidia-smi?
    Yes. Use the Docker API endpoint /containers/{id}/json and inspect the HostConfig.DeviceRequests field, or query the nvidia.com/gpu.present label that the toolkit automatically adds.
  4. My CI pipeline still fails with “invalid filter format” after upgrading. What should I check?
    Ensure the Docker daemon on the CI runner was restarted after the upgrade, verify that /etc/docker/daemon.json contains the "features": {"gpu": true} entry, and confirm that the client and daemon versions match (both ≥ 19.03).
  5. Do I need to set --gpus '"device=0,1"' for every container, or can I set a default?
    You can set a default in daemon.json under "default-gpus" (available in toolkit 1.5+), but explicit per‑container specifications are recommended for reproducibility in multi‑tenant environments.