AMD GPU CLI Argument Parsing Failure During Disaster Recovery

In this article

Table of Contents

AMD GPU CLI Argument Parsing Failure During Disaster Recovery

Problem Description (Symptoms and Impact)

During automated disaster‑recovery runs on AMD GPU clusters, the recovery scripts invoke ROCm utilities such as rocm-smi, rocminfo, and rocm-bandwidth-test. The scripts abort with parsing errors, causing the scheduler (e.g., Slurm) to mark all GPUs as unavailable. Typical log excerpts are:


[2024-05-12 03:14:27] ERROR: Invalid argument '--gpu' for command 'rocm-smi'
[2024-05-12 03:14:27] rocm-smi: argument parsing error (code 2)
[2024-05-12 03:14:28] Failed to parse GPU allocation flags: unrecognized option '--device-ids=0,1'

Consequences include:

Job resubmission loops because the scheduler cannot allocate GPUs.
Extended recovery windows – in the Azure HPC outage (Q3 2023) the delay added ~45 minutes.
Potential resource leakage when stale allocation entries persist.

Root Cause Analysis

The failure originates from how ROCm command‑line parsers handle list arguments. According to the rocm-smi Manual – Argument Syntax and Validation, supported list flags must be supplied either as repeated flags (--gpu 0 --gpu 1) or as a single quoted, space‑separated list (--gpu="0 1"). Comma‑separated lists (--gpu=0,1) are explicitly undocumented and trigger the generic “Invalid argument” error.

Two interacting factors exacerbate the problem in disaster‑recovery contexts:

Script‑generated argument strings. Recovery automation platforms (Azure Automation runbooks, internal Python wrappers) often build the flag string by joining an integer array with commas, e.g., "--gpu=" + ",".join(gpu_ids). This pattern matches the community‑reported issue in GitHub #842.
ROCm 5.x known parsing limitation. The “Known Issues” section of the ROCm 5.x release notes lists “Parsing failures for ‘–gpu’ and ‘–device’ arguments when supplied as comma‑separated values in non‑interactive scripts.” The parser does not normalize commas to spaces, leading to early termination with error code 2.

Therefore, the root cause is a mismatch between the script‑generated CLI syntax and the strict argument grammar enforced by ROCm utilities.

Investigation and Debugging Steps

Follow this checklist to reproduce and isolate the parsing failure:

Capture the exact command line generated by the recovery job.


# Example from Azure Automation log
2024-05-12T03:14:26Z RUNNING: rocm-smi --gpu=0,1 --setfan=80

Validate the flag against the official CLI guide.


$ rocm-smi --help | grep -A2 "\-\-gpu"
  --gpu             Set GPU index (repeatable)
  --gpu-list ""    Space‑separated list of GPU IDs (quoted)

Test alternative flag formats manually.


# Rejected format (observed failure)
$ rocm-smi --gpu=0,1
Error: Invalid argument '--gpu' for command 'rocm-smi'

# Accepted formats
$ rocm-smi --gpu 0 --gpu 1
GPU[0] : Power = 120W
GPU[1] : Power = 115W

$ rocm-smi --gpu-list "0 1"
GPU[0] : Power = 120W
GPU[1] : Power = 115W

Inspect the parsing code path (optional). The ROCm source under src/cli/parser.cpp shows a split(',') is not performed; only whitespace tokenization is applied.
Check for environment‑specific overrides. Some distributions ship a wrapper script (/usr/local/bin/rocm-smi) that pre‑processes arguments. Verify its presence with type -a rocm-smi.

Resolution (Fix Implementation)

Update the recovery scripts to generate flags that conform to the documented syntax. Below are before/after code snippets for a typical Python‑based recovery wrapper.

Before (faulty argument construction)


def build_rocm_smi_cmd(gpu_ids):
    # gpu_ids is a list of integers, e.g., [0, 1]
    gpu_arg = "--gpu=" + ",".join(map(str, gpu_ids))
    return ["rocm-smi", gpu_arg, "--setfan=80"]

After (compliant argument construction)


def build_rocm_smi_cmd(gpu_ids):
    # Use repeatable flag or quoted space‑separated list
    # Option 1: repeatable --gpu flags
    cmd = ["rocm-smi"]
    for gid in gpu_ids:
        cmd.extend(["--gpu", str(gid)])
    cmd.append("--setfan=80")
    return cmd

# Option 2: single quoted list
def build_rocm_smi_cmd_list(gpu_ids):
    gpu_list = " ".join(map(str, gpu_ids))
    return ["rocm-smi", f'--gpu-list="{gpu_list}"', "--setfan=80"]

Deploy the corrected script and rerun the recovery job. The parser now accepts the arguments, and GPU allocation proceeds without error.

Verification (Validation Steps)

After applying the fix, perform the following checks:

Run the command manually.


$ rocm-smi --gpu 0 --gpu 1 --setfan=80
GPU[0] : Fan Speed = 80%
GPU[1] : Fan Speed = 80%

Inspect the recovery job logs for the absence of parsing errors.


2024-05-12 03:17:02 INFO: Executed: rocm-smi --gpu 0 --gpu 1 --setfan=80
2024-05-12 03:17:02 INFO: GPU allocation successful

Confirm scheduler view. In Slurm, scontrol show node <node> should list the GPUs as State=IDLE rather than DOWN.
Run a quick workload test. Submit a dummy training job that requests --gpus=2 and verify it starts.

Operational Experience (Lessons Learned)

Misleading symptom: Initial inspection suggested a driver regression because the same command worked in interactive shells. The failure only manifested when the command string was assembled by automation.
Common incorrect assumption: “If the flag works in the terminal, the same string works in scripts.” In fact, quoting rules differ between shells and the Python subprocess module.
Production edge case: Azure Automation’s PowerShell runbooks concatenate arrays with commas by default. Explicitly converting to space‑separated strings solved the issue without code changes to the ROCm tools.
Community workaround: Some teams patched rocm-smi to accept commas (see GitHub issue #1175), but upstream has not merged the change due to ambiguity with future flag extensions.

Best Practices and Prevention

Practice	Why it helps
Generate CLI flags using language‑specific argument arrays (e.g., `subprocess.run([...])`) rather than string concatenation.	Avoids quoting/escaping errors and guarantees correct tokenization.
Prefer repeatable flags (`--gpu 0 --gpu 1`) over list flags when the tool supports them.	Repeatable flags are unambiguous across shells and scripting languages.
Validate argument syntax with a dry‑run step before invoking the actual tool.	Catches malformed strings early; e.g., `rocm-smi --dry-run --gpu-list "0 1"`.
Monitor ROCm tool exit codes (0 = success, 2 = parsing error) and alert on non‑zero codes during recovery jobs.	Provides immediate feedback that a script‑generated command is invalid.
Pin ROCm version and track “Known Issues” in release notes before upgrades.	Prevents surprise regressions; the 5.x parsing limitation is documented.

FAQ (Related Questions)

Why does rocm-smi accept --gpu 0 1 but reject --gpu=0,1? The parser only tokenizes whitespace; commas are treated as part of the token, resulting in an unrecognized option.
Can I use environment variables to pass GPU lists to ROCm tools? Yes. Set ROCM_GPU_LIST="0 1" and invoke rocm-smi --gpu-list "$ROCM_GPU_LIST". Ensure the variable is quoted to preserve spaces.
Is there a way to make the parser accept commas without code changes? No built‑in flag; you must transform the list before invoking the tool or wrap the call in a small shell script that replaces commas with spaces.
Do newer ROCm releases (e.g., 6.0) fix this parsing issue? As of ROCm 6.0‑rc1, the documentation still recommends whitespace‑separated lists. No change has been announced.
How should I handle GPU masks (--gpu-mask) in recovery scripts? The mask expects a hexadecimal bitmask (e.g., 0x3 for GPUs 0 and 1). Convert your list to a mask programmatically and pass it as a single argument.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub