RAG Citation Formatting Issues With AMD GPU In Kubernetes

Q: 1. Align ROCm driver and PyTorch versions

Downgrade the driver to 5.4 (compatible with torch==2.2.0) or upgrade PyTorch to a ROCm‑compatible build (torch==2.3.0+rocm5.6). The example below uses the driver downgrade, which is the least disruptive for existing workloads.

Q: 2. Synchronize citation schema between services

Update the generation service’s CitationFormatter to accept the full schema and forward missing fields unchanged.

Q: 3. Fix citation cache volume permissions

Mount the cache with fsGroup: 1001 so the application user can write.

In this article

Table of Contents

Problem Description

In a Kubernetes‑based microservices architecture that uses AMD GPUs for Retrieval‑Augmented Generation (RAG), the generated citations are malformed. Observed symptoms include:

Missing source fields in the JSON payload returned by the generation service.
Inconsistent or empty metadata.title and metadata.author entries.
Intermittent duplicate citation entries when pods restart under memory pressure.
Log excerpts such as:


2026-06-30T12:14:02.317Z ERROR CitationFormattingError: missing required field 'source' in citation payload
2026-06-30T12:14:02.319Z WARN  RAGPipelineError: metadata fields mismatch – expected ['title','url','author'] but received []
2026-06-30T12:14:03.001Z INFO  PodCrashLoopBackOff: /app/formatter.py raised ValueError: citation metadata incomplete

The issue only appears when the inference pod runs on a ROCm‑enabled container; the same code path works correctly on NVIDIA GPUs or CPU‑only nodes.

Root Cause Analysis

The malfunction stems from a combination of driver‑runtime incompatibility and schema drift between the retrieval and generation microservices:

ROCm driver version mismatch – After a cluster‑wide ROCm driver upgrade to 5.6, the installed torch wheel (version 2.2.0) became incompatible (AMD ROCm Documentation – Installation guide). The runtime silently fell back to CPU execution, bypassing GPU‑accelerated token extraction that populates citation fields.
JSON schema divergence – The retrieval service emits citations conforming to the schema defined in v1.4 of the internal contract ({source, metadata:{title,url,author}}). The generation service, built against an older v1.2 contract, expects only source and url. When the two services are deployed on different nodes, the mismatch causes the CitationFormatter to drop unrecognized fields, resulting in empty metadata sections (ROCm Runtime for Kubernetes – Best practices).
Pod restart side‑effects – GPU memory pressure triggers pod restarts. Because the shared citation cache volume is mounted without fsGroup permissions on ROCm nodes, the formatter cannot write temporary JSON files, leading to duplicate or truncated entries (Haystack discussion).

Investigation and Debugging

The following steps reproduced the failure and isolated the root causes:

Verify ROCm driver and torch compatibility

kubectl exec -it rag-gen-0 -- bash -c "rocminfo | grep 'Driver Version'"
# Expected output: ROCm driver version: 5.6

kubectl exec -it rag-gen-0 -- bash -c "python -c 'import torch; print(torch.__version__)'"
# Output: 2.2.0

The combination above triggers the known incompatibility warning from the ROCm‑OpenAI integration guide (ROCm‑OpenAI Integration Guide).

Inspect citation payloads at the retrieval‑generation boundary

# In the retrieval pod
curl -s http://localhost:8080/retrieve?query="climate change" | jq '.citations'
[
  {
    "source": "https://example.com/article1",
    "metadata": {
      "title": "Impact of Climate Change",
      "author": "J. Doe",
      "url": "https://example.com/article1"
    }
  },
  ...
]

# In the generation pod (failing)
curl -s http://localhost:8081/generate | jq '.citations'
[
  {
    "source": "https://example.com/article1",
    "metadata": {}
  },
  ...
]

The empty metadata object confirms the schema mismatch.

Check volume permissions on ROCm nodes

kubectl exec -it rag-formatter-0 -- bash -c "ls -l /var/cache/citations"
drwxr-xr-x 2 root root 4096 Jun 30 12:00 .

The pod runs as user appuser (uid 1001) and cannot write to the directory, causing silent failures in CitationFormatter.

Resolution

Apply the following coordinated fixes across the cluster.

1. Align ROCm driver and PyTorch versions

Downgrade the driver to 5.4 (compatible with torch==2.2.0) or upgrade PyTorch to a ROCm‑compatible build (torch==2.3.0+rocm5.6). The example below uses the driver downgrade, which is the least disruptive for existing workloads.

# DaemonSet that installs the correct driver on each node
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: rocm-driver-installer
spec:
  selector:
    matchLabels:
      name: rocm-driver
  template:
    metadata:
      labels:
        name: rocm-driver
    spec:
      containers:
      - name: installer
        image: amd/rocm-installer:5.4
        securityContext:
          privileged: true
        env:
        - name: ROCM_VERSION
          value: "5.4"
        volumeMounts:
        - name: dev
          mountPath: /dev
      volumes:
      - name: dev
        hostPath:
          path: /dev

After rollout, verify:

kubectl exec -it rag-gen-0 -- bash -c "rocminfo | grep 'Driver Version'"
# Output: ROCm driver version: 5.4

2. Synchronize citation schema between services

Update the generation service’s CitationFormatter to accept the full schema and forward missing fields unchanged.

# Before (generation service)
def format_citation(citation):
    if 'source' not in citation:
        raise CitationFormattingError("missing required field 'source'")
    # Only source and url are kept
    return {
        "source": citation['source'],
        "url": citation.get('url', '')
    }

# After (generation service)
def format_citation(citation):
    required = ['source']
    for field in required:
        if field not in citation:
            raise CitationFormattingError(f"missing required field '{field}'")
    # Preserve full metadata
    return {
        "source": citation['source'],
        "metadata": citation.get('metadata', {})
    }

Rebuild the container image and roll out the updated deployment.

3. Fix citation cache volume permissions

Mount the cache with fsGroup: 1001 so the application user can write.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: citation-cache-pvc
spec:
  accessModes: [ "ReadWriteMany" ]
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-formatter
spec:
  template:
    spec:
      securityContext:
        fsGroup: 1001
      containers:
      - name: formatter
        image: myorg/rag-formatter:latest
        volumeMounts:
        - name: citation-cache
          mountPath: /var/cache/citations
      volumes:
      - name: citation-cache
        persistentVolumeClaim:
          claimName: citation-cache-pvc

Verification

After applying the three fixes, perform the following checks:

Confirm driver‑torch compatibility:

kubectl exec -it rag-gen-0 -- bash -c "python -c 'import torch; print(torch.version.rocm)'" 
# Output: 5.4

Run an end‑to‑end request and inspect the final citation JSON:

curl -s http://rag-gateway/api/generate?query="AI safety" | jq '.citations[0]'
{
  "source": "https://example.com/ai-safety",
  "metadata": {
    "title": "AI Safety Research",
    "author": "A. Smith",
    "url": "https://example.com/ai-safety"
  }
}

All expected fields are present and correctly populated.

Check pod logs for absence of previous error messages:

kubectl logs -l app=rag-formatter --tail=20
2026-06-30T12:45:10.112Z INFO  CitationFormatter: formatted 42 citations successfully

Validate that pod restarts no longer produce duplicate citations:

kubectl get pods -l app=rag-formatter -o wide
NAME           READY   STATUS    RESTARTS   AGE
rag-formatter-0 1/1    Running   0          2h

Prevention and Best Practices

Pin ROCm and PyTorch versions together in a requirements.txt or conda environment and automate compatibility checks in CI.
Version‑lock the citation schema using a shared OpenAPI definition stored in a central repository; enforce the contract with contract‑testing tools (e.g., pact).
Enable health probes that validate citation payload structure before the generation service marks the pod as ready.
Monitor GPU driver and runtime mismatches via Prometheus metrics exported by rocm-smi and alert on torch.version.rocm discrepancies.
Set fsGroup or use init containers to adjust volume permissions on all ROCm nodes, preventing silent write failures.

Related Topic Hub: GPU Infrastructure Troubleshooting Hub

FAQ

Why does the citation formatter work on NVIDIA GPUs but fail on AMD GPUs?
Because the NVIDIA stack uses CUDA‑compatible PyTorch wheels that match the driver version, while the ROCm stack requires explicit version alignment. A driver upgrade without a matching PyTorch build forces a CPU fallback, which skips GPU‑accelerated metadata extraction.
How can I detect a schema mismatch before deployment?
Add contract tests that serialize a sample citation from the retrieval service and feed it to the generation service’s formatter. Fail the CI pipeline if any required field is dropped.
Is there a way to make the citation cache resilient to pod restarts?
Mount the cache as a ReadWriteMany volume with proper fsGroup permissions, and configure the formatter to recreate missing files on startup.
What ROCm driver version should I pair with PyTorch 2.2.0?
According to the AMD ROCm OpenAI Integration Guide, PyTorch 2.2.0 is compatible with ROCm driver 5.4.x. For newer drivers (e.g., 5.6), upgrade to a PyTorch wheel built for that driver (e.g., torch==2.3.0+rocm5.6).
Can I use the same container image for both NVIDIA and AMD nodes?
Yes, if the image contains both CUDA and ROCm libraries and selects the appropriate backend at runtime. However, you must still ensure the host driver versions match the bundled libraries for each architecture.