MLflow Tracking Server Auth Fails After Secret Rotation

Q: Root Cause – Stale Secret Reference in the Running Container

MLflow reads the backend store connection string from MLFLOW_BACKEND_STORE_URI (or SQLALCHEMY_DATABASE_URI) at process start, as documented in the MLflow CLI and environment variable precedence. When a secret rotation occurs (e.g., via AWS Secrets Manager, HashiCorp Vault, or GKE Secrets), the new value is made available in a Secret volume or environment variable injection, but the already‑running tracking server process never reloads the variable.

Q: Option 2 – Sidecar that Watches Secret Changes and Triggers Process Reload

Deploy a lightweight sidecar (e.g., kiamol/secret-reloader) that watches the mounted secret files and sends a SIGHUP or restarts the MLflow process when a change is detected.

In this article

Table of Contents

Problem – Inconsistent Authentication After Secret Rotation

During a blue‑green rollout of an ML pipeline on Kubernetes, the MLflow tracking server stopped logging experiments. The server reported authentication failures against the backend database despite the new deployment using the updated secret ARN.

Typical log excerpt:

2024-06-28 14:32:11,842 ERROR mlflow.store.sqlalchemy_store SQLAlchemyStore: Backend store connection failed: sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: password authentication failed for user "mlflow"
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/mlflow/store/sqlalchemy_store.py", line 112, in __init__
    self._engine = sqlalchemy.create_engine(self.backend_store_uri)
  File ".../sqlalchemy/engine/create.py", line 645, in create_engine
    return engine_from_config(configuration, prefix)
...

Other observed symptoms:

Intermittent mlflow.exceptions.RestException: INVALID_AUTHENTICATION – Unable to authenticate with the backend store errors from client SDKs.
Health‑check endpoint /health returns 503 Service Unavailable after the green pod starts.
Metrics for experiment creation drop to zero while the blue deployment continues to work.

Root Cause – Stale Secret Reference in the Running Container

MLflow reads the backend store connection string from MLFLOW_BACKEND_STORE_URI (or SQLALCHEMY_DATABASE_URI) at process start, as documented in the MLflow CLI and environment variable precedence. When a secret rotation occurs (e.g., via AWS Secrets Manager, HashiCorp Vault, or GKE Secrets), the new value is made available in a Secret volume or environment variable injection, but the already‑running tracking server process never reloads the variable.

In a blue‑green deployment the “green” pod is created from the same container image that still contains the old MLFLOW_BACKEND_STORE_URI value baked into the environment at build time. The secret volume mount does not trigger a restart of the container; the file under /var/run/secrets/.../password is updated, but MLflow continues to use the cached password it loaded during initialization. This matches the pattern described in the official MLflow on Kubernetes guide and the community issue #4595.

Debug – Investigation Steps

Confirm the secret rotation. Use the cloud provider CLI to fetch the latest secret value.

# AWS Secrets Manager
aws secretsmanager get-secret-value --secret-id mlflow-db-cred
# GKE secret
kubectl get secret mlflow-db-cred -o yaml

Inspect the pod’s environment.

# Inside the green pod
printenv | grep MLFLOW_BACKEND_STORE_URI
cat /var/run/secrets/kubernetes.io/serviceaccount/password

Expected output shows the new password, but the process may still be using the old one.

Check MLflow server logs for the exact connection string. MLflow logs the sanitized URI on startup (masked password). Compare with the secret you retrieved.
Validate that the container image does not embed the password. Search Dockerfile or Helm values for hard‑coded DATABASE_URL.
Reproduce the failure locally. Run the same mlflow server command with the stale environment variable and observe the same OperationalError.

Solution – Refreshing Credentials During Blue‑Green Switch

The fix consists of ensuring that the tracking server process reads the latest secret before it starts handling requests. Two common patterns are:

Option 1 – Init Container that Resolves Secrets at Pod Start

Use an init container to fetch the current secret and write it to a file that is then sourced by the main container.

# values.yaml (Helm)
initContainers:
  - name: secret-fetcher
    image: amazon/aws-cli
    command: ["sh", "-c"]
    args:
      - |
        SECRET=$(aws secretsmanager get-secret-value --secret-id mlflow-db-cred --query SecretString --output text)
        echo $SECRET > /etc/mlflow/secret.env
    volumeMounts:
      - name: mlflow-secret
        mountPath: /etc/mlflow
volumes:
  - name: mlflow-secret
    emptyDir: {}

Then modify the main container entrypoint to source the file:

# entrypoint.sh
#!/bin/sh
set -e
if [ -f /etc/mlflow/secret.env ]; then
  export $(cat /etc/mlflow/secret.env | xargs)
fi
exec mlflow server \
  --backend-store-uri "$MLFLOW_BACKEND_STORE_URI" \
  --host 0.0.0.0 --port 5000

Option 2 – Sidecar that Watches Secret Changes and Triggers Process Reload

Deploy a lightweight sidecar (e.g., kiamol/secret-reloader) that watches the mounted secret files and sends a SIGHUP or restarts the MLflow process when a change is detected.

# deployment.yaml excerpt
containers:
  - name: mlflow-tracking
    image: mlflow:2.9.0
    envFrom:
      - secretRef:
          name: mlflow-db-cred
    ports: [{containerPort: 5000}]
  - name: secret-reloader
    image: ghcr.io/kiamol/secret-reloader:latest
    args: ["--watch-dir=/var/run/secrets/mlflow-db-cred"]
    volumeMounts:
      - name: mlflow-db-cred
        mountPath: /var/run/secrets/mlflow-db-cred
    env:
      - name: RELOAD_COMMAND
        value: "pkill -HUP mlflow"

Before / After Comparison of Deployment Manifest

Aspect	Before (Failing)	After (Fixed)
Secret handling	Mounted secret volume only; no reload mechanism.	Init container resolves secret at pod start OR sidecar reloads on change.
Env vars	Static `MLFLOW_BACKEND_STORE_URI` baked into image.	Exported dynamically from resolved secret.
Deployment strategy	Blue‑green swap without pod recreation.	Green pod always starts fresh with latest secret; blue pod is drained.

Verify – Validation Steps After Applying the Fix

Trigger a secret rotation (e.g., rotate the DB password in Secrets Manager).
Deploy the updated Helm chart or manifest.

Confirm the pod logs contain the new sanitized URI:

2024-06-28 14:45:02,113 INFO mlflow.store.sqlalchemy_store SQLAlchemyStore: Using backend store URI postgresql://mlflow:*****@db.example.com/mlflow

Run a quick experiment logging command:
```
mlflow run ./my_project -P param=1
mlflow experiments list
```
Verify that the experiment appears without authentication errors.

Check health endpoint:

curl -s http://mlflow-service:5000/health | jq .status
# Expected output: "healthy"

Inspect metrics (e.g., Prometheus mlflow_experiment_created_total) for a non‑zero increase.

Prevent – Best Practices for Secret Management with MLflow

Never embed credentials in container images. Use secret references exclusively.
Make secret consumption a start‑up activity. Either resolve secrets in an init container or ensure the process restarts when the secret file changes.
Leverage Kubernetes secret-reload sidecars. Projects such as kiamol/secret-reloader or bitnami/sealed-secrets provide reliable watch‑and‑reload semantics.
Include liveness/readiness probes that fail fast on DB connection errors. This forces the orchestrator to restart pods with fresh secrets.
Version your secret ARN/ID in the deployment manifest. Updating the reference forces a new pod creation during blue‑green swaps.
Monitor authentication errors. Alert on sqlalchemy.exc.OperationalError with message “password authentication failed” to catch stale credentials early.

FAQ – Common Follow‑Up Questions

Why does the blue pod continue to work after the secret rotation?
Because it was started before the rotation and retains the old password in memory. Only newly started pods load the refreshed secret.
Can I avoid a full pod restart and still refresh the password?
Yes. Use a sidecar that watches the secret volume and sends a SIGHUP or triggers a graceful process restart, as shown in Option 2.
What if I use HashiCorp Vault dynamic credentials?
Configure a Vault Agent sidecar with auto_auth and template to write the DATABASE_URL to a file, then let the init container or reload sidecar pick it up.
How do I test that secret rotation works before deploying to production?
Create a staging namespace, rotate the secret, redeploy the tracking server with the new manifest, and run a smoke test that logs an experiment and checks the health endpoint.
Is there a way to make MLflow re‑read environment variables without restarting?
MLflow does not support runtime env‑var reload. The process must be restarted or signaled to re‑initialize the SQLAlchemy engine, which is effectively a restart of the server process.

Related Topic Hub: Model Serving Troubleshooting Hub