secret-management

MLflow tracking server auth fails after secret rotation

by
a close-up of a server room

Problem – Inconsistent Authentication After Secret Rotation During a blue‑green rollout of an ML pipeline on Kubernetes, the MLflow tracking server stopped logging experiments. The server reported authentication failures against the backend database despite the new deployment using the updated secret ARN. Typical log excerpt: 2024-06-28 14:32:11,842 ERROR mlflow.store.sqlalchemy_store SQLAlchemyStore: Backend store connection failed: sqlalchemy.exc.OperationalError: … Read more

Kubernetes secret rotation failure for multi-GPU training pods

by
a group of blue boxes

Problem: Kubernetes Secret Rotation Failure for Multi‑GPU Training Pods In a production AI training platform, each training job runs in a pod that requests multiple GPUs. Access to the GPU resource provider (e.g., NVIDIA GPU Cloud, a proprietary licensing server, or a cloud‑native GPU quota service) is gated by short‑lived credentials stored in a Secret. … Read more