cable network

LangChain PVC stuck in pending state after deployment

by

Problem – LangChain PVC Stuck in Pending After Deployment

When deploying LangChain components (e.g., the vector‑store or document‑loader pods) on a managed Kubernetes cluster with dynamic storage provisioning enabled, the associated PersistentVolumeClaim (PVC) often remains in the Pending state. The pod cannot start, leading to crash loops and a non‑functional AI workflow.

Typical symptoms observed in the cluster:

  • Pod status: CrashLoopBackOff or Init:0/1 because the volume cannot be mounted.
  • PVC description shows status: Pending and no bound PV.
  • Controller manager logs contain errors such as:
persistentvolumeclaim "langchain-vectorstore-pvc" is not bound: no persistent volumes available for claim
failed to provision volume with storage class "standard-rwo": storageclass.storage.k8s.io "standard-rwo" not found
error creating volume: failed to create volume for claim "langchain-vectorstore-pvc": insufficient quota

Root Cause Analysis

1. StorageClass name mismatch

LangChain’s Helm chart (see GitHub repository) defaults to storageClassName: "standard-rwo" for read‑write‑once (RWO) volumes. On many managed clusters (GKE, AKS, OpenShift) the available provisioner is named standard or managed-premium. When the PVC references a non‑existent class, the dynamic provisioner never triggers, leaving the claim pending.

2. Access‑mode incompatibility

Some environments expose only ReadWriteMany (RWX) storage (e.g., Azure managed-premium), while LangChain requests ReadWriteOnce (RWO). The provisioner cannot satisfy the claim, resulting in the same pending state.

3. Disabled dynamic provisioning or missing quota

In OpenShift projects where allowVolumeExpansion is false or the namespace lacks a default StorageClass, the PVC will not be auto‑bound. Additionally, insufficient storage quota will cause the provisioner to reject the request.

These root causes align with real incidents documented in the evidence package: the GKE incident with standard-rwo vs standard, the AKS case with RWX vs RWO, and the OpenShift namespace lacking a provisioner.

Investigation and Debugging Steps

Step 1 – Inspect the PVC

kubectl get pvc -n langchain -o wide

Sample output:

NAME                       STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
langchain-vectorstore-pvc  Pending   <none>   10Gi       RWO            standard-rwo   2m

Step 2 – Verify available StorageClasses

kubectl get storageclass

Typical output on GKE:

NAME            PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard        pd.csi.storage.googleapis.com   Delete          Immediate           true                  45d
standard-rwo    pd.csi.storage.googleapis.com   Delete          Immediate           true                  45d

If the expected class is missing, the PVC cannot bind.

Step 3 – Check controller manager logs for provisioning errors

kubectl logs -n kube-system -l component=controller-manager -c kube-controller-manager --tail=100

Search for the PVC name or the storage class name.

Step 4 – Validate namespace quota

kubectl describe quota -n langchain

Look for storage limits and used values.

Step 5 – Confirm access‑mode support

kubectl get sc standard -o yaml | grep -i accessModes

Some cloud‑providers expose only ReadWriteMany for certain classes.

Resolution – Making the PVC Bind

Option A: Align StorageClass name

Update the Helm values (or raw manifest) to reference an existing class.

Before (default values.yaml excerpt):

persistence:
  enabled: true
  storageClassName: "standard-rwo"
  accessModes:
    - ReadWriteOnce
  size: 10Gi

After – set to the cluster’s default class (standard) or any class that exists:

persistence:
  enabled: true
  storageClassName: "standard"
  accessModes:
    - ReadWriteOnce
  size: 10Gi

Apply the change:

helm upgrade langchain ./langchain-chart -f values.yaml

Option B: Adjust Access Mode to match provisioner

If only RWX is offered (e.g., Azure managed-premium), modify the claim:

persistence:
  enabled: true
  storageClassName: "managed-premium"
  accessModes:
    - ReadWriteMany
  size: 10Gi

Option C: Create a static PersistentVolume (OpenShift case)

When dynamic provisioning is disabled, define a PV that satisfies the claim:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: langchain-static-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: "manual"
  hostPath:
    path: /mnt/data/langchain
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: langchain-vectorstore-pvc
  namespace: langchain
spec:
  storageClassName: "manual"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Apply both objects, then redeploy the pod.

Option D: Increase quota or enable dynamic provisioning

If logs indicate “insufficient quota”, request a larger quota from the cloud provider or clean up unused PVCs.

Verification – Confirming the Fix

  1. Re‑query the PVC:
kubectl get pvc -n langchain langchain-vectorstore-pvc -o yaml

Expected status.phase should be Bound and a volumeName populated.

  1. Check the pod status:
kubectl get pod -n langchain -l app=langchain-vectorstore

Pod should transition to Running without mount errors.

  1. Inspect container logs for successful start‑up messages (see LangChain deployment guide).
kubectl logs -n langchain -l app=langchain-vectorstore --tail=20

Typical success line:

INFO: Vector store initialized with persistent volume /data/vectorstore

Prevention – Operational Guardrails

  • Validate manifests against the cluster’s storage catalog before applying. Automate with kubectl apply --dry-run=client -k and a CI check that runs kubectl get storageclass to ensure the referenced class exists.
  • Standardize a default StorageClass in each environment (GKE: standard, AKS: managed-premium) and pin the LangChain chart to that name via Helm values.
  • Document required access mode per cloud provider. For multi‑node deployments that need shared storage, override accessModes to ReadWriteMany explicitly.
  • Enable quota monitoring and set alerts on PVC creation failures using the kube_persistentvolumeclaim_status_phase metric.
  • Run a post‑deployment sanity check that queries all PVCs in the langchain namespace and fails the CI pipeline if any remain Pending after a configurable timeout.

FAQ – Common Follow‑Up Questions

  1. Why does the PVC bind on my local minikube cluster but stay pending on GKE?
    Because minikube ships with a default standard StorageClass that matches the chart’s default, while GKE may only have standard-rwo or a custom class. The mismatch prevents dynamic provisioning.
  2. Can I use a ReadWriteMany class with LangChain’s vector store?
    Yes. Override accessModes: ["ReadWriteMany"] and ensure the underlying provisioner supports RWX (e.g., Azure Files). The application will treat the volume as a shared directory.
  3. What does “failed to provision volume with storage class … not found” mean?
    The controller manager attempted to locate a StorageClass object with the name specified in the PVC but could not find it. This is usually a typo or a class that was not created in the cluster.
  4. How do I troubleshoot “insufficient quota” errors?
    Run kubectl describe quota -n <ns> to see current usage. If the requested size exceeds the quota, request an increase from the cloud admin or delete unused PVCs.
  5. Is it safe to change the storage class of an existing PVC?
    Kubernetes does not allow changing the storageClassName of a bound PVC. You must delete the PVC (and any dependent pod) and recreate it with the correct class, or migrate data to a new PVC.

Related Topic Hub: RAG Systems Troubleshooting Hub