Weaviate Container Crash Loop In Kubernetes Deployment

Q: Problem – Weaviate Container CrashLoopBackOff in Kubernetes

A production deployment of Weaviate on a multi‑node Kubernetes cluster repeatedly enters CrashLoopBackOff. The pod terminates shortly after start, causing the service to be unavailable for queries. Typical symptoms observed in the cluster:

Q: Debug – Systematic Investigation Steps 1. Gather Pod State and Events

kubectl get pods -n weaviate -o wide kubectl describe pod weaviate-0 -n weaviate Key sections to inspect:

Q: 2. Stream Container Logs

kubectl logs weaviate-0 -n weaviate --previous Typical log excerpts:

Q: 3. Verify Persistent Volume Mount

kubectl get pvc -n weaviate kubectl describe pvc weaviate-data -n weaviate Confirm that the PVC status is Bound and that the storage class supports ReadWriteMany if multiple replicas need concurrent write access.

Q: 4. Check Resource Allocation

kubectl top pod weaviate-0 -n weaviate kubectl get pod weaviate-0 -n weaviate -o jsonpath="{.spec.containers[0].resources}" Missing resources.limits.memory is a common cause of OOMKilled during peak load (see the community forum thread).

Q: 5. Validate External Service Connectivity

curl -s -o /dev/null -w "%{http_code}" https://auth.mycompany.com/healthz nc -zv auth.mycompany.com 443 A non‑200 response or a connection timeout points to the authentication microservice issue described in the real incident.

Q: 6. Probe Configuration Review

Inspect the Helm chart values for readinessProbe and livenessProbe:

Q: Solution – Fixing the CrashLoop 1. Ensure Required Environment Variables

Missing env vars cause panics. Add them to the Helm values.yaml under env:

Problem – Weaviate Container CrashLoopBackOff in Kubernetes

A production deployment of Weaviate on a multi‑node Kubernetes cluster repeatedly enters CrashLoopBackOff. The pod terminates shortly after start, causing the service to be unavailable for queries. Typical symptoms observed in the cluster:

Pod status: CrashLoopBackOff after a few seconds.

Readiness probe errors such as:

Readiness probe failed: Get http://localhost:8080/v1/.well-known/ready: dial tcp 127.0.0.1:8080: connect: connection refused

Container logs contain panic messages, e.g.:

panic: runtime error: invalid memory address or nil pointer dereference
goroutine 1 [running]:
github.com/weaviate/weaviate/... (…)

Occasional OOMKilled events during peak query bursts.
Startup failures with errors like:
```
Failed to open database file
```

Root Cause – Why the CrashLoop Happens

Weaviate’s startup sequence is tightly coupled to several external dependencies and resource constraints. The most common failure modes that lead to a crash loop are:

Failure Mode	Underlying Reason
Missing or malformed environment variables	Weaviate panics during initialization because required configuration (e.g., `AUTHENTICATION_ENDPOINT`) is nil. See the official configuration reference.
PVC mount issues (ReadWriteOnceMany incompatibility)	When the PVC is bound with a storage class that does not support multi‑node write, the process cannot open the database file, resulting in “Failed to open database file”. This mirrors GitHub issue #2123.
Insufficient memory limits	High query throughput exhausts the container’s memory, triggering the kubelet’s OOM killer. The community forum thread on “High query load leads to container OOMKilled” documents this scenario.
Unreachable external webhook/auth service	Weaviate validates the authentication microservice at startup. If the endpoint is unreachable, the process panics with “unable to reach auth endpoint”. Refer to the real incident where an auth service outage caused a crash loop.
Readiness/liveness probe misconfiguration	Probes that target the wrong port or path cause Kubernetes to restart the pod before the service is ready, as seen in the Stack Overflow question 771945.

In most production incidents, one or more of the above factors are present simultaneously, compounding the restart frequency.

Debug – Systematic Investigation Steps

1. Gather Pod State and Events

kubectl get pods -n weaviate -o wide
kubectl describe pod weaviate-0 -n weaviate

Key sections to inspect:

Events – look for OOMKilled, FailedMount, or Readiness probe failed.
Container status – exit code, termination reason.

2. Stream Container Logs

kubectl logs weaviate-0 -n weaviate --previous

Typical log excerpts:

2024-05-12T08:13:42Z panic: runtime error: invalid memory address or nil pointer dereference
2024-05-12T08:13:42Z stacktrace:
github.com/weaviate/weaviate/... (…)
2024-05-12T08:13:42Z error: failed to open database file

3. Verify Persistent Volume Mount

kubectl get pvc -n weaviate
kubectl describe pvc weaviate-data -n weaviate

Confirm that the PVC status is Bound and that the storage class supports ReadWriteMany if multiple replicas need concurrent write access.

4. Check Resource Allocation

kubectl top pod weaviate-0 -n weaviate
kubectl get pod weaviate-0 -n weaviate -o jsonpath="{.spec.containers[0].resources}"

Missing resources.limits.memory is a common cause of OOMKilled during peak load (see the community forum thread).

5. Validate External Service Connectivity

curl -s -o /dev/null -w "%{http_code}" https://auth.mycompany.com/healthz
nc -zv auth.mycompany.com 443

A non‑200 response or a connection timeout points to the authentication microservice issue described in the real incident.

6. Probe Configuration Review

Inspect the Helm chart values for readinessProbe and livenessProbe:

helm get values weaviate -n weaviate

Solution – Fixing the CrashLoop

1. Ensure Required Environment Variables

Missing env vars cause panics. Add them to the Helm values.yaml under env:

Before:

env:
  - name: QUERY_DEFAULTS_LIMIT
    value: "20"
  # AUTHENTICATION_ENDPOINT omitted

After:

env:
  - name: QUERY_DEFAULTS_LIMIT
    value: "20"
  - name: AUTHENTICATION_ENDPOINT
    value: "https://auth.mycompany.com"
  - name: AUTHENTICATION_HEADER
    value: "Authorization"

2. Align PVC Access Mode with Replica Count

If you run more than one replica, switch to a storage class that supports ReadWriteMany (e.g., nfs-csi) or use a sidecar etcd for shared state. Update the Helm chart:

Before:

persistence:
  enabled: true
  storageClass: standard
  accessModes:
    - ReadWriteOnce

After:

persistence:
  enabled: true
  storageClass: nfs-csi
  accessModes:
    - ReadWriteMany
  size: 100Gi

3. Set Memory Limits and Enable Query Throttling

Following the scaling guide, define limits and configure MAX_CONCURRENT_QUERIES:

resources:
  limits:
    cpu: "2"
    memory: "4Gi"
  requests:
    cpu: "500m"
    memory: "2Gi"
env:
  - name: MAX_CONCURRENT_QUERIES
    value: "100"

This prevents OOMKilled events observed during high load.

4. Adjust Health Probes

Make the readiness probe tolerant of warm‑up time and target the correct endpoint (/v1/.well-known/ready) on port 8080:

readinessProbe:
  httpGet:
    path: /v1/.well-known/ready
    port: 8080
  initialDelaySeconds: 30
  periodSeconds: 10
  failureThreshold: 6
livenessProbe:
  httpGet:
    path: /v1/.well-known/healthz
    port: 8080
  initialDelaySeconds: 60
  periodSeconds: 30

5. Verify External Service Availability

If the authentication webhook is optional, configure a fallback or disable auth during rollout:

env:
  - name: AUTHENTICATION_ENDPOINT
    value: "https://auth.mycompany.com"
  - name: AUTHENTICATION_DISABLED
    value: "false"   # set to "true" to bypass during debugging

6. Redeploy

helm upgrade weaviate weaviate/weaviate \
  -n weaviate \
  -f values.yaml \
  --reuse-values

Monitor the rollout:

kubectl rollout status statefulset/weaviate -n weaviate

Verify – Confirming the Fix

Pod health: kubectl get pods -n weaviate should show Running with READY 1/1.
Readiness probe: No Readiness probe failed events in kubectl describe pod.

Logs: Absence of panic traces or “Failed to open database file”. Example successful startup log:

2024-06-14T12:00:01Z info: Weaviate version 1.22.0 started
2024-06-14T12:00:01Z info: Connected to persistent storage at /var/lib/weaviate
2024-06-14T12:00:01Z info: Auth endpoint reachable, proceeding

Metrics: Observe memory usage via kubectl top pod staying below the defined limit.
Functional test: Run a simple query:
```
curl -s http://weaviate.mycompany.com/v1/objects | jq .
```
The response should contain the expected JSON payload.

Prevent – Operational Guardrails and Best Practices

Continuous health checks: Export Weaviate’s built‑in metrics to Prometheus and set alerts on process_resident_memory_bytes exceeding 80% of the limit.
Pod disruption budgets: Define a PDB to avoid simultaneous restarts of all replicas during upgrades.
Storage class validation: Automate a pre‑flight check that verifies PVC access mode compatibility with the replica count.
Resource budgeting: Align MAX_CONCURRENT_QUERIES with the memory limit; use the scaling guide to compute safe values.
Dependency health monitoring: Add synthetic health probes for external auth/webhook services; trigger a fallback configuration if they become unavailable.
Version pinning: Use exact chart versions and test upgrades in a staging environment before promoting to production.

FAQ – Common Follow‑Up Questions

Why does the readiness probe fail even though the container is running?
The probe targets localhost:8080 before Weaviate finishes its initialization (e.g., schema loading). Increasing initialDelaySeconds or adjusting failureThreshold gives the process time to become ready.
Can I run multiple Weaviate replicas with a standard ReadWriteOnce PVC?
No. Each replica would attempt to write to the same volume, causing “Failed to open database file”. Use a ReadWriteMany storage class or external distributed storage (e.g., NFS, CephFS).
How do I know if OOMKilled is due to query load or a memory leak?
Correlate spike patterns in process_resident_memory_bytes with request rates. A steady increase independent of load suggests a leak; a sharp rise concurrent with high QPS points to insufficient limits.
What should I do if the external auth service is temporarily down?
Enable the AUTHENTICATION_DISABLED=true flag during the outage or configure a short retry back‑off in the auth client. Ensure the flag is not left on in production.
Is there a way to debug the “Failed to open database file” error without restarting the pod?
Exec into the container and inspect the mount point:
```
kubectl exec -it weaviate-0 -n weaviate -- sh
ls -l /var/lib/weaviate
stat /var/lib/weaviate
```
Check permissions, mount options, and confirm the PVC size matches the size field in the Helm values.

Related Topic Hub: Vector Databases Troubleshooting Hub