PostgreSQL webhook timeout during rolling update in Kubernetes

Problem: Webhook timeout during rolling update of PostgreSQL pods in Kubernetes

During a CI/CD‑driven rolling update of a PostgreSQL StatefulSet (or a Patroni‑managed cluster), the admission webhook that validates pod creation repeatedly fails with a timeout. Typical error messages observed in the controller logs are:

Failed calling webhook "postgresql-admission-webhook": context deadline exceeded
Error from server (InternalError): admission webhook "pg-webhook.k8s.io" denied the request: webhook timeout
Timed out waiting for the condition while waiting for PostgreSQL pod to become ready during rolling update

The failure aborts the rollout, leaves the StatefulSet partially updated, and can cause replication lag or temporary loss of write capability.

Root Cause Analysis

The timeout originates from the interaction of three Kubernetes mechanisms:

1. Admission webhook request timeout – By default, Kubernetes aborts a webhook call after 30s (see Kubernetes Documentation – Admission Controllers > Webhook Configuration). The failurePolicy setting determines whether a timeout is treated as a failure.
2. PostgreSQL recovery / replication catch‑up time – When a pod is terminated, the new pod must start PostgreSQL, recover WAL, and re‑establish streaming replication. In production clusters, especially with large WAL volumes or slow disks, this can exceed 30 seconds (see PostgreSQL Documentation – Chapter 27: Streaming Replication and Logical Replication).
3. Pod termination grace period and readiness probes – Rolling updates respect terminationGracePeriodSeconds and readinessProbe settings. If the readiness probe waits for replication to be in sync, the webhook sees the pod as not ready while it is still performing recovery (see Kubernetes Documentation – Rolling Updates for Deployments).

When the webhook’s timeoutSeconds (default 30 s) expires before the new PostgreSQL pod reports Ready, the webhook returns context deadline exceeded. If failurePolicy: Fail is configured (as in the CrunchyData incident), the rollout is aborted.

Investigation and Debugging

Follow these steps to isolate the failure path:

1. Inspect webhook configuration

kubectl get mutatingwebhookconfiguration pg-webhook -o yaml

Key fields:

2. Check pod termination and startup logs

kubectl logs -f postgres-0 -n db
# Look for lines such as:
2026-06-02 14:12:03.123 UTC [1] LOG:  database system was shut down at 2026-06-02 14:11:58 UTC
2026-06-02 14:12:05.456 UTC [1] LOG:  starting up replication slot "_some_slot"
2026-06-02 14:12:45.789 UTC [1] LOG:  database system is ready to accept connections

Notice the ~40 s gap between shutdown and readiness – longer than the webhook timeout.

3. Capture the webhook request/response latency

kubectl get events -n db --field-selector reason=FailedCallingWebhook
# Example output:
LAST SEEN   TYPE      REASON                OBJECT               MESSAGE
2m          Warning   FailedCallingWebhook  pod/postgres-0       context deadline exceeded

4. Validate replication slot existence

kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT * FROM pg_replication_slots;"

If the slot is missing, the readiness probe will wait for its recreation, extending startup time (see the “Replication slot does not exist” error in community sources).

Resolution

Two complementary categories of fixes are required: extend the webhook timeout and ensure PostgreSQL can become ready within that window.

1. Increase webhook timeout and relax failure policy

Modify the MutatingWebhookConfiguration (or ValidatingWebhookConfiguration) to allow a longer timeout and change failurePolicy to Ignore for non‑critical failures.

# Before
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: pg-webhook
webhooks:
- name: pg-webhook.k8s.io
  clientConfig:
    service:
      name: pg-admission
      namespace: db
      path: /mutate
    caBundle: ...
  timeoutSeconds: 30
  failurePolicy: Fail

# After
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: pg-webhook
webhooks:
- name: pg-webhook.k8s.io
  clientConfig:
    service:
      name: pg-admission
      namespace: db
      path: /mutate
    caBundle: ...
  timeoutSeconds: 60   # extended to cover worst‑case recovery
  failurePolicy: Ignore

2. Adjust PostgreSQL pod lifecycle parameters

• Increase terminationGracePeriodSeconds to give the old pod enough time to finish WAL streaming. Example:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  template:
    spec:
      terminationGracePeriodSeconds: 120   # previously 30

readinessProbe:
  exec:
    command: ["pg_isready", "-U", "postgres"]
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3

# Patroni config snippet
postgresql:
  parameters:
    max_replication_slots: 10
  replication:
    slots:
      - name: _some_slot
        type: physical

kubectl rollout restart statefulset/postgres -n db
kubectl get pods -w -n db
# Expected output snippet:
postgres-0   1/1     Running   0          2m
postgres-1   1/1     Running   0          2m

kubectl exec -it postgres-0 -- psql -U postgres -c "SELECT client_addr, state FROM pg_stat_replication;"
# Expected rows show state = 'streaming' for all replicas.