PostgreSQL HPA Not Scaling Under High CPU Usage

Q: Investigation and Debugging Steps 1. Verify HPA Configuration

kubectl get hpa postgres-hpa -o yaml Check that targetCPUUtilizationPercentage is set and that minReplicas/maxReplicas are reasonable.

Q: 2. Inspect Pod Resource Requests

kubectl get deployment postgres -o yaml | grep -A5 resources Expected output (incorrect example):

Q: 3. Confirm Metrics‑Server Health

kubectl get deployment metrics-server -n kube-system kubectl logs -n kube-system deployment/metrics-server Look for errors about TLS or kubelet connectivity. If the flag --kubelet-insecure-tls is absent, add it.

Q: 4. Directly Query Metrics

kubectl top pod -l app=postgres If the command returns “CPU(cores) MEMORY(bytes) …” with “<none>” for CPU, the metrics pipeline is broken.

Q: 5. Check PostgreSQL Runtime Configuration

kubectl exec -it $(kubectl get pod -l app=postgres -o jsonpath="{.items[0].metadata.name}") -- psql -U postgres -c "SHOW shared_buffers; SHOW work_mem; SHOW max_connections;" Values from PostgreSQL 15 documentation (Chapter 19) should be tuned to the container limits.

In this article

Table of Contents

Problem – PostgreSQL HPA Not Scaling Under High CPU Usage

The PostgreSQL deployment runs in a Kubernetes cluster that is part of an AI‑driven orchestration platform. During traffic spikes the API gateway generates thousands of concurrent requests per second, causing PostgreSQL CPU utilization to rise sharply. Expected behavior is that the Horizontal Pod Autoscaler (HPA) increases the replica count, but the replica count remains stuck at the minimum value, leading to:

Slow query response times (latency > 2 s)
Connection errors once max_connections is exhausted
Increased request timeouts observed by downstream services

Typical error messages seen in kubectl describe hpa postgres-hpa and pod logs include:


Events:
  Type    Reason                         Age   From                       Message
  ----    ------                         ----  ----                       -------
  Normal  SuccessfulRescale              5m    horizontal-pod-autoscaler  New size: 2; reason: cpu utilization above target
  Warning FailedGetResourceMetric         2m    horizontal-pod-autoscaler  failed to get cpu utilization: metric not found
  Warning FailedComputeReplicas          1m    horizontal-pod-autoscaler  Unable to compute desired replica count: missing metrics for resource cpu

PostgreSQL logs also show resource pressure:


2026-06-07 12:34:56.789 UTC [12345] LOG:  could not allocate shared memory: No space left on device
2026-06-07 12:34:56.790 UTC [12345] FATAL:  out of memory

Root Cause Analysis

Missing or Incomplete Resource Requests

The metrics server only emits CPU utilization when pods have explicit resources.requests.cpu defined. In the incident where “HPA failed to increase replicas because the PostgreSQL containers lacked explicit CPU requests/limits,” the HPA controller logged Unable to compute desired replica count: missing metrics for resource cpu. Without a request value, the metrics-server reports <unknown>, so the HPA never sees a utilization percentage above the target.

Metrics‑Server Connectivity Issues

Another common failure mode is a mis‑configured metrics‑server. When deployed without the --kubelet-insecure-tls flag on clusters with TLS‑enabled kubelets, the server cannot scrape node metrics, resulting in “failed to get cpu utilization: metric not found.” This matches the real‑world incident documented in the evidence package.

CPU Throttling vs. I/O‑Bound Load

PostgreSQL queries that are I/O‑bound may keep CPU usage below the HPA threshold while the database is saturated by max_connections. The incident where “PostgreSQL pods hit the ‘max_connections’ limit under load; HPA did not scale because the pod’s CPU usage remained below the threshold” demonstrates a mismatch between the metric type (CPU) and the actual bottleneck (connection pool / I/O).

cgroup v2 Exposure Problems

On nodes using cgroup v2, the kubelet may fail to expose CPU usage for containers, leading to “no metrics found” errors. This was observed in the production outage where “cgroup v2 on the Kubernetes nodes prevented the kubelet from exposing CPU usage for PostgreSQL pods.”

Investigation and Debugging Steps

1. Verify HPA Configuration


kubectl get hpa postgres-hpa -o yaml

Check that targetCPUUtilizationPercentage is set and that minReplicas/maxReplicas are reasonable.

2. Inspect Pod Resource Requests


kubectl get deployment postgres -o yaml | grep -A5 resources

Expected output (incorrect example):


resources:
  limits:
    cpu: "2"
    memory: "4Gi"

Missing requests leads to missing metrics.

3. Confirm Metrics‑Server Health


kubectl get deployment metrics-server -n kube-system
kubectl logs -n kube-system deployment/metrics-server

Look for errors about TLS or kubelet connectivity. If the flag --kubelet-insecure-tls is absent, add it.

4. Directly Query Metrics


kubectl top pod -l app=postgres

If the command returns “CPU(cores) MEMORY(bytes) …” with “<none>” for CPU, the metrics pipeline is broken.

5. Check PostgreSQL Runtime Configuration


kubectl exec -it $(kubectl get pod -l app=postgres -o jsonpath="{.items[0].metadata.name}") -- psql -U postgres -c "SHOW shared_buffers; SHOW work_mem; SHOW max_connections;"

Values from PostgreSQL 15 documentation (Chapter 19) should be tuned to the container limits.

6. Examine Node‑Level Cgroup Reporting


ssh node01
cat /sys/fs/cgroup/cpu.stat

If cgroup v2 is active, verify that cpu.stat reports usage; otherwise the kubelet cannot surface metrics.

Resolution – Making HPA Scale PostgreSQL Correctly

Step 1 – Define Explicit CPU Requests

Update the Deployment manifest to include matching requests and limits. Align the request with the expected baseline load.

Before:


resources:
  limits:
    cpu: "2"
    memory: "4Gi"

After:


resources:
  requests:
    cpu: "500m"
    memory: "2Gi"
  limits:
    cpu: "2"
    memory: "4Gi"

Step 2 – Adjust HPA Target and Add Custom Metrics (Optional)

If CPU never exceeds the threshold because the workload is I/O‑bound, add a custom metric based on pg_stat_activity or connection count.


apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: postgres-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: postgres
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: pg_connections
      target:
        type: AverageValue
        averageValue: 200

Step 3 – Fix Metrics‑Server TLS Flag

Patch the metrics‑server deployment:


kubectl edit deployment metrics-server -n kube-system

Add the flag to the container args:


- --kubelet-insecure-tls

Step 4 – Verify cgroup Compatibility

If nodes run cgroup v2, upgrade the kubelet to a version that supports the --cgroup-driver=systemd flag, or enable the --cgroup-root=/sys/fs/cgroup option. Restart the kubelet after the change.

Step 5 – Tune PostgreSQL Runtime Settings

Set shared_buffers and work_mem proportionally to the container memory limit, and increase max_connections if the connection pool can be safely expanded.


# postgresql.conf snippet
shared_buffers = 1GB          # ~25% of 4Gi limit
work_mem = 16MB
max_connections = 500

Validation – Confirming That Scaling Works

Generate load that drives CPU above the target, e.g., using pgbench:


kubectl exec -it $(kubectl get pod -l app=postgres -o jsonpath="{.items[0].metadata.name}") -- \
  pgbench -c 20 -j 4 -T 300 -U postgres

Watch HPA status:


kubectl describe hpa postgres-hpa --watch

Expected output shows a transition from DesiredReplicas: 2 to a higher number (e.g., 5) once CPU crosses 70%.

Validate new pods become Ready and PostgreSQL logs no longer contain “could not allocate shared memory”.


kubectl get pods -l app=postgres
kubectl logs $(kubectl get pod -l app=postgres -o jsonpath="{.items[-1].metadata.name}") | grep "shared memory"

Absence of the error confirms successful scaling.

Prevention – Operational Guardrails

Enforce resource request policies: Use an OPA gatekeeper rule or admission controller to reject Deployments without CPU requests.
Monitor HPA health: Create alerts on HPA events like “FailedComputeReplicas” or “missing metrics for resource cpu”.
Collect custom PostgreSQL metrics: Export pg_stat_activity and connection counts via Prometheus Exporter; set alerts when max_connections utilization exceeds 80%.
Regularly test metrics‑server TLS connectivity: Include a CI job that runs kubectl top pods against a test namespace and fails on missing metrics.
Validate cgroup version compatibility: Pin node images to a known cgroup configuration and verify kubelet flags during node provisioning.

FAQ – Common Follow‑Up Questions

Why does the HPA scale when I add CPU requests but not when I only increase the limit?
The metrics server calculates utilization as current usage / request. Without a request value the utilization is undefined, so the HPA cannot make scaling decisions.
Can I rely on memory utilization instead of CPU for PostgreSQL scaling?
Memory can be a useful signal, but PostgreSQL often becomes CPU‑bound under heavy query load. Using both metrics together reduces the chance of missing a bottleneck.
How do I expose custom PostgreSQL metrics to the HPA?
Deploy the postgres-exporter (Prometheus exporter) and configure an ExternalMetric or Pods metric in an HPA v2beta2 manifest, as shown in the solution section.
What is the impact of cgroup v2 on HPA scaling?
On nodes with cgroup v2, older kubelet versions may not expose CPU usage to the metrics server, leading to “no metrics found.” Upgrade kubelet or enable the appropriate flags.
My HPA still doesn’t scale after fixing requests; what else should I check?
Verify that the metrics‑server can reach the kubelet (TLS flags), ensure the API server’s --requestheader-client-ca-file is correctly configured, and confirm that the HPA’s maxReplicas is not being limited by a PodDisruptionBudget or ResourceQuota.

Related Topic Hub: Data Infrastructure Troubleshooting Hub