Redis HPA Scaling Issues With Prometheus Metrics

Q: Root Cause Analysis Why redis_memory_used_bytes Misleads the HPA

According to the Redis official documentation, the used_memory field reported by the INFO memory command includes:

Q: Contributing Factors Observed in Production FactorImpact on MetricEvidence Memory fragmentation during burst trafficRapid rise in used_memory without proportional increase in command rateIncident: "Burst traffic caused Redis memory fragmentation; the used_memory metric spiked, HPA over‑provisioned pods" Eviction policy freeing memory under sustained loadDrop in used_memory while request latency risesIncident: "Redis eviction policy freed memory during sustained load; memory_used metric dropped while request latency rose" Prometheus scrape interval misconfigurationTemporary spikes become visible to HPAIncident: "Prometheus scrape interval misconfiguration produced a temporary “memory used” spike; HPA reacted with a scale‑up then immediate scale‑down" Exporter version mismatch across replicasInconsistent metric values per podIncident: "Multiple Redis replicas reported inconsistent memory metrics due to exporter version mismatch" Network partition between Prometheus and Redis nodesMissing metric values → HPA falls back to default replica countCommon error: "FailedGetExternalMetric: request failed with status code 500 for metric redis_memory_used_bytes" These factors collectively break the assumption that redis_memory_used_bytes is a monotonic indicator of request load, leading the HPA to make inappropriate scaling decisions. Investigation and Debugging Steps 1. Verify HPA Configuration

kubectl get hpa redis-consumer -o yaml Typical snippet:

Q: 2. Inspect Prometheus Metric Values

curl -G http://prometheus.example.com/api/v1/query \ --data-urlencode 'query=redis_memory_used_bytes{redis_instance="cache"}' Expected output (simplified):

Q: 3. Check Exporter Version Consistency

kubectl exec -it redis-0 -- redis_exporter --version kubectl exec -it redis-1 -- redis_exporter --version All replicas should report the same version (e.g., v1.44.0). Mismatched versions often expose different sets of memory metrics, as discussed in the exporter README.

Q: 4. Review HPA Events and Controller Logs

kubectl describe hpa redis-consumer | grep -i event kubectl logs -n kube-system -l component=controller-manager -c kube-controller-manager | grep redis_memory_used_bytes Typical events:

Q: 5. Correlate Metric with Request Rate

curl -G http://prometheus.example.com/api/v1/query \ --data-urlencode 'query=sum(rate(redis_commands_processed_total{redis_instance="cache"}[1m]))' If the command rate stays flat while redis_memory_used_bytes spikes, the metric is not reflecting load.

Q: Resolution 1. Switch to a More Representative Metric

Use redis_memory_used_rss_bytes (resident set size) or the derived metric redis_memory_fragmentation_ratio to filter out allocator fragmentation. A common pattern from the Kubernetes HPA docs is to target the ratio of used memory to maxmemory:

Q: Before – Problematic HPA

apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: redis-consumer spec: minReplicas: 2 maxReplicas: 10 metrics: - type: External external: metric: name: redis_memory_used_bytes target: type: Utilization averageUtilization: 75 After – Revised HPA Using Utilization Ratio apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: redis-consumer spec: minReplicas: 2 maxReplicas: 10 metrics: - type: External external: metric: name: redis_memory_utilization selector: matchLabels: redis_instance: cache target: type: AverageValue averageValue: 0.75 # 75 % of maxmemory 2. Tune Prometheus Scrape Interval Set the scrape interval to 15s (or lower) to avoid missing short spikes and to give the HPA a smoother data series.

Q: 3. Enable Memory‑Fragmentation Metric Guardrails

Add an alert that fires when redis_memory_fragmentation_ratio exceeds 1.5. In the HPA, use a minReplicas guard to prevent scaling below a safe baseline during transient drops.

Problem Description

In a Kubernetes‑based real‑time data pipeline, Redis is used as an in‑memory cache. The Horizontal Pod Autoscaler (HPA) is configured to scale the downstream consumer pods based on the Prometheus metric redis_memory_used_bytes exposed by redis_exporter. Operators observed the following inconsistent behavior:

During brief traffic spikes the HPA adds 3‑4 extra pods, but the request latency does not improve.
Under sustained load the HPA scales the pods down to a single replica, while Redis memory usage stays high and request latency climbs sharply, eventually causing time‑outs.
Frequent scale‑up/scale‑down “thrashing” events appear in the HPA controller logs.

Typical log excerpts include:


HPA event: "the HPA was unable to compute the replica count: missing metric values for redis_memory_used_bytes"
Controller manager: "error syncing HPA: external metric retrieval timed out after 30s"
redis_exporter: "failed to collect memory stats: connection refused"
Prometheus alert: "RedisMemoryHigh" firing repeatedly with values > 80% of configured maxmemory

These symptoms indicate that the memory metric used by the HPA does not accurately reflect the actual workload demand.

Root Cause Analysis

Why `redis_memory_used_bytes` Misleads the HPA

According to the Redis official documentation, the used_memory field reported by the INFO memory command includes:

Allocated memory fragments caused by the allocator (jemalloc/tcmalloc).
Memory reserved for internal data structures that may not be actively used.
Transient spikes from large batch inserts that are later freed by the eviction policy.

Because the HPA targets a percentage of redis_memory_used_bytes, any short‑term fragmentation or eviction activity can cause the metric to swing dramatically, even when the actual request rate is stable.

Contributing Factors Observed in Production

Factor	Impact on Metric	Evidence
Memory fragmentation during burst traffic	Rapid rise in `used_memory` without proportional increase in command rate	Incident: “Burst traffic caused Redis memory fragmentation; the used_memory metric spiked, HPA over‑provisioned pods”
Eviction policy freeing memory under sustained load	Drop in `used_memory` while request latency rises	Incident: “Redis eviction policy freed memory during sustained load; memory_used metric dropped while request latency rose”
Prometheus scrape interval misconfiguration	Temporary spikes become visible to HPA	Incident: “Prometheus scrape interval misconfiguration produced a temporary “memory used” spike; HPA reacted with a scale‑up then immediate scale‑down”
Exporter version mismatch across replicas	Inconsistent metric values per pod	Incident: “Multiple Redis replicas reported inconsistent memory metrics due to exporter version mismatch”
Network partition between Prometheus and Redis nodes	Missing metric values → HPA falls back to default replica count	Common error: “FailedGetExternalMetric: request failed with status code 500 for metric redis_memory_used_bytes”

These factors collectively break the assumption that redis_memory_used_bytes is a monotonic indicator of request load, leading the HPA to make inappropriate scaling decisions.

Investigation and Debugging Steps

1. Verify HPA Configuration

kubectl get hpa redis-consumer -o yaml

Typical snippet:


spec:
  maxReplicas: 10
  minReplicas: 2
  metrics:
  - type: External
    external:
      metric:
        name: redis_memory_used_bytes
        selector:
          matchLabels:
            redis_instance: cache
      target:
        type: Utilization
        averageUtilization: 75

2. Inspect Prometheus Metric Values

curl -G http://prometheus.example.com/api/v1/query \
  --data-urlencode 'query=redis_memory_used_bytes{redis_instance="cache"}'

Expected output (simplified):


{
  "status":"success",
  "data":{
    "resultType":"vector",
    "result":[
      {"metric":{"instance":"redis-0","job":"redis"},"value":[1625256000,"1.2e+09"]},
      {"metric":{"instance":"redis-1","job":"redis"},"value":[1625256000,"1.1e+09"]}
    ]
  }
}

If the values jump from ~1 GiB to >2 GiB within a single scrape, check the exporter logs for fragmentation warnings.

3. Check Exporter Version Consistency

kubectl exec -it redis-0 -- redis_exporter --version
kubectl exec -it redis-1 -- redis_exporter --version

All replicas should report the same version (e.g., v1.44.0). Mismatched versions often expose different sets of memory metrics, as discussed in the exporter README.

4. Review HPA Events and Controller Logs

kubectl describe hpa redis-consumer | grep -i event
kubectl logs -n kube-system -l component=controller-manager -c kube-controller-manager | grep redis_memory_used_bytes

Typical events:


Events:
  Type    Reason                     Age   From                     Message
  ----    ------                     ----  ----                     -------
  Normal  SuccessfulRescale          5m    horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target
  Warning FailedGetExternalMetric    2m    horizontal-pod-autoscaler  request failed with status code 500 for metric redis_memory_used_bytes

5. Correlate Metric with Request Rate

curl -G http://prometheus.example.com/api/v1/query \
  --data-urlencode 'query=sum(rate(redis_commands_processed_total{redis_instance="cache"}[1m]))'

If the command rate stays flat while redis_memory_used_bytes spikes, the metric is not reflecting load.

Resolution

1. Switch to a More Representative Metric

Use redis_memory_used_rss_bytes (resident set size) or the derived metric redis_memory_fragmentation_ratio to filter out allocator fragmentation. A common pattern from the Kubernetes HPA docs is to target the ratio of used memory to maxmemory:


# prometheus-adapter config snippet (rules.yaml)
rules:
- seriesQuery: 'redis_memory_used_bytes{redis_instance!="",job="redis"}'
  resources:
    overrides:
      redis_instance:
        resource: redis_instance
  metricsQuery: |
    sum by (redis_instance) (
      redis_memory_used_bytes{redis_instance!=""}
    ) / on(redis_instance) group_left max_over_time(redis_memory_max_bytes{redis_instance!=""}[5m])

Update the HPA to reference the new metric name (redis_memory_utilization).

Before – Problematic HPA


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: redis-consumer
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: redis_memory_used_bytes
      target:
        type: Utilization
        averageUtilization: 75

After – Revised HPA Using Utilization Ratio


apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: redis-consumer
spec:
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: External
    external:
      metric:
        name: redis_memory_utilization
        selector:
          matchLabels:
            redis_instance: cache
      target:
        type: AverageValue
        averageValue: 0.75   # 75 % of maxmemory

2. Tune Prometheus Scrape Interval

Set the scrape interval to 15s (or lower) to avoid missing short spikes and to give the HPA a smoother data series.


# prometheus.yml
scrape_configs:
  - job_name: 'redis'
    scrape_interval: 15s
    static_configs:
      - targets: ['redis-0:9121','redis-1:9121']

3. Enable Memory‑Fragmentation Metric Guardrails

Add an alert that fires when redis_memory_fragmentation_ratio exceeds 1.5. In the HPA, use a minReplicas guard to prevent scaling below a safe baseline during transient drops.


# alerts.yml
- alert: RedisFragmentationHigh
  expr: redis_memory_fragmentation_ratio{redis_instance="cache"} > 1.5
  for: 2m
  labels:
    severity: warning
  annotations:
    summary: "Redis memory fragmentation is high"

Verification

Confirm the new metric appears in the custom metrics API:

kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" | jq '.resources[] | select(.name=="redis_memory_utilization")'

Watch the HPA status while generating load:

kubectl get hpa redis-consumer -w

Expected output after fix:


NAME            REFERENCE               TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
redis-consumer  Deployment/redis-consumer 75%/75%   2         10        4          5m

Validate that the memory utilization metric stays within the target range during a sustained load test (e.g., hey -c 200 -n 50000 http://app).
Check that no HPA events about missing metrics appear for at least 30 minutes.

Prevention and Best Practices

Metric selection: Prefer metrics that directly correlate with request rate (e.g., redis_commands_processed_total) or a derived utilization ratio rather than raw used_memory.
Exporter consistency: Deploy the same redis_exporter version across all Redis pods; pin the image tag in the Deployment spec.
Scrape reliability: Use a short (15s) scrape interval and enable honor_labels: true to avoid duplicate series.
Staleness detection: Configure the Prometheus adapter’s queryTimeout and metricRelabelings to drop metrics that haven’t been scraped for more than two intervals.
Alerting: Set alerts for metric staleness, high fragmentation, and sudden drops in redis_memory_used_bytes that are not accompanied by a drop in command rate.
Capacity guardrails: Define a non‑zero minReplicas and a “scale‑down cooldown” (e.g., --horizontal-pod-autoscaler-downscale-delay=5m) to avoid thrashing.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the HPA still scale down when memory usage stays high?
Because the used_memory metric can drop when Redis evicts keys. The HPA interprets the drop as reduced load, even though request latency is increasing. Switching to a utilization ratio or a command‑rate metric resolves this.
Can I combine memory and CPU metrics in a single HPA?
Yes. Define multiple metrics entries in the HPA spec. The controller will compute the replica count that satisfies *all* targets (the highest value wins).
How do I know if the metric is stale due to a network partition?
The Prometheus adapter returns a “missing metric” error (e.g., FailedGetExternalMetric: request failed with status code 500). Configure the adapter’s fallback behavior or add a metricRelabelings rule that drops series older than two scrape intervals.
Is redis_memory_used_rss_bytes always better than redis_memory_used_bytes?
used_rss excludes allocator fragmentation, making it a more stable indicator of actual resident memory. However, if you rely on maxmemory limits, compute the ratio used_rss / maxmemory for a consistent target.
What scrape interval is recommended for HPA‑driven scaling?
A 15‑second interval balances freshness with load on the exporter. Ensure the HPA’s behavior section uses a stabilizationWindowSeconds of at least 60 seconds to smooth out brief spikes.