cable network

LangChain HPA not scaling pods after CPU threshold breach

by

Problem

The LangChain service is deployed as a stateless Deployment behind an API gateway in a Kubernetes cluster. An HorizontalPodAutoscaler (HPA) is configured to scale the Deployment based on CPU utilization (target 70%) and a custom latency metric exported to Prometheus. Under load the request latency spikes and the CPU usage reported by the metrics-server exceeds the target, yet the number of pods remains constant. Clients experience timeouts and the service becomes unresponsive.

Root Cause

Investigation revealed two intertwined issues:

  • Metric source mismatch: The HPA was referencing the resource metric cpu from metrics-server while the custom latency metric was being scraped from Prometheus via the external metric type. The HPA definition mistakenly combined them in a single spec.metrics array without setting type: External for the latency metric, causing the controller to ignore the external metric.
  • Stale metrics API version: The cluster ran Kubernetes v1.21, but the HPA manifest used apiVersion: autoscaling/v2beta2. In this version the external metric requires a metricSelector that references a PrometheusAdapter ServiceMonitor. The adapter was mis‑configured, exposing the latency metric under the name langchain_request_latency_seconds while the HPA expected request_latency_seconds. Consequently the HPA never received a valid external metric value.

Because the HPA could not obtain a valid external metric, it fell back to CPU only. However, the metrics-server was reporting CPU usage per pod based on the cgroup limit (500m) rather than the actual request load, resulting in a reported utilization of ~55% even when the pod was saturated. The combination of ignored latency metric and inaccurate CPU reading prevented any scaling decision.

Debug

1. Verify HPA status and metric values

kubectl get hpa langchain-hpa -o yaml

Sample output showing the missing external metric:

status:
  currentMetrics:
  - type: Resource
    resource:
      name: cpu
      currentAverageUtilization: 55
  conditions:
  - type: AbleToScale
    status: "True"
  - type: ScalingActive
    status: "True"
  - type: ScalingLimited
    status: "False"
  - type: MetricAvailable
    status: "False"
    reason: ExternalMetricNotFound
    message: "unable to get external metric request_latency_seconds: metric not found"

2. Inspect Prometheus Adapter configuration

kubectl -n monitoring get servicemonitor prometheus-adapter -o yaml

Key snippet:

spec:
  endpoints:
  - port: metrics
    path: /metrics
    metricRelabelings:
    - sourceLabels: [__name__]
      regex: ^langchain_request_latency_seconds$
      action: replace
      targetLabel: __name__
      replacement: request_latency_seconds

The regex incorrectly matches the metric name, causing the adapter to expose request_latency_seconds only when the original metric name is langchain_request_latency_seconds. The actual metric exported by the LangChain container is langchain_request_latency_seconds, but the adapter’s relabeling rule drops the langchain_ prefix, resulting in a mismatch with the HPA expectation.

3. Check metrics-server CPU reporting

kubectl top pod -l app=langchain

Typical output:

NAME                CPU(cores)   MEMORY(bytes)
langchain-5d9f9c7b8-abcde   260m         128Mi
langchain-5d9f9c7b8-fghij   270m         130Mi

With a pod limit of 500m, the utilization appears to be ~52%, even though the pod is queuing requests. This is a known limitation of metrics-server when the container runtime reports CPU usage as a cumulative counter without accounting for throttling.

4. Review HPA manifest

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: langchain-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: langchain
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: External
    external:
      metric:
        name: request_latency_seconds
        selector:
          matchLabels:
            service: langchain
      target:
        type: Value
        value: 200ms

Solution

1. Align metric names between Prometheus, adapter, and HPA

Update the Prometheus Adapter ConfigMap to expose the metric under the exact name expected by the HPA.

# Before (ConfigMap snippet)
metricRelabelings:
- sourceLabels: [__name__]
  regex: ^langchain_request_latency_seconds$
  action: replace
  targetLabel: __name__
  replacement: request_latency_seconds

# After
metricRelabelings:
- sourceLabels: [__name__]
  regex: ^langchain_request_latency_seconds$
  action: replace
  targetLabel: __name__
  replacement: request_latency_seconds

Alternatively, change the HPA to request langchain_request_latency_seconds directly, removing the relabeling entirely.

2. Upgrade HPA API version to autoscaling/v2

Kubernetes v1.23+ supports autoscaling/v2 with stable external metric handling. Apply the updated manifest:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: langchain-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: langchain
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: External
    external:
      metric:
        name: request_latency_seconds
        selector:
          matchLabels:
            service: langchain
      target:
        type: Value
        value: 200ms

3. Improve CPU metric accuracy

Replace metrics-server with the kube-state-metrics + custom-metrics-apiserver stack, or enable the --kubelet-insecure-tls flag to allow the resource metric to use the container_cpu_usage_seconds_total counter from cAdvisor, which reflects throttling.

Alternatively, add a container resource request/limit that matches the observed load (e.g., cpu: "1") so that utilization percentages become more representative.

4. Deploy a test load generator to confirm scaling

# Deploy a simple load generator
apiVersion: batch/v1
kind: Job
metadata:
  name: langchain-load
spec:
  template:
    spec:
      containers:
      - name: hey
        image: rakyll/hey
        args: ["-z", "2m", "-c", "100", "http://langchain.mycompany.com/api/v1/query"]
      restartPolicy: Never

Verify

1. Confirm HPA now receives both metrics

kubectl get hpa langchain-hpa -o yaml | grep -A5 "currentMetrics"

Expected output:

currentMetrics:
- type: Resource
  resource:
    name: cpu
    currentAverageUtilization: 85
- type: External
  external:
    metric:
      name: request_latency_seconds
    currentValue: 350ms

2. Observe scaling activity

kubectl get hpa langchain-hpa -w

When the load job runs, you should see the HPA increase desiredReplicas from 2 to 5 (or higher) within a few minutes.

3. Validate latency reduction

kubectl logs -l app=langchain -c langchain -f | grep latency

Post‑scaling logs should show average latency dropping below the 200 ms target.

Prevent

  • Standardize metric naming conventions across services, Prometheus exporters, and HPA definitions.
  • Pin HPA manifests to autoscaling/v2 and test them against the cluster version before deployment.
  • Prefer custom.metrics.k8s.io or external.metrics.k8s.io adapters that expose the exact metric name without relabeling.
  • Set realistic CPU requests/limits that reflect peak load to make utilization metrics meaningful.
  • Enable alerting on HPA_status_condition{type="MetricAvailable",status="False"} to catch missing metrics early.
  • Periodically run a synthetic load test in a staging environment to verify that both CPU and custom metrics trigger scaling.

FAQ

  1. Why does the HPA ignore the external latency metric? The metric name exposed by the Prometheus Adapter did not match the name referenced in the HPA manifest, causing the controller to report ExternalMetricNotFound. Aligning the names resolves the issue.
  2. Can I rely solely on CPU utilization for scaling LangChain? CPU alone is insufficient because LangChain’s request processing time can be dominated by I/O or model inference latency. Combining CPU with a latency metric provides a more accurate scaling signal.
  3. What version of the HPA API should I use? Use autoscaling/v2 (or newer) for stable support of external metrics. The older v2beta2 version has stricter selector requirements and may drop metrics if the adapter configuration is imperfect.
  4. How do I troubleshoot stale metrics from metrics-server? Verify the metrics-server flags --kubelet-insecure-tls and --metric-resolution. If CPU utilization remains inaccurate, switch to a custom metrics pipeline that reads container_cpu_usage_seconds_total directly from cAdvisor.
  5. Is it safe to increase the CPU limit to improve utilization readings? Raising the limit changes the denominator for utilization calculations, but it does not affect actual throttling. It is better to set limits that match expected peak usage and rely on external latency metrics for scaling decisions.

Related Topic Hub: RAG Systems Troubleshooting Hub

Related Articles