Intermittent OAuth2 Token Acquisition Failure In Hybrid Elasticsearch After Provider Timeout

Q: Configuration Review

Check the current token provider settings (elasticsearch.yml excerpt):

In this article

Table of Contents

Problem

In a hybrid Elasticsearch deployment (on‑premises nodes + public‑cloud nodes) the indexing and search APIs intermittently return 401 Unauthorized errors. The failure trace points to the OAuth2 token acquisition step performed by the Elasticsearch security module. The issue appears as a burst of token‑acquisition failures lasting from a few seconds up to a minute, after which normal operation resumes.

Symptoms and Impact

Log entries similar to:

2024-10-07T12:34:56.789Z [WARN ][o.e.x.s.t.c.c.TokenProvider] [es-node-01] Failed to acquire OAuth2 token: request timed out (deadline exceeded)

HTTP responses:

HTTP/1.1 401 Unauthorized
WWW-Authenticate: Bearer realm="Elasticsearch", error="invalid_token", error_description="Token acquisition failed"

Indexing latency spikes (10‑30 s) and occasional bulk request failures.
Metrics: es.security.oauth2.token_acquisition_failures spikes in Prometheus.

Root Cause

The Elasticsearch security plugin contacts a centralized OAuth2 provider (e.g., Keycloak) via the /protocol/openid-connect/token endpoint using a client‑credentials grant. The provider is deployed behind a load balancer that enforces a 5 s idle timeout on idle TCP connections. When the Elasticsearch node reuses an existing HTTP connection that has been silently closed by the load balancer, the request hangs until the client‑side read timeout (30 s) expires, producing the observed “request timed out” errors.

Because the failure is tied to connection reuse, it manifests intermittently: only the requests that hit a stale connection fail, while fresh connections succeed. The hybrid topology aggravates the problem because on‑prem nodes traverse a VPN with higher latency, increasing the probability that the load balancer’s idle timeout is reached before the next token request.

Debug

Log Analysis

Search for token‑provider warnings:

grep "TokenProvider" /var/log/elasticsearch/elasticsearch.log | grep -i "failed"

Typical pattern:

2024-10-07T12:34:56.789Z [WARN ][o.e.x.s.t.c.c.TokenProvider] [es-node-01] Failed to acquire OAuth2 token: request timed out (deadline exceeded)

Network Capture

Capture traffic between the Elasticsearch node and the OAuth2 provider:

tcpdump -i eth0 host 10.20.30.40 and port 443 -w token_issue.pcap

Inspect the capture for TCP RST or FIN packets after a period of inactivity. A typical symptom is a silent drop without a FIN/RST, indicating the load balancer closed the idle connection.

HTTP Client Diagnostics

Elasticsearch uses the Apache HttpAsyncClient. Enable wire logging:

logger.org.elasticsearch.xpack.security.authc.oauth2.client: TRACE

Sample debug output showing a reused connection that never receives a response:

2024-10-07T12:34:56.790Z [TRACE][o.e.x.s.t.c.c.TokenProvider] [es-node-01] Executing POST https://auth.example.com/protocol/openid-connect/token
2024-10-07T12:34:56.791Z [TRACE][org.apache.http.impl.nio.client] [es-node-01] Connection request [id: 0][route: {target: https://auth.example.com:443}][state: REUSABLE]
... (30 s later)
2024-10-07T12:35:26.792Z [WARN ][o.e.x.s.t.c.c.TokenProvider] [es-node-01] Failed to acquire OAuth2 token: request timed out (deadline exceeded)

Configuration Review

Check the current token provider settings (elasticsearch.yml excerpt):

xpack.security.authc.oauth2:
  client_id: "es-service"
  client_secret: "********"
  token_endpoint: "https://auth.example.com/protocol/openid-connect/token"
  connection_timeout: 30s
  read_timeout: 30s
  max_connections: 10
  keep_alive: true

The keep_alive flag enables HTTP connection reuse, which is the root of the problem.

Solution

Disable Connection Reuse for Token Requests

Modify the token provider configuration to force a new HTTP connection for each token acquisition. This eliminates reliance on potentially stale keep‑alive sockets.

Before:

xpack.security.authc.oauth2:
  keep_alive: true
  connection_timeout: 30s
  read_timeout: 30s

After:

xpack.security.authc.oauth2:
  keep_alive: false          # disable HTTP keep‑alive
  connection_timeout: 10s   # lower timeout to fail fast
  read_timeout: 10s
  max_connections: 5        # reduce pool size, not critical for client‑credentials flow

Restart the Elasticsearch nodes for the new settings to take effect:

systemctl restart elasticsearch

Optional: Adjust Load Balancer Idle Timeout

If disabling keep‑alive is not desirable (e.g., for performance reasons), increase the idle timeout on the load balancer to exceed the longest expected token‑acquisition interval (e.g., 60 s). This change must be coordinated with the network team.

Optional: Enable Token Caching

Configure a longer token TTL and enable client‑side caching to reduce the frequency of token requests:

xpack.security.authc.oauth2:
  token_cache_ttl: 5m   # cache token for 5 minutes

This reduces the window where a stale connection could be used.

Verify

Functional Test

Run a simple search request that triggers token acquisition:

curl -k -u elastic:changeme -XGET "https://es-node-01:9200/_search?pretty"

Expected output: HTTP 200 with search results, no 401 warnings in the logs.

Log Confirmation

After the change, the warning should disappear:

grep "Failed to acquire OAuth2 token" /var/log/elasticsearch/elasticsearch.log

Result: no matches.

Metrics Check

Prometheus query to ensure the failure counter stays at zero:

es_security_oauth2_token_acquisition_failures{job="elasticsearch"} == 0

Load Test

Simulate a burst of indexing operations (e.g., using esbulk) for 5 minutes and verify that the failure rate remains zero.

Prevent

Connection hygiene: For any external authentication provider, disable HTTP keep‑alive unless the provider guarantees idle‑connection persistence.
Idle timeout alignment: Ensure load balancer or reverse‑proxy idle timeouts are greater than the client’s keep‑alive timeout, or configure the client to match the shorter value.
Monitoring: Alert on es.security.oauth2.token_acquisition_failures > 0 for a 5‑minute window.
Token caching: Use a reasonable token_cache_ttl to reduce request frequency.
Health check endpoint: Deploy a lightweight endpoint that periodically forces a token refresh; monitor its success to catch regressions early.

Related Topic Hub: Data Infrastructure Troubleshooting Hub

FAQ

Why does the issue appear only during peak traffic? Peak traffic increases the number of concurrent token requests, raising the probability that a stale keep‑alive connection is selected from the pool.
Can I keep keep‑alive enabled and still avoid the timeout? Only if the load balancer’s idle timeout is configured to exceed the client’s keep‑alive timeout, or if you implement a TCP keep‑alive probe on the Elasticsearch side.
Does disabling keep‑alive impact performance? The token request latency is typically <200 ms; the overhead of establishing a new TLS connection is negligible compared to the cost of failed requests and retries.
What if the OAuth2 provider itself closes idle connections? Enable Connection: close header in the token request by setting keep_alive: false, which forces the provider to close the socket after each response.
Is there a way to see which connection was reused? Enable Apache HttpAsyncClient wire logs (org.apache.http.impl.nio.client) at TRACE level; the log will show “Connection request … [state: REUSABLE]”.