Jenkins CI job connection timeout to Haystack API server

Problem

During the execution of integration tests on a Jenkins build agent, the CI job fails with a connection timeout when trying to reach the Haystack API server. In some runs the logs also show DNS resolution failures. The failure aborts the test suite and marks the build as unstable.

[2026-10-13 02:14:37] INFO  - Starting integration tests...
[2026-10-13 02:14:38] ERROR - Request to http://haystack-api.internal/v1/search timed out after 10000ms
[2026-10-13 02:14:38] ERROR - java.net.UnknownHostException: haystack-api.internal

Impact:

• Pipeline stalls for ~10 seconds per test before timing out.
• All downstream stages that depend on the API (e.g., data validation, contract tests) are skipped.
• Developers receive false‑negative test results, eroding confidence in the CI pipeline.

Root Cause

The timeout and DNS errors stem from a combination of network isolation and stale DNS cache on the Jenkins agents:

1. Agent network policy – The Kubernetes NetworkPolicy applied to the jenkins-agent namespace only allowed egress to ports 443 and 80 on the default namespace. The Haystack API runs in the haystack namespace on port 8080, which was not whitelisted.
2. Service name mismatch – The integration tests reference haystack-api.internal, but the actual Service DNS entry is haystack-api.haystack.svc.cluster.local. The short name resolves only inside the haystack namespace, causing UnknownHostException when the agent runs in a different namespace.
3. DNS cache poisoning – The systemd-resolved cache on the agent node retained an old negative entry for haystack-api.internal. Subsequent test runs hit the cached failure until the cache TTL expired.

Debug

Step‑by‑step investigation performed on a failing build agent:

1. Verify DNS resolution from the agent container

$ kubectl exec -ti jenkins-agent-7f9c9d9c9b-ktz9q -- nslookup haystack-api.internal
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find haystack-api.internal: NXDOMAIN

2. Check service FQDN

$ kubectl get svc -n haystack haystack-api -o jsonpath='{.spec.clusterIP}'
10.103.45.22
$ kubectl exec -ti jenkins-agent-7f9c9d9c9b-ktz9q -- nslookup haystack-api.haystack.svc.cluster.local
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   haystack-api.haystack.svc.cluster.local
Address: 10.103.45.22

3. Inspect NetworkPolicy

$ kubectl get netpol -n jenkins-agent -o yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: jenkins-agent-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: default
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

4. Examine DNS cache on the node

$ sudo systemd-resolve --statistics
Cache: 1024 entries (hits: 3123, misses: 874, evictions: 57)

5. Review Jenkins job configuration

pipeline {
    agent { label 'linux' }
    stages {
        stage('Integration Tests') {
            steps {
                sh '''
                export HAYSTACK_API_URL=http://haystack-api.internal/v1
                ./run_integration_tests.sh
                '''
            }
        }
    }
}

Solution

Three corrective actions were applied:

1. Update NetworkPolicy to allow egress to Haystack namespace on port 8080

Before (restricted egress):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: jenkins-agent-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: default
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443

After (added Haystack egress):

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: jenkins-agent-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: default
    ports:
    - protocol: TCP
      port: 80
    - protocol: TCP
      port: 443
  - to:
    - namespaceSelector:
        matchLabels:
          name: haystack
    ports:
    - protocol: TCP
      port: 8080

2. Use the fully‑qualified Service DNS name in the test configuration

Modified run_integration_tests.sh to reference the FQDN:

- export HAYSTACK_API_URL=http://haystack-api.internal/v1
+ export HAYSTACK_API_URL=http://haystack-api.haystack.svc.cluster.local:8080/v1

3. Flush stale DNS cache on agent nodes

# On each Jenkins agent node
sudo systemd-resolve --flush-caches
# Verify cache is empty
sudo systemd-resolve --statistics

Optionally, add a postStart hook to the agent pod spec to clear the cache on every start:

apiVersion: v1
kind: Pod
metadata:
  name: jenkins-agent
spec:
  containers:
  - name: agent
    image: jenkins/inbound-agent:latest
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "systemd-resolve --flush-caches"]

Verify

Run the pipeline again and confirm successful connectivity:

[2026-10-13 02:14:37] INFO  - Starting integration tests...
[2026-10-13 02:14:38] INFO  - Connecting to http://haystack-api.haystack.svc.cluster.local:8080/v1/search
[2026-10-13 02:14:38] INFO  - Received 200 OK (duration: 124ms)
[2026-10-13 02:14:45] INFO  - All integration tests passed

Additional validation steps:

• Check egress flow logs (e.g., Calico or Cilium) for allowed traffic to 10.103.45.22:8080.
• Run a one‑off curl from the agent container:

$ kubectl exec -ti jenkins-agent-7f9c9d9c9b-ktz9q -- curl -s -o /dev/null -w "%{http_code}" http://haystack-api.haystack.svc.cluster.local:8080/healthz
200

Prevent

Implement guardrails to avoid recurrence:

• NetworkPolicy review: Include a CI/CD egress rule template that lists required service namespaces and ports. Enforce via CI linting.
• Configuration lint: Add a static analysis step that checks environment variables for fully‑qualified DNS names (e.g., regex .*\.svc\.cluster\.local).
• DNS health check: Deploy a DaemonSet that periodically resolves critical service names and reports failures to Prometheus.
• Agent image hardening: Base the agent image on a minimal OS without persistent DNS cache, or configure systemd-resolved to use a very short negative TTL (e.g., 30 s).
• Alerting: Create an alert on kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff",container="agent"} combined with increase(http_request_duration_seconds_count{code="5xx"}[5m]) > 0 to surface connectivity issues early.

Related Topic Hub: RAG Systems Troubleshooting Hub

FAQ

1. Why does the DNS name work locally but not in the Jenkins job?
   Local development pods run in the same haystack namespace, so the short name resolves via the cluster DNS suffix. Jenkins agents run in a separate namespace where the short name is not searchable, requiring the full service FQDN.
2. Can I keep using the short name by adding a DNS search suffix?
   Yes. Adding search haystack.svc.cluster.local to /etc/resolv.conf of the agent pods would make the short name resolve, but using the FQDN is more explicit and avoids accidental cross‑namespace collisions.
3. Do I need to modify the Haystack API Service definition?
   No. The Service was correctly defined. The issue was only the client side DNS name and egress policy.
4. What if the API moves to a different port?
   Update the NetworkPolicy egress rule to include the new port and adjust the environment variable accordingly. Consider parameterizing the port in a ConfigMap shared between CI and production.
5. How can I detect similar DNS‑related failures in other pipelines?
   Enable a global dns_error_total Prometheus counter in the agent image that increments on UnknownHostException. Alert when the counter spikes for any pipeline.