Lambda event processing fails to resolve Pinecone index endpoint after VPC update

Lambda event processing fails to resolve Pinecone index endpoint after VPC update

Problem

An event‑driven Lambda function (or containerized microservice) receives a payload, extracts a vector, and issues a similarity query to a Pinecone index. After a recent VPC configuration change the function throws a ConnectionError / ENOTFOUND and cannot resolve the index endpoint. All downstream processing stalls, causing a backlog of events.

Typical log excerpt:

2026-10-08T12:34:56.123Z	ERROR	Traceback (most recent call last):
2026-10-08T12:34:56.124Z	  File "/var/task/handler.py", line 42, in lambda_handler
2026-10-08T12:34:56.124Z	    results = pinecone_client.query(
2026-10-08T12:34:56.124Z	  File "/opt/python/pinecone/client.py", line 215, in query
2026-10-08T12:34:56.124Z	    response = self._session.get(url, timeout=self._timeout)
2026-10-08T12:34:56.124Z	  File "/opt/python/requests/sessions.py", line 540, in get
2026-10-08T12:34:56.124Z	    resp = self.request('GET', url, **kwargs)
2026-10-08T12:34:56.124Z	  File "/opt/python/requests/sessions.py", line 530, in request
2026-10-08T12:34:56.124Z	    raise ConnectionError(e, request=request)
2026-10-08T12:34:56.124Z	ConnectionError: HTTPSConnectionPool(host='my-index-abc123.svc.us-east-1.pinecone.io', port=443): Max retries exceeded with url: /query (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -3] Name or service not known'))

Impact:

• Event processing latency spikes > 30 seconds.
• Lambda throttles due to repeated retries, incurring extra cost.
• Downstream services that depend on similarity results experience timeouts.

Root Cause

The VPC update introduced a private DNS resolver for the *.svc.us-east-1.pinecone.io domain. When the Lambda function is attached to the updated VPC, DNS queries are sent to the VPC resolver, which does not have forwarders for Pinecone’s public DNS records. Consequently the resolver returns NXDOMAIN, and the client cannot resolve the index endpoint.

Key points:

1. Lambda in VPC – networking is forced through ENIs; DNS resolution follows the VPC resolver settings.
2. Private DNS enabled – the VPC’s enableDnsHostnames and enableDnsSupport are true, but the dnsResolver is set to the default AmazonProvidedDNS only. No forwarder for external zones.
3. Pinecone endpoint is public – it resolves via public Route 53; the private resolver never reaches the public internet.

Debug & Investigation

Step‑by‑step verification that isolates the DNS problem.

1. Confirm VPC DNS settings:

aws ec2 describe-vpcs --vpc-ids vpc-0a1b2c3d4e5f6g7h \
  --query 'Vpcs[0].{DnsSupport:DnsSupport, DnsHostnames:DnsHostnames}'

Expected output:

{
    "DnsSupport": "enable",
    "DnsHostnames": "enable"
}

Test DNS resolution from inside the VPC (using a temporary EC2 instance or Lambda aws-cli container):

dig +short my-index-abc123.svc.us-east-1.pinecone.io

Observed output after VPC change:

;; connection timed out; no servers could be reached

Check the resolver configuration on the instance:

cat /etc/resolv.conf

Typical VPC resolver entry:

nameserver 169.254.169.253
options ndots:2

Validate that the public DNS works when bypassing the VPC resolver:

nslookup my-index-abc123.svc.us-east-1.pinecone.io 8.8.8.8

Result:

Name:   my-index-abc123.svc.us-east-1.pinecone.io
Address: 34.215.12.34

Inspect Lambda ENI network interfaces for missing outbound internet access:

aws ec2 describe-network-interfaces --filters Name=attachment.instance-id,Values=$(aws lambda get-function-configuration --function-name MyHandler --query 'VpcConfig.SubnetIds[0]' --output text)

Review recent VPC change history (CloudTrail):

aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ModifyVpcAttribute \
  --start-time 2026-09-30T00:00:00Z --end-time 2026-10-08T00:00:00Z

Solution

Two viable approaches:

Option 1 – Add a DNS forwarder to the VPC for Pinecone’s domain

Configure a Route 53 Resolver inbound endpoint that forwards svc.us-east-1.pinecone.io queries to the public DNS (e.g., 8.8.8.8). Then associate a resolver rule with the VPC.

# Create resolver rule
aws route53resolver create-resolver-rule \
  --creator-request-id $(date +%s) \
  --rule-type FORWARD \
  --domain-name svc.us-east-1.pinecone.io \
  --target-ips Ip=8.8.8.8,Port=53 Ip=8.8.4.4,Port=53 \
  --name PineconeForwardRule

# Associate rule with VPC
aws route53resolver associate-resolver-rule \
  --resolver-rule-id rslvr-rr-0a1b2c3d4e5f6g7h \
  --vpc-id vpc-0a1b2c3d4e5f6g7h

After association, the same dig command returns the correct IP.

Option 2 – Move the Lambda out of the VPC or add a NAT gateway

If the function does not need private subnet access, detach it from the VPC. Otherwise, attach a NAT gateway to the private subnet so that outbound DNS queries can reach the internet.

# Detach VPC config (example using AWS CLI)
aws lambda update-function-configuration \
  --function-name MyHandler \
  --vpc-config SubnetIds=[],SecurityGroupIds=[]

Or, add a NAT gateway and update the route table:

# Create NAT gateway
aws ec2 create-nat-gateway --subnet-id subnet-0a1b2c3d4e5f6g7h --allocation-id eipalloc-0123456789abcdef

# Update route table
aws ec2 create-route --route-table-id rtb-0a1b2c3d4e5f6g7h \
  --destination-cidr-block 0.0.0.0/0 --nat-gateway-id nat-0a1b2c3d4e5f6g7h

Why the fix works

• Forwarder rule ensures the VPC resolver can resolve Pinecone’s public domain without exposing the entire VPC to the internet.
• NAT gateway provides a path for DNS (UDP/TCP 53) and HTTPS traffic to reach external services while keeping the Lambda in a private subnet.

Verification

After applying the chosen fix, repeat the DNS test and invoke the Lambda.

dig +short my-index-abc123.svc.us-east-1.pinecone.io
# Expected: 34.215.12.34

Invoke the function (or replay a single event) and confirm successful query:

aws lambda invoke --function-name MyHandler --payload '{"id":"test"}' response.json
cat response.json
# Expected JSON contains "matches": [...]

Metrics to watch:

• Lambda duration drops back to baseline (< 200 ms for query).
• CloudWatch logs no longer contain ConnectionError or ENOTFOUND.
• Pinecone request count increases, confirming traffic flow.

Prevention & Best Practices

• Document DNS dependencies for all external SaaS endpoints used by VPC‑bound workloads.
• Automate resolver rule checks in CI/CD pipelines (e.g., Terraform validation that aws_route53resolver_rule exists for each external domain).
• Prefer NAT gateways over public subnets for workloads that need outbound internet but must stay private.
• Enable health‑check alarms on Lambda error count and on DNS resolution failures (CloudWatch metric filter on ENOTFOUND).
• Version‑control VPC changes and require peer review for modifications to enableDnsSupport or resolver configurations.

FAQ

1. Why does the same Lambda work in the dev VPC but fail after the prod VPC update?
   Because the dev VPC still uses the default AmazonProvidedDNS which forwards unknown domains to the internet, while the prod VPC was changed to a private resolver without forwarders for Pinecone’s domain.
2. Can I use Route 53 Private Hosted Zones for Pinecone?
   No. Pinecone’s endpoints are public and managed by AWS Route 53 public zones. Creating a private hosted zone with the same name would shadow the public records and cause resolution failures.
3. Do I need to open additional security‑group ports for the DNS forwarder?
   Only outbound UDP/TCP 53 to the forwarder IPs (e.g., 8.8.8.8). Ensure the Lambda’s security group allows egress to those IPs.
4. Will adding a NAT gateway increase latency for Pinecone queries?
   The additional hop adds < 5 ms of network latency in most AWS regions, which is negligible compared to the vector search time.
5. How can I test DNS resolution automatically during deployment?
   Add a post‑deployment script that runs dig +short <index>.svc.<region>.pinecone.io and fails the pipeline if the result is empty.

Related Topic Hub: Vector Databases Troubleshooting Hub