Female speaker presenting in front of a projector screen

RAG pipeline returns empty retrieval results on AWS EC2 local dev

by

Problem: RAG pipeline returns empty retrieval results on an AWS EC2 instance (local development)

During local development on an EC2 t3.medium (Ubuntu 22.04) the LangChain retrieval‑augmented generation (RAG) pipeline executes without errors but always yields the log message “No documents found”. The same code runs correctly on a developer laptop.

Typical symptom in the application logs:


2026-09-18 10:12:34,567 - langchain.retrievers - INFO - Retrieval result: []
2026-09-18 10:12:34,568 - langchain.chains - WARNING - No documents found for query: "What is the refund policy?"

Other observed errors that may appear intermittently:

  • ConnectionError: [Errno 111] Connection refused – when the vector store endpoint cannot be reached.
  • AWS credentials not found – when the instance role lacks required permissions.
  • RuntimeError: FAISS index not loaded – file not found or corrupted – when the index file is missing or unreadable.
  • MemoryError: unable to allocate X bytes for FAISS index – when the instance runs out of RAM.

Root Cause Analysis

The empty retrieval result is rarely caused by a single factor. In production‑grade EC2 environments the following interactions are the most common culprits, as documented in the evidence package:

Potential Cause Why it leads to empty results Evidence Source
Security group blocks outbound HTTPS/9200 Vector store (OpenSearch, Elasticsearch) cannot be contacted; LangChain falls back to an empty list without raising a hard error. deepset‑ai/haystack#3841
IAM role missing s3:GetObject for index bucket FAISS index stored in S3 fails to download; the loader silently returns an empty index. Real incident – missing S3 permissions
Missing runtime library (libtorch, libstdc++) FAISS backend crashes during index load, raising RuntimeError that is caught and logged as “No documents found”. langchain‑ai/langchain#5872
Insufficient RAM for FAISS index Index load aborts with MemoryError; the pipeline proceeds with an empty in‑memory store. Reddit discussion on FAISS RAM limits
Incorrect AWS_REGION or missing env var Boto3 cannot resolve the OpenSearch endpoint, resulting in a connection failure. Stack Overflow – missing AWS_REGION

In the specific incident described in the evidence package, two root causes overlapped:

  1. The EC2 instance’s security group lacked outbound HTTPS (port 443) and OpenSearch port 9200, preventing the vector store from being reached.
  2. The attached IAM role did not include s3:GetObject for the bucket that holds the pre‑computed FAISS index, so the index never loaded.

Investigation and Debugging Steps

1. Verify network connectivity to the vector store

# Replace with your OpenSearch endpoint
ENDPOINT="search-my-rag-domain.us-east-1.es.amazonaws.com"

# Test TCP connectivity on port 443 (HTTPS) and 9200 (OpenSearch)
nc -zv $ENDPOINT 443
nc -zv $ENDPOINT 9200

Expected output when the security group allows traffic:


Connection to search-my-rag-domain.us-east-1.es.amazonaws.com 443 port [tcp/https] succeeded!
Connection to search-my-rag-domain.us-east-1.es.amazonaws.com 9200 port [tcp/elasticsearch] succeeded!

If you see Connection timed out or Connection refused, inspect the EC2 security group and any VPC NACLs (AWS EC2 User Guide – Networking and Security).

2. Check IAM role permissions

aws sts get-caller-identity
aws iam get-role --role-name MyRagInstanceRole
aws iam list-attached-role-policies --role-name MyRagInstanceRole

Ensure the role includes policies such as AmazonS3ReadOnlyAccess (or a custom policy with s3:GetObject) and AmazonOpenSearchServiceFullAccess for the OpenSearch domain.

3. Confirm environment variables

echo $AWS_REGION
echo $OPENSEARCH_ENDPOINT

If AWS_REGION is empty, LangChain’s OpenSearch client cannot resolve the endpoint, leading to silent failures (Stack Overflow reference).

4. Validate FAISS index loading

python - <<'PY'
from langchain.vectorstores import FAISS
import os

index_path = os.getenv("FAISS_INDEX_PATH", "/data/faiss.index")
try:
    vectorstore = FAISS.load_local(index_path, embeddings)
    print("FAISS index loaded, vectors:", len(vectorstore.index_to_docstore_id))
except Exception as e:
    print("FAISS load error:", e)
PY

Typical error when the index file is missing or unreadable:


FAISS load error: RuntimeError: FAISS index not loaded – file not found or corrupted

5. Inspect application logs for suppressed exceptions

LangChain often catches RuntimeError from the vector store and logs only “No documents found”. Increase log verbosity:

export LANGCHAIN_LOG_LEVEL=DEBUG
python run_rag.py

Look for stack traces that point to network timeouts or permission errors.

Resolution

1. Update Security Group

Allow outbound HTTPS (443) and OpenSearch (9200) to the VPC endpoint or public OpenSearch domain.

Rule Type Protocol Port Destination
Outbound 1 HTTPS TCP 443 0.0.0.0/0 (or VPC CIDR)
Outbound 2 OpenSearch TCP 9200 VPC CIDR or specific OpenSearch SG

Example AWS CLI command:

aws ec2 authorize-security-group-egress \
    --group-id sg-0abc123def456ghi \
    --protocol tcp --port 443 --cidr 0.0.0.0/0

aws ec2 authorize-security-group-egress \
    --group-id sg-0abc123def456ghi \
    --protocol tcp --port 9200 --cidr 10.0.0.0/16

2. Attach a proper IAM role

Create or modify a role with the required policies and associate it with the EC2 instance.

aws iam create-policy \
    --policy-name RAGS3ReadAccess \
    --policy-document '{
        "Version":"2012-10-17",
        "Statement":[{
            "Effect":"Allow",
            "Action":["s3:GetObject"],
            "Resource":["arn:aws:s3:::my-rag-bucket/*"]
        }]
    }'

aws iam attach-role-policy \
    --role-name MyRagInstanceRole \
    --policy-arn arn:aws:iam::123456789012:policy/RAGS3ReadAccess

3. Install missing runtime libraries

On Ubuntu, ensure libtorch and libstdc++6 are present:

sudo apt-get update
sudo apt-get install -y libtorch-dev libstdc++6

After installation, re‑run the index loader to confirm the FAISS index loads without RuntimeError.

4. Set required environment variables

# .bashrc or systemd service file
export AWS_REGION=us-east-1
export OPENSEARCH_ENDPOINT=https://search-my-rag-domain.us-east-1.es.amazonaws.com
export FAISS_INDEX_PATH=/data/faiss.index

5. Adjust instance size if RAM is insufficient

FAISS indexes for >10k documents often need >4 GiB of RAM. Upgrade to t3.large or add swap space as a temporary mitigation.

# Add 2 GiB swap
sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile

Verification

After applying the fixes, run the pipeline with debug logging and confirm that documents are returned.

export LANGCHAIN_LOG_LEVEL=DEBUG
python run_rag.py "What is the refund policy?"

Expected log excerpt:


2026-09-18 10:45:12,001 - langchain.retrievers - INFO - Retrieval result: [{'page_content': 'Our refund policy ...', 'metadata': {...}}]
2026-09-18 10:45:12,005 - langchain.chains - INFO - Generated answer: "The refund policy allows ..."

Additional validation steps:

  • Run curl -s $OPENSEARCH_ENDPOINT/_cat/indices?v to ensure the OpenSearch index is visible.
  • Check CloudWatch metrics for successful Search requests (HTTP 200) from the EC2 instance.
  • Execute a unit test that queries the vector store directly and asserts len(results) > 0.

Prevention and Best Practices

  • Network hygiene: Keep a dedicated security group for RAG instances that explicitly allows outbound traffic to vector store ports (443, 9200) and inbound health‑check ports if needed.
  • IAM least‑privilege: Attach a role with only s3:GetObject for the index bucket and es:ESHttp* for the OpenSearch domain.
  • Dependency management: Pin OS packages (e.g., libtorch, libstdc++6) in a Dockerfile or AMI image; verify them after OS upgrades.
  • Resource sizing: Profile FAISS index memory usage; provision instances with at least 2× the estimated RAM requirement.
  • Health checks: Add a startup script that attempts a simple vector store query and aborts the service if the result list is empty.
  • Observability: Emit a custom CloudWatch metric “RAGEmptyRetrievals” and set an alarm when the count exceeds a threshold.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

  1. Why does the RAG pipeline work on my laptop but not on EC2?

    The laptop typically runs with unrestricted outbound internet access and has the required libraries installed. EC2 instances are isolated by security groups, VPC endpoints, and IAM roles, which can block network traffic or deny S3/OpenSearch access.

  2. What does “ConnectionError: [Errno 111] Connection refused” indicate in this context?

    The EC2 security group or VPC NACL is blocking the port used by the vector store (usually 443 for HTTPS or 9200 for OpenSearch). Updating the outbound rules resolves the error.

  3. How can I confirm that the FAISS index was actually loaded?

    After calling FAISS.load_local(), print the number of vectors: len(vectorstore.index_to_docstore_id). A non‑zero count confirms a successful load.

  4. Is the AWS_REGION environment variable always required?

    Yes. Boto3 uses AWS_REGION to construct the endpoint for services like OpenSearch. Missing or mismatched values cause endpoint resolution failures (Stack Overflow evidence).

  5. Can insufficient RAM cause the index to load silently?

    FAISS raises a MemoryError when allocation fails. If the exception is caught and ignored by the application, the pipeline proceeds with an empty in‑memory store, resulting in “No documents found”. Monitoring RAM usage and provisioning adequate instance size prevents this.