FAISS index access denied after EC2 instance reboot

Problem – FAISS index access denied after EC2 instance reboot

During a disaster‑recovery (DR) drill an EC2 instance was launched to replace a failed node. The instance runs a Python service that restores a FAISS index from an S3 bucket using boto3 and then loads the index with faiss.read_index. After the reboot the service crashes with:

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied

or, when the download silently fails and FAISS attempts to open a local file that never materialised:

PermissionError: [Errno 13] Permission denied: '/mnt/faiss/index.bin'
faiss.io.read_index: IOError: [Errno 13] Permission denied

The root cause is an IAM permission mismatch between the EC2 instance profile and the S3 bucket policy, which only becomes apparent after a fresh instance is launched (the original instance had a manually attached policy).

Root Cause Analysis

• Instance profile missing S3 actions. The default EC2 role attached to the replacement instance does not include s3:GetObject (and often s3:ListBucket and s3:PutObject) for the bucket that stores the FAISS index files. This matches the discussion in GitHub issue #1245 where the same symptom was traced to a missing s3:GetObject permission.
• Bucket policy constraints. In one production incident the bucket policy restricted access to a specific VPC endpoint. The fail‑over instance was launched in a different subnet, causing the same AccessDenied error even though the instance role technically had the required actions. This is documented in the AWS S3 Access Control guide.
• FAISS I/O wrapper expectations. FAISS itself does not speak S3; the Python wrapper script uses boto3 to download the binary file to a local path before calling faiss.read_index. If the download fails, the subsequent file‑open raises PermissionError, which is the second error shown above.

Investigation and Debugging Steps

1. Confirm the instance profile. Run on the EC2 host:

curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/

Note the role name (e.g., EC2InstanceRole) and then inspect its policies:

aws iam get-role-policy --role-name EC2InstanceRole --policy-name S3AccessPolicy

Check CloudWatch logs for the exact error. Typical log entry:

2026-05-28T14:12:03.456Z ERROR botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
User: arn:aws:sts::123456789012:assumed-role/EC2InstanceRole/i-0abcd1234ef567890 is not authorized to perform: s3:GetObject on resource: arn:aws:s3:::faiss-backups/2026/05/28/index.bin

Validate bucket policy. Retrieve the bucket policy and look for Condition blocks that limit access by VPC endpoint or source IP.

aws s3api get-bucket-policy --bucket faiss-backups

Reproduce the download manually. Use the same credentials the application uses:

aws s3 cp s3://faiss-backups/2026/05/28/index.bin /tmp/index.bin

If this fails with AccessDenied, the problem is definitely IAM‑related.

Inspect local filesystem permissions. After a failed download, the target directory may be owned by root or have 700 permissions, causing FAISS to raise PermissionError. Verify:

ls -ld /mnt/faiss
ls -l /mnt/faiss/index.bin

Resolution – Align IAM and Bucket Policies

Step 1 – Create a minimal S3 access policy

Before (policy attached to the original instance, but missing on the new one):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::faiss-backups"
        }
    ]
}

After – add the required object actions and restrict to the specific bucket prefix used for FAISS indices:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket"
            ],
            "Resource": "arn:aws:s3:::faiss-backups",
            "Condition": {
                "StringLike": {
                    "s3:prefix": "2026/*"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject"
            ],
            "Resource": "arn:aws:s3:::faiss-backups/2026/*"
        }
    ]
}

Attach this policy to the instance role (or create a new managed policy) and ensure the role is associated with the EC2 instance profile.

Step 2 – Adjust bucket policy if VPC‑endpoint restriction exists

If the bucket policy contains a condition similar to:

"Condition": { "StringEquals": { "aws:sourceVpce": "vpce-0a1b2c3d4e5f6g7h8" } }

add the fail‑over subnet’s endpoint ID or broaden the condition to include all approved endpoints used by DR instances.

Step 3 – Ensure filesystem writeability

After the S3 download succeeds, FAISS will write a temporary file if faiss.write_index is called. Guarantee the mount point is writable by the service user:

sudo chown -R faisssvc:faisssvc /mnt/faiss
chmod 750 /mnt/faiss

Step 4 – Deploy the corrected role

Either stop the instance and replace its IAM role, or use the AWS Systems Manager aws:attachIamInstanceProfile automation to attach the updated profile without a reboot.

Verification – Confirm Successful Index Restoration

• Run the download command manually and verify the file size matches the S3 object:

aws s3 cp s3://faiss-backups/2026/05/28/index.bin /tmp/index.bin
aws s3api head-object --bucket faiss-backups --key 2026/05/28/index.bin
ls -l /tmp/index.bin

Execute the FAISS load script and watch logs for a successful message:

2026-05-28T14:18:12.001Z INFO Downloaded index.bin (12.4 MB) from S3
2026-05-28T14:18:12.123Z INFO faiss.read_index succeeded, vector count: 4,589,321

Check CloudWatch metric FAISSIndexLoadSuccess (custom metric) increments to 1.

Perform a simple vector search to ensure the index is operational:

import faiss, numpy as np
index = faiss.read_index('/tmp/index.bin')
query = np.random.random((1, 128)).astype('float32')
D, I = index.search(query, k=5)
print(I)

Successful output (e.g., [[12345 67890 ...]]) confirms the index is usable.

Prevention – Operational Guardrails

Guardrail	Implementation
IAM baseline policy	Attach a managed policy named FAISS_S3_Access to every EC2 role that runs FAISS services. Include s3:GetObject, s3:PutObject, and s3:ListBucket scoped to the bucket prefix.
Bucket policy audit	Enable AWS Config rule s3-bucket-policy-prohibit-public-access and a custom rule that checks for aws:sourceVpce conditions covering all DR VPC endpoints.
Startup health check	In the service entrypoint, attempt a cheap HeadObject request for a known test file. Exit with non‑zero status if the call fails; the orchestrator will then retry on a correctly provisioned instance.
Filesystem permissions	Provision the mount point via CloudFormation with fsx:FileSystem or EFS and set PosixUser to match the service UID.
Monitoring & alerting	Create a CloudWatch metric filter on logs for AccessDenied from the FAISS restore script and trigger a high‑severity alarm.

FAQ – Common Follow‑up Questions

1. Why does the same EC2 role work on the primary node but not on the DR node? The primary node may have been launched with an older role that already contained the S3 permissions. DR instances are often created from a generic launch template that only attaches the default EC2 role, which lacks the custom S3AccessPolicy. Verify the exact role ARN in the instance metadata.
2. Can I avoid downloading the index to local disk and read directly from S3? FAISS does not natively support streaming from S3. You must download the binary file first (or use an EFS mount that is backed by S3 via DataSync). The download step must succeed before faiss.read_index is called.
3. What bucket policy condition should I use to allow access from any DR subnet? Use the aws:sourceVpce condition with a list of approved VPC endpoint IDs, or broaden to aws:sourceVpc if all subnets belong to the same VPC. Example:

"Condition": {
    "StringEquals": {
        "aws:sourceVpc": ["vpc-0a1b2c3d"]
    }
}

4. How do I test IAM permissions without restarting the service? Use the AWS CLI or boto3 to perform a HeadObject call for the index key. If it succeeds, the role has the required s3:GetObject permission.
5. Is there a way to automatically attach the correct role during a DR launch? Yes. Include the IamInstanceProfile attribute in the Auto Scaling launch template or CloudFormation stack that creates the DR instance. You can also use an EC2 instance lifecycle hook to run a script that validates the attached role before the service starts.

Related Topic Hub: Vector Databases Troubleshooting Hub