EC2 nightly batch inference fails with IAM AccessDenied on S3

Problem

Nightly batch inference jobs run on Amazon EC2 instances inside a VPC. Each job launches a TensorFlow script that:

  • Downloads a TensorFlow model checkpoint from s3://ml-models-prod/checkpoints/
  • Reads input data from s3://ml-inputs/nightly/
  • Writes inference results to s3://ml-outputs/nightly/
  • Updates a DynamoDB table InferenceRunLog with job metadata

Since the most recent deployment the jobs terminate within seconds, emitting an AccessDenied error for both S3 and DynamoDB. No changes were made to the inference code itself.

Root Cause

The failure originates from an IAM permission mismatch introduced during a routine security hardening sprint:

  • The EC2 launch template was updated to use a new instance profile ec2-batch-inference-prod that does not include the AmazonS3ReadOnlyAccess and AmazonDynamoDBFullAccess managed policies previously attached.
  • Bucket policies on ml-models-prod and ml-outputs contain a aws:SourceVpce condition that restricts access to a specific VPC endpoint ID. The new instance profile’s role ARN was not added to the aws:PrincipalArn condition list.
  • The DynamoDB table’s resource‑based policy also restricts access to a specific IAM role ARN, which was changed but not reflected in the table policy.

Because the role assumed by the EC2 instance no longer satisfies the resource policies, AWS returns AccessDenied for every S3 and DynamoDB API call.

Debug

Step‑by‑step investigation that reproduced the issue and isolated the missing permissions.

1. Examine CloudWatch Logs


2024-10-31T02:15:12.345Z ERROR boto3.exceptions.S3UploadFailedError: AccessDenied: Access Denied
2024-10-31T02:15:12.347Z Traceback (most recent call last):
  File "/opt/inference/run.py", line 78, in main
    s3_client.upload_fileobj(buf, output_bucket, key)
  File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 140, in upload_fileobj
    raise S3UploadFailedError(error_msg)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

2. Verify the IAM role attached to the instance


$ curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
ec2-batch-inference-prod
$ curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-batch-inference-prod
{
  "Code" : "Success",
  "LastUpdated" : "2024-10-31T01:58:12Z",
  "Type" : "AWS-Hybrid",
  "AccessKeyId" : "ASIA...",
  "SecretAccessKey" : "...",
  "Token" : "...",
  "Expiration" : "2024-11-01T07:58:12Z"
}

3. Inspect the role’s policies


$ aws iam list-attached-role-policies --role-name ec2-batch-inference-prod
{
    "AttachedPolicies": [
        {
            "PolicyName": "EC2BatchInferenceBase",
            "PolicyArn": "arn:aws:iam::123456789012:policy/EC2BatchInferenceBase"
        }
    ]
}
$ aws iam get-policy-version --policy-arn arn:aws:iam::123456789012:policy/EC2BatchInferenceBase --version-id v1
{
    "PolicyVersion": {
        "Document": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Action": [
                        "logs:CreateLogGroup",
                        "logs:CreateLogStream",
                        "logs:PutLogEvents"
                    ],
                    "Resource": "*"
                }
            ]
        },
        "VersionId": "v1",
        "IsDefaultVersion": true
    }
}

The policy only grants CloudWatch Logs permissions – no S3 or DynamoDB actions.

4. Test S3 access with the instance role


$ aws s3 ls s3://ml-models-prod/checkpoints/ --no-sign-request
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

5. Review bucket policies


{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"AllowVPCERead",
      "Effect":"Allow",
      "Principal":{"AWS":"arn:aws:iam::123456789012:role/ec2-batch-inference-old"},
      "Action":["s3:GetObject","s3:ListBucket"],
      "Resource":["arn:aws:s3:::ml-models-prod","arn:aws:s3:::ml-models-prod/*"],
      "Condition":{"StringEquals":{"aws:SourceVpce":"vpce-0a1b2c3d4e5f6g7h"}}
    }
  ]
}

The principal ARN does not match the new role.

Solution

Restore the required permissions by updating both the IAM role and the resource‑based policies.

1. Extend the instance role policy

Attach a custom policy that grants the exact S3 and DynamoDB actions needed.

Before (policy attached to ec2-batch-inference-prod)


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

After (new policy EC2BatchInferenceS3DDB)


{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::ml-models-prod",
                "arn:aws:s3:::ml-models-prod/*",
                "arn:aws:s3:::ml-inputs",
                "arn:aws:s3:::ml-inputs/*",
                "arn:aws:s3:::ml-outputs",
                "arn:aws:s3:::ml-outputs/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:PutItem",
                "dynamodb:UpdateItem",
                "dynamodb:GetItem",
                "dynamodb:Query"
            ],
            "Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/InferenceRunLog"
        }
    ]
}

Attach the policy:


aws iam attach-role-policy \
    --role-name ec2-batch-inference-prod \
    --policy-arn arn:aws:iam::123456789012:policy/EC2BatchInferenceS3DDB

2. Update bucket policies to include the new role ARN

Replace the hard‑coded Principal.AWS entry with a list that contains both the old and new role ARNs (or use a wildcard for the account).


{
  "Version":"2012-10-17",
  "Statement":[
    {
      "Sid":"AllowVPCEReadWrite",
      "Effect":"Allow",
      "Principal":{"AWS":[
        "arn:aws:iam::123456789012:role/ec2-batch-inference-old",
        "arn:aws:iam::123456789012:role/ec2-batch-inference-prod"
      ]},
      "Action":["s3:GetObject","s3:ListBucket","s3:PutObject"],
      "Resource":["arn:aws:s3:::ml-models-prod","arn:aws:s3:::ml-models-prod/*",
                  "arn:aws:s3:::ml-inputs","arn:aws:s3:::ml-inputs/*",
                  "arn:aws:s3:::ml-outputs","arn:aws:s3:::ml-outputs/*"],
      "Condition":{"StringEquals":{"aws:SourceVpce":"vpce-0a1b2c3d4e5f6g7h"}}
    }
  ]
}

Apply with the CLI:


aws s3api put-bucket-policy \
    --bucket ml-models-prod \
    --policy file://bucket-policy.json

3. Update DynamoDB table resource policy (if present)


{
    "Version":"2012-10-17",
    "Statement":[
        {
            "Effect":"Allow",
            "Principal":{"AWS":"arn:aws:iam::123456789012:role/ec2-batch-inference-prod"},
            "Action":["dynamodb:PutItem","dynamodb:UpdateItem"],
            "Resource":"arn:aws:dynamodb:us-east-1:123456789012:table/InferenceRunLog"
        }
    ]
}

4. Redeploy the launch template

Ensure the new instance profile is referenced and that the latest AMI includes the updated awscli version (optional but recommended).


aws ec2 modify-launch-template \
    --launch-template-id lt-0abcd1234efgh5678 \
    --default-version 2

Verify

After applying the changes, run a manual test job and confirm each step succeeds.

  1. Launch an EC2 instance with the updated launch template.
  2. SSH into the instance and execute the inference script with --dry-run (script prints S3 URLs before access).

Expected log excerpt:


2024-10-31T03:00:01.112Z INFO Downloading model from s3://ml-models-prod/checkpoints/model.ckpt
2024-10-31T03:00:02.845Z INFO Input data listed: 12,345 objects in s3://ml-inputs/nightly/
2024-10-31T03:15:47.321Z INFO Inference completed, writing results to s3://ml-outputs/nightly/run-20241031/
2024-10-31T03:15:48.004Z INFO DynamoDB entry created for run-20241031

Validate via AWS console or CLI:


aws s3 ls s3://ml-outputs/nightly/run-20241031/
2024-10-31 03:15:48       12345678 results.parquet

aws dynamodb get-item \
    --table-name InferenceRunLog \
    --key '{"RunId":{"S":"run-20241031"}}'
{
    "Item": {
        "RunId": {"S":"run-20241031"},
        "Status": {"S":"COMPLETED"},
        "StartTime": {"S":"2024-10-31T03:00:01Z"},
        "EndTime": {"S":"2024-10-31T03:15:48Z"}
    }
}

Prevent

  • Policy as Code: Store IAM and bucket policies in a version‑controlled repository (e.g., Terraform, CloudFormation) and enforce PR reviews.
  • Least‑Privilege Audits: Use IAM Access Analyzer to detect unused permissions after each deployment.
  • Automated Policy Validation: Add a CI step that runs aws iam simulate-principal-policy against the role to confirm required actions are allowed.
  • VPC Endpoint Consistency: Reference the VPC endpoint ID via a SSM Parameter Store entry so that bucket policies automatically pick up changes.
  • Health‑Check Lambda: Periodically invoke a lightweight script that attempts a HeadObject on each bucket and a GetItem on the DynamoDB table; raise an alarm on failure.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

  1. Why does the AccessDenied error appear only after the security hardening sprint?
    The sprint replaced the instance profile with a new role that lacked the S3/DynamoDB permissions and the bucket policies still referenced the old role ARN.
  2. Can I use a wildcard (*) in the bucket policy principal to avoid future role changes?
    Yes, but it widens the attack surface. Prefer adding the new role ARN to an explicit list or use an IAM condition that matches the account ID.
  3. Do VPC endpoint conditions affect cross‑region bucket access?
    The aws:SourceVpce condition only matches requests that travel through the specified endpoint. If the bucket is in a different region without a matching endpoint, the request will be denied.
  4. How can I test IAM permissions without launching a full EC2 instance?
    Use aws sts assume-role with the target role ARN and then run aws s3 ls and aws dynamodb describe-table from a local workstation.
  5. Is it safe to grant AmazonS3FullAccess to the role?
    It resolves the immediate error but violates the principle of least privilege. Grant only the specific actions and resource ARNs required by the batch job.