Problem
Nightly batch inference jobs run on Amazon EC2 instances inside a VPC. Each job launches a TensorFlow script that:
- Downloads a TensorFlow model checkpoint from
s3://ml-models-prod/checkpoints/ - Reads input data from
s3://ml-inputs/nightly/ - Writes inference results to
s3://ml-outputs/nightly/ - Updates a DynamoDB table
InferenceRunLogwith job metadata
Since the most recent deployment the jobs terminate within seconds, emitting an AccessDenied error for both S3 and DynamoDB. No changes were made to the inference code itself.
Root Cause
The failure originates from an IAM permission mismatch introduced during a routine security hardening sprint:
- The EC2 launch template was updated to use a new instance profile
ec2-batch-inference-prodthat does not include theAmazonS3ReadOnlyAccessandAmazonDynamoDBFullAccessmanaged policies previously attached. - Bucket policies on
ml-models-prodandml-outputscontain aaws:SourceVpcecondition that restricts access to a specific VPC endpoint ID. The new instance profile’s role ARN was not added to theaws:PrincipalArncondition list. - The DynamoDB table’s resource‑based policy also restricts access to a specific IAM role ARN, which was changed but not reflected in the table policy.
Because the role assumed by the EC2 instance no longer satisfies the resource policies, AWS returns AccessDenied for every S3 and DynamoDB API call.
Debug
Step‑by‑step investigation that reproduced the issue and isolated the missing permissions.
1. Examine CloudWatch Logs
2024-10-31T02:15:12.345Z ERROR boto3.exceptions.S3UploadFailedError: AccessDenied: Access Denied
2024-10-31T02:15:12.347Z Traceback (most recent call last):
File "/opt/inference/run.py", line 78, in main
s3_client.upload_fileobj(buf, output_bucket, key)
File "/usr/local/lib/python3.9/site-packages/boto3/s3/inject.py", line 140, in upload_fileobj
raise S3UploadFailedError(error_msg)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
2. Verify the IAM role attached to the instance
$ curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/
ec2-batch-inference-prod
$ curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/ec2-batch-inference-prod
{
"Code" : "Success",
"LastUpdated" : "2024-10-31T01:58:12Z",
"Type" : "AWS-Hybrid",
"AccessKeyId" : "ASIA...",
"SecretAccessKey" : "...",
"Token" : "...",
"Expiration" : "2024-11-01T07:58:12Z"
}
3. Inspect the role’s policies
$ aws iam list-attached-role-policies --role-name ec2-batch-inference-prod
{
"AttachedPolicies": [
{
"PolicyName": "EC2BatchInferenceBase",
"PolicyArn": "arn:aws:iam::123456789012:policy/EC2BatchInferenceBase"
}
]
}
$ aws iam get-policy-version --policy-arn arn:aws:iam::123456789012:policy/EC2BatchInferenceBase --version-id v1
{
"PolicyVersion": {
"Document": {
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
},
"VersionId": "v1",
"IsDefaultVersion": true
}
}
The policy only grants CloudWatch Logs permissions – no S3 or DynamoDB actions.
4. Test S3 access with the instance role
$ aws s3 ls s3://ml-models-prod/checkpoints/ --no-sign-request
An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
5. Review bucket policies
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"AllowVPCERead",
"Effect":"Allow",
"Principal":{"AWS":"arn:aws:iam::123456789012:role/ec2-batch-inference-old"},
"Action":["s3:GetObject","s3:ListBucket"],
"Resource":["arn:aws:s3:::ml-models-prod","arn:aws:s3:::ml-models-prod/*"],
"Condition":{"StringEquals":{"aws:SourceVpce":"vpce-0a1b2c3d4e5f6g7h"}}
}
]
}
The principal ARN does not match the new role.
Solution
Restore the required permissions by updating both the IAM role and the resource‑based policies.
1. Extend the instance role policy
Attach a custom policy that grants the exact S3 and DynamoDB actions needed.
Before (policy attached to ec2-batch-inference-prod)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "*"
}
]
}
After (new policy EC2BatchInferenceS3DDB)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:GetObject",
"s3:ListBucket",
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::ml-models-prod",
"arn:aws:s3:::ml-models-prod/*",
"arn:aws:s3:::ml-inputs",
"arn:aws:s3:::ml-inputs/*",
"arn:aws:s3:::ml-outputs",
"arn:aws:s3:::ml-outputs/*"
]
},
{
"Effect": "Allow",
"Action": [
"dynamodb:PutItem",
"dynamodb:UpdateItem",
"dynamodb:GetItem",
"dynamodb:Query"
],
"Resource": "arn:aws:dynamodb:us-east-1:123456789012:table/InferenceRunLog"
}
]
}
Attach the policy:
aws iam attach-role-policy \
--role-name ec2-batch-inference-prod \
--policy-arn arn:aws:iam::123456789012:policy/EC2BatchInferenceS3DDB
2. Update bucket policies to include the new role ARN
Replace the hard‑coded Principal.AWS entry with a list that contains both the old and new role ARNs (or use a wildcard for the account).
{
"Version":"2012-10-17",
"Statement":[
{
"Sid":"AllowVPCEReadWrite",
"Effect":"Allow",
"Principal":{"AWS":[
"arn:aws:iam::123456789012:role/ec2-batch-inference-old",
"arn:aws:iam::123456789012:role/ec2-batch-inference-prod"
]},
"Action":["s3:GetObject","s3:ListBucket","s3:PutObject"],
"Resource":["arn:aws:s3:::ml-models-prod","arn:aws:s3:::ml-models-prod/*",
"arn:aws:s3:::ml-inputs","arn:aws:s3:::ml-inputs/*",
"arn:aws:s3:::ml-outputs","arn:aws:s3:::ml-outputs/*"],
"Condition":{"StringEquals":{"aws:SourceVpce":"vpce-0a1b2c3d4e5f6g7h"}}
}
]
}
Apply with the CLI:
aws s3api put-bucket-policy \
--bucket ml-models-prod \
--policy file://bucket-policy.json
3. Update DynamoDB table resource policy (if present)
{
"Version":"2012-10-17",
"Statement":[
{
"Effect":"Allow",
"Principal":{"AWS":"arn:aws:iam::123456789012:role/ec2-batch-inference-prod"},
"Action":["dynamodb:PutItem","dynamodb:UpdateItem"],
"Resource":"arn:aws:dynamodb:us-east-1:123456789012:table/InferenceRunLog"
}
]
}
4. Redeploy the launch template
Ensure the new instance profile is referenced and that the latest AMI includes the updated awscli version (optional but recommended).
aws ec2 modify-launch-template \
--launch-template-id lt-0abcd1234efgh5678 \
--default-version 2
Verify
After applying the changes, run a manual test job and confirm each step succeeds.
- Launch an EC2 instance with the updated launch template.
- SSH into the instance and execute the inference script with
--dry-run(script prints S3 URLs before access).
Expected log excerpt:
2024-10-31T03:00:01.112Z INFO Downloading model from s3://ml-models-prod/checkpoints/model.ckpt
2024-10-31T03:00:02.845Z INFO Input data listed: 12,345 objects in s3://ml-inputs/nightly/
2024-10-31T03:15:47.321Z INFO Inference completed, writing results to s3://ml-outputs/nightly/run-20241031/
2024-10-31T03:15:48.004Z INFO DynamoDB entry created for run-20241031
Validate via AWS console or CLI:
aws s3 ls s3://ml-outputs/nightly/run-20241031/
2024-10-31 03:15:48 12345678 results.parquet
aws dynamodb get-item \
--table-name InferenceRunLog \
--key '{"RunId":{"S":"run-20241031"}}'
{
"Item": {
"RunId": {"S":"run-20241031"},
"Status": {"S":"COMPLETED"},
"StartTime": {"S":"2024-10-31T03:00:01Z"},
"EndTime": {"S":"2024-10-31T03:15:48Z"}
}
}
Prevent
- Policy as Code: Store IAM and bucket policies in a version‑controlled repository (e.g., Terraform, CloudFormation) and enforce PR reviews.
- Least‑Privilege Audits: Use IAM Access Analyzer to detect unused permissions after each deployment.
- Automated Policy Validation: Add a CI step that runs
aws iam simulate-principal-policyagainst the role to confirm required actions are allowed. - VPC Endpoint Consistency: Reference the VPC endpoint ID via a SSM Parameter Store entry so that bucket policies automatically pick up changes.
- Health‑Check Lambda: Periodically invoke a lightweight script that attempts a
HeadObjecton each bucket and aGetItemon the DynamoDB table; raise an alarm on failure.
Related Topic Hub: Cloud Infrastructure Troubleshooting Hub
FAQ
- Why does the AccessDenied error appear only after the security hardening sprint?
The sprint replaced the instance profile with a new role that lacked the S3/DynamoDB permissions and the bucket policies still referenced the old role ARN. - Can I use a wildcard (
*) in the bucket policy principal to avoid future role changes?
Yes, but it widens the attack surface. Prefer adding the new role ARN to an explicit list or use an IAM condition that matches the account ID. - Do VPC endpoint conditions affect cross‑region bucket access?
Theaws:SourceVpcecondition only matches requests that travel through the specified endpoint. If the bucket is in a different region without a matching endpoint, the request will be denied. - How can I test IAM permissions without launching a full EC2 instance?
Useaws sts assume-rolewith the target role ARN and then runaws s3 lsandaws dynamodb describe-tablefrom a local workstation. - Is it safe to grant
AmazonS3FullAccessto the role?
It resolves the immediate error but violates the principle of least privilege. Grant only the specific actions and resource ARNs required by the batch job.