IAM permission denied when attaching role during blue‑green EC2 deployment

Problem – IAM Permission Denied When Attaching Role During Blue‑Green EC2 Deployment

A fully automated blue‑green deployment creates a fresh EC2 fleet, assigns a pre‑defined IAM role to each instance, switches the load balancer target group, and finally terminates the old fleet. During the AssociateIamInstanceProfile step the deployment script aborts with:

ClientError: An error occurred (UnauthorizedOperation) when calling the AssociateIamInstanceProfile operation: 
User: arn:aws:sts::123456789012:assumed-role/DeployAutomationRole/i-0a1b2c3d4e5f6g7h is not authorized to perform: iam:PassRole on resource: arn:aws:iam::123456789012:role/BlueGreenAppRole

The failure prevents the new fleet from receiving the permissions it needs, causing the traffic cut‑over to stall.

Root Cause – Missing iam:PassRole Permission and Trust Mismatch

• What the API does: AssociateIamInstanceProfile requires the caller to have iam:PassRole on the instance profile role that will be attached to the EC2 instance.
• What was assumed: The DeployAutomationRole already had broad ec2:* permissions, so the team omitted an explicit iam:PassRole grant.
• Why it failed only in production: The development environment uses a different instance profile (DevAppRole) that is explicitly listed in the policy; the production role (BlueGreenAppRole) was not.
• Additional factor: The target role’s trust policy allowed only ec2.amazonaws.com, but the deployment script uses an assumed role to call the API, which is not covered by the trust relationship. The API therefore rejects the pass attempt even if the permission existed.

Debug – Step‑by‑Step Investigation

1. Inspect the failing CloudTrail event

{
    "eventVersion": "1.08",
    "userIdentity": {
        "type": "AssumedRole",
        "arn": "arn:aws:sts::123456789012:assumed-role/DeployAutomationRole/i-0a1b2c3d4e5f6g7h",
        "accountId": "123456789012",
        "accessKeyId": "ASIA...",
        "sessionContext": { ... }
    },
    "eventSource": "ec2.amazonaws.com",
    "eventName": "AssociateIamInstanceProfile",
    "errorCode": "UnauthorizedOperation",
    "errorMessage": "User ... is not authorized to perform: iam:PassRole ..."
}

2. Simulate the policy with IAM Policy Simulator

aws iam simulate-principal-policy \
    --policy-source-arn arn:aws:iam::123456789012:role/DeployAutomationRole \
    --action-names iam:PassRole \
    --resource-arns arn:aws:iam::123456789012:role/BlueGreenAppRole

Result: explicitDeny – no matching iam:PassRole statement.

3. Verify the target role’s trust policy

aws iam get-role --role-name BlueGreenAppRole | jq '.Role.AssumeRolePolicyDocument'

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "Service": "ec2.amazonaws.com" },
      "Action": "sts:AssumeRole"
    }
  ]
}

The trust does not include the deployment automation account, so even with iam:PassRole the call would be rejected.

Solution – Grant iam:PassRole and Align Trust Policies

Before – Policy attached to DeployAutomationRole

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "ec2:Describe*"
            ],
            "Resource": "*"
        }
        // No iam:PassRole entry
    ]
}

After – Updated policy with least‑privilege iam:PassRole

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "ec2:RunInstances",
                "ec2:TerminateInstances",
                "ec2:Describe*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::123456789012:role/BlueGreenAppRole"
        }
    ]
}

Update the target role’s trust policy to accept the automation role

aws iam update-assume-role-policy \
    --role-name BlueGreenAppRole \
    --policy-document '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "AWS": "arn:aws:iam::123456789012:role/DeployAutomationRole"
                },
                "Action": "sts:AssumeRole"
            },
            {
                "Effect": "Allow",
                "Principal": { "Service": "ec2.amazonaws.com" },
                "Action": "sts:AssumeRole"
            }
        ]
    }'

Now the automation role can both pass the instance profile and be trusted by the instance when it assumes the role.

Verify – Confirm the Fix Works

1. Re‑run the blue‑green deployment script.
2. Observe the CloudWatch Logs for the AssociateIamInstanceProfile call – it should return 200 OK with no error.
3. Check the instance metadata on a newly launched host:

curl -s http://169.254.169.254/latest/meta-data/iam/security-credentials/BlueGreenAppRole

{
  "Code" : "Success",
  "LastUpdated" : "2026-10-20T12:34:56Z",
  "Type" : "AWS-Hybrid",
  "AccessKeyId" : "ASIA...",
  "SecretAccessKey" : "...",
  "Token" : "...",
  "Expiration" : "2026-10-20T18:34:56Z"
}

Run a functional smoke test against the new target group to ensure the application can access required AWS services (e.g., S3, DynamoDB).

Prevent – Operational Guardrails

• Policy as Code: Store IAM policies in a version‑controlled repository and run terraform validate or cfn‑lint checks that enforce presence of iam:PassRole for any role used in deployment pipelines.
• IAM Access Analyzer: Enable Analyzer on the account; it will flag any role that can be passed without an explicit trust relationship.
• Automated Tests: Include an integration test that launches a single instance with the target profile and verifies curl against the metadata endpoint.
• Least‑Privilege Review: Periodically run aws iam get-policy-simulation across all deployment roles to detect drift.
• Alerting: Create a CloudWatch metric filter on CloudTrail for eventName=AssociateIamInstanceProfile with errorCode=UnauthorizedOperation and trigger an SNS alarm.

FAQ – Common Follow‑Up Questions

1. Why does the error appear only for the production role and not for dev?
   Because the dev role was explicitly listed in the original iam:PassRole statement, while the production role was added later without updating the policy.
2. Can I avoid adding the automation role to the instance role’s trust policy?
   Yes, if you use an instance profile that the automation role passes, the instance itself only needs the ec2.amazonaws.com principal. The trust addition is required only when the automation role also needs to assume the target role directly (e.g., for STS token exchange).
3. Is iam:PassRole required for RunInstances when specifying an instance profile?
   Yes. When --iam-instance-profile is supplied, EC2 internally calls PassRole. The caller must have that permission on the exact role ARN.
4. How do I limit the scope of iam:PassRole to a specific environment?
   Use resource ARNs that include the environment tag, e.g., arn:aws:iam::123456789012:role/BlueGreenAppRole-prod, and add a condition on aws:ResourceTag/Environment if you tag roles.
5. What CloudWatch metric should I monitor to catch this regression early?
   Create a metric filter on { $.eventName = "AssociateIamInstanceProfile" && $.errorCode = "UnauthorizedOperation" } and set an alarm to fire on any occurrence.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub