IAM permission denied calling Anthropic Claude API from training loop

Problem

During a distributed PyTorch training run the train_step() function invokes the Anthropic Claude API to generate reference responses. After the first few iterations the job aborts with an IAM permission denied error. The failure appears in the worker logs and prevents any further API calls, causing the entire training pipeline to stall.

Observed Symptoms

• Training job exits with RuntimeError: IAM permission denied (status code 403).
• CloudWatch log entry (example):

2026-10-20T14:32:11.842Z ERROR  [worker-3] Request to https://api.anthropic.com/v1/complete failed: 403 Forbidden
2026-10-20T14:32:11.842Z ERROR  [worker-3] Response body: {"type":"error","error":{"type":"permission_error","message":"IAM permission denied"}} 

• Only the processes that assume the ml-training-role encounter the error; other services using the same secret succeed.
• Metrics show a sudden drop in api_success_rate from 100 % to 0 % after iteration 5.

Root Cause

The training containers run on Amazon EKS and obtain credentials via the node’s IAM role (arn:aws:iam::123456789012:role/eks-nodegroup-role). The code uses the anthropic Python SDK, which reads the ANTHROPIC_API_KEY from AWS Secrets Manager. The SDK internally calls sts:AssumeRole to retrieve temporary credentials for the secret.

Two policy gaps caused the denial:

1. Missing secretsmanager:GetSecretValue permission on the secret that stores the Anthropic API key. The role only had secretsmanager:DescribeSecret.
2. Missing iam:PassRole permission for the role that the SDK tries to assume (anthropic-api-access-role). Without PassRole the STS call fails, and the SDK translates the STS error into “IAM permission denied”.

Because the training loop retries the API call on each iteration, the first successful call (when the container still held a cached credential) masks the problem until the credential cache expires, after which every request fails.

Debug

Step‑by‑step investigation

1. Inspect the failing container’s environment:

# Inside the training pod
env | grep ANTHROPIC
ANTHROPIC_API_KEY=arn:aws:secretsmanager:us-east-1:123456789012:secret:anthropic/api-key-abc123

2. Check the IAM role attached to the pod (EKS pod IAM role via IRSA):

# From the control plane
aws eks describe-nodegroup \
    --cluster-name ml-cluster \
    --nodegroup-name training-ng \
    --query 'nodegroup.iam.instanceProfileArn' \
    --output text
arn:aws:iam::123456789012:instance-profile/eks-nodegroup-profile

3. List the inline policies for the role:

aws iam list-attached-role-policies \
    --role-name eks-nodegroup-role \
    --output table
+---------------------------+---------------------------------------+
|      PolicyName           |               PolicyArn               |
+---------------------------+---------------------------------------+
| AmazonEKSWorkerNodePolicy | arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy |
| CustomTrainingPolicy      | arn:aws:iam::123456789012:policy/CustomTrainingPolicy |
+---------------------------+---------------------------------------+

4. Examine CustomTrainingPolicy contents:

aws iam get-policy-version \
    --policy-arn arn:aws:iam::123456789012:policy/CustomTrainingPolicy \
    --version-id v1 \
    --query 'PolicyVersion.Document' \
    --output json | jq .
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:anthropic/api-key-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRole"
      ],
      "Resource": "arn:aws:iam::123456789012:role/anthropic-api-access-role"
    }
  ]
}

The policy lacks secretsmanager:GetSecretValue and iam:PassRole for the assumed role.

Reproduce the failure locally

# Simulate the SDK call with the same role
aws sts assume-role \
    --role-arn arn:aws:iam::123456789012:role/anthropic-api-access-role \
    --role-session-name test-session
# Expected error:
An error occurred (AccessDenied) when calling the AssumeRole operation: User: arn:aws:sts::123456789012:assumed-role/eks-nodegroup-role/... is not authorized to perform: iam:PassRole on resource: arn:aws:iam::123456789012:role/anthropic-api-access-role

Solution

Policy amendment

Update CustomTrainingPolicy to grant the missing actions. The following JSON shows the corrected policy.

Before:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["secretsmanager:DescribeSecret"],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:anthropic/api-key-*"
    },
    {
      "Effect": "Allow",
      "Action": ["sts:AssumeRole"],
      "Resource": "arn:aws:iam::123456789012:role/anthropic-api-access-role"
    }
  ]
}

After (add secretsmanager:GetSecretValue and iam:PassRole):

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:DescribeSecret",
        "secretsmanager:GetSecretValue"
      ],
      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:anthropic/api-key-*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "sts:AssumeRole",
        "iam:PassRole"
      ],
      "Resource": "arn:aws:iam::123456789012:role/anthropic-api-access-role"
    }
  ]
}

Apply the updated policy:

aws iam create-policy-version \
    --policy-arn arn:aws:iam::123456789012:policy/CustomTrainingPolicy \
    --policy-document file://custom_training_policy_v2.json \
    --set-as-default

Code adjustment (optional)

If the SDK is configured to use a custom role, ensure the role name matches the policy:

# training_loop.py
import anthropic
import os
from boto3 import client as boto3_client

def get_anthropic_client():
    # Retrieve the API key from Secrets Manager
    sm = boto3_client('secretsmanager')
    secret_arn = os.getenv('ANTHROPIC_API_KEY')
    resp = sm.get_secret_value(SecretId=secret_arn)
    api_key = resp['SecretString']

    # Explicitly pass the role ARN if using a custom role
    return anthropic.Anthropic(
        api_key=api_key,
        # New: provide the role ARN so the SDK does not attempt an extra assume
        # (requires iam:PassRole which we now grant)
        assume_role_arn='arn:aws:iam::123456789012:role/anthropic-api-access-role'
    )

Verify

1. Redeploy the training job (or restart the pods) so the new IAM credentials are fetched.
2. Watch the pod logs for the first API call:

2026-10-20T14:45:02.113Z INFO  [worker-1] Claude response received: {"completion":"..."} 

Confirm the CloudWatch metric api_success_rate stays at 100 % across iterations.

Run a one‑off test script:

# test_anthropic.py
from training_loop import get_anthropic_client
client = get_anthropic_client()
print(client.completions.create(
    model="claude-2.1",
    prompt="Hello, world!",
    max_tokens=10
))

Expected output (truncated):

{'completion': 'Hello! How can I help you today?'}

Verify that aws sts get-caller-identity inside the container returns the eks-nodegroup-role ARN, confirming the role assumption succeeded.

Prevent

• Least‑privilege IAM design: Include secretsmanager:GetSecretValue whenever a secret’s value is required, not just DescribeSecret.
• Explicit iam:PassRole audit: Use IAM Access Analyzer to detect missing PassRole permissions for roles that are assumed by SDKs.
• Credential cache awareness: Set AWS_METADATA_SERVICE_TIMEOUT=5 and AWS_METADATA_SERVICE_NUM_ATTEMPTS=3 in the container to force early refresh and surface permission issues sooner.
• Automated policy tests: Add a CI step that runs aws iam simulate-principal-policy against the training role for the required actions.
• Monitoring: Create a CloudWatch metric filter on log pattern "IAM permission denied" and set an alarm to trigger on the first occurrence.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

1. Why does the error appear only after a few training iterations?
   The Anthropic SDK caches the secret value and temporary credentials. The first few calls succeed with the cached token; once the cache expires (default 1 hour) the SDK fetches a new token, hits the missing permissions, and returns 403.
2. Do I need iam:PassRole if I use sts:AssumeRole directly?
   Yes. sts:AssumeRole requires the caller to have iam:PassRole on the target role. Without it the STS call fails with AccessDenied, which the SDK surfaces as “IAM permission denied”.
3. Can I avoid using an assumed role and store the API key directly in the pod?
   You could, but it violates secret management best practices. Using Secrets Manager with proper IAM policies provides auditability, rotation, and least‑privilege access.
4. Is there a way to test the IAM permissions without running the full training job?
   Yes. Use aws iam simulate-principal-policy with the training role ARN and the actions secretsmanager:GetSecretValue and iam:PassRole on the target resources.
5. What if the training job runs on Fargate instead of EKS?
   The same principle applies: the task execution role must include secretsmanager:GetSecretValue and iam:PassRole. Adjust the task definition’s executionRoleArn accordingly.