Milvus IAM Permission Denied When Creating Collection During Batch Inference

Q: Solution

Update the IAM policy for eval-pipeline-sa to include the required actions and adjust the resource wildcard to match the evaluation naming convention.

In this article

Table of Contents

Milvus IAM Permission Denied When Creating Collection During Batch Inference

Problem

During the model‑evaluation pipeline a batch inference job attempts to:

Create a new Milvus collection to store embedding vectors.
Insert the generated vectors.
Run similarity searches for validation.

All three operations fail with an IAM permission denied error. The pipeline aborts, leaving no evaluation results.

Typical error messages (captured from the job logs):


2026-10-14 09:42:07,123 - ERROR - milvus_client - PermissionDenied: IAM permission denied for action CreateCollection on resource milvus:myproject:collections:test_eval_20231014
2026-10-14 09:42:07,128 - ERROR - milvus_client - PermissionDenied: IAM permission denied for action Insert on resource milvus:myproject:collections:test_eval_20231014
2026-10-14 09:42:07,132 - ERROR - milvus_client - PermissionDenied: IAM permission denied for action Search on resource milvus:myproject:collections:test_eval_20231014

The failure occurs only in the evaluation environment; the same code runs fine in development and staging.

Root Cause

The evaluation environment uses a dedicated service account (eval-pipeline-sa) that is bound to a restrictive IAM policy. The policy grants read‑only access to existing collections but does not include the milvus.collections.create, milvus.vectors.insert, or milvus.vectors.search actions. Because Milvus enforces IAM checks at the API gateway, any attempt to invoke those actions is rejected before the request reaches the server.

Two subtle factors contributed:

Policy drift: The original policy was copied from the staging environment where collections are pre‑provisioned, so the CreateCollection permission was never required.
Resource naming: The evaluation job creates collections with a timestamp suffix (e.g., test_eval_20231014). The IAM policy used a wildcard pattern milvus:myproject:collections/*_staging, which does not match the evaluation naming scheme, causing the wildcard mismatch to deny the request.

Debug

Step‑by‑step investigation that reproduced the issue and identified the missing permissions.

Confirm the service account identity used by the job:


$ curl -s http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email \
  -H "Metadata-Flavor: Google"
eval-pipeline-sa@myproject.iam.gserviceaccount.com

Inspect the attached IAM policy:


$ gcloud iam service-accounts get-iam-policy eval-pipeline-sa@myproject.iam.gserviceaccount.com \
  --format=json
{
  "bindings": [
    {
      "role": "roles/milvus.viewer",
      "members": [
        "serviceAccount:eval-pipeline-sa@myproject.iam.gserviceaccount.com"
      ]
    }
  ]
}

The roles/milvus.viewer role only includes milvus.collections.get and milvus.vectors.search on existing collections.

Attempt a minimal API call with explicit IAM test using the Milvus Python SDK:


from pymilvus import MilvusClient, CollectionSchema, FieldSchema, DataType

client = MilvusClient(uri="https://milvus.myproject.io", token="...")
try:
    client.create_collection(
        collection_name="debug_test",
        schema=CollectionSchema(
            fields=[
                FieldSchema(name="id", dtype=DataType.INT64, is_primary=True),
                FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=128)
            ]
        )
    )
except Exception as e:
    print(e)

Output:


PermissionDenied: IAM permission denied for action CreateCollection on resource milvus:myproject:collections:debug_test

Check the wildcard pattern in the existing policy (if any):


$ gcloud projects get-iam-policy myproject \
  --flatten="bindings[].members" \
  --format="table(bindings.role, bindings.members)" \
  | grep milvus
roles/milvus.viewer   serviceAccount:eval-pipeline-sa@myproject.iam.gserviceaccount.com

No custom condition or resource name pattern is present.

Solution

Update the IAM policy for eval-pipeline-sa to include the required actions and adjust the resource wildcard to match the evaluation naming convention.

Before (current policy)


{
  "bindings": [
    {
      "role": "roles/milvus.viewer",
      "members": [
        "serviceAccount:eval-pipeline-sa@myproject.iam.gserviceaccount.com"
      ]
    }
  ]
}

After (updated policy)


{
  "bindings": [
    {
      "role": "roles/milvus.viewer",
      "members": [
        "serviceAccount:eval-pipeline-sa@myproject.iam.gserviceaccount.com"
      ]
    },
    {
      "role": "roles/milvus.editor",
      "members": [
        "serviceAccount:eval-pipeline-sa@myproject.iam.gserviceaccount.com"
      ],
      "condition": {
        "title": "EvalCollectionWildcard",
        "description": "Allow collection creation for evaluation jobs",
        "expression": "resource.name.startsWith('milvus:myproject:collections:test_eval_')"
      }
    }
  ]
}

Explanation of the fix:

roles/milvus.editor grants CreateCollection, Insert, Search, and DeleteCollection actions.
The condition limits the permission to collections whose names start with test_eval_, preventing accidental privilege escalation.

Apply the policy with:


$ cat updated-policy.json
# (contents as shown above)

$ gcloud iam service-accounts set-iam-policy \
    eval-pipeline-sa@myproject.iam.gserviceaccount.com \
    updated-policy.json

Code change (optional)

If the pipeline previously hard‑coded the collection name, add a prefix constant to keep it aligned with the IAM condition.


# Before
COLLECTION_NAME = f"eval_{run_id}"

# After
EVAL_PREFIX = "test_eval_"
COLLECTION_NAME = f"{EVAL_PREFIX}{run_id}"

Verify

Re‑run the batch inference job and confirm successful creation, insertion, and search.

Check the job logs for the absence of PermissionDenied messages.
Validate the collection exists via the Milvus CLI:


$ milvus-cli list_collections
test_eval_20231014

Run a quick insert‑and‑search sanity test:


from pymilvus import MilvusClient, DataType

client = MilvusClient(uri="https://milvus.myproject.io", token="...")
client.insert(
    collection_name="test_eval_20231014",
    data=[
        [1, [0.1]*128],
        [2, [0.2]*128]
    ]
)
results = client.search(
    collection_name="test_eval_20231014",
    data=[[0.15]*128],
    limit=2
)
print(results)

Expected output (truncated):


[{'id': 1, 'distance': 0.0012}, {'id': 2, 'distance': 0.0045}]

Prevent

Implement guardrails to avoid similar IAM mismatches in future environments.

Practice	Why it helps
Define a dedicated IAM role for evaluation pipelines (e.g., `roles/milvus.eval`)	Encapsulates required actions and can be audited separately.
Use resource‑name conditions instead of broad wildcards	Limits the scope of permissions to predictable naming patterns.
Automate policy validation in CI/CD	Run `gcloud iam policies lint` against a test service account before promotion.
Version‑control IAM policies as JSON/YAML	Ensures changes are reviewed and traceable.
Tag collections with a label (e.g., `environment=eval`) and enforce label‑based IAM	Provides an additional layer of isolation beyond naming.

FAQ

Why does the same code work in staging but not in evaluation?
Staging uses a service account with the roles/milvus.editor role, which includes collection‑creation rights. Evaluation uses a more restrictive viewer role, so the missing actions surface only there.
Can I grant milvus.editor globally instead of using a condition?
Technically yes, but it violates the principle of least privilege. A conditional binding restricts access to the specific collection naming pattern used by the evaluation pipeline, reducing risk of accidental data modification.
What if my pipeline creates collections with random UUIDs instead of a fixed prefix?
Add a label‑based condition to the IAM binding, e.g., resource.labels.environment == 'eval', and ensure the pipeline tags each collection accordingly.
Do I need to restart the pipeline after updating the IAM policy?
No. The Milvus API gateway re‑evaluates IAM on each request, so the next API call will see the updated permissions immediately.
How can I audit which service accounts performed collection creation?
Enable Milvus audit logs (or Cloud Audit Logs if using a managed service) and filter for methodName="CreateCollection" and principalEmail fields.

Related Topic Hub: Vector Databases Troubleshooting Hub