milvus

Milvus IAM permission denied when creating collection during batch inference

by
woman leaning on white metal fence

Milvus IAM Permission Denied When Creating Collection During Batch Inference Problem During the model‑evaluation pipeline a batch inference job attempts to: Create a new Milvus collection to store embedding vectors. Insert the generated vectors. Run similarity searches for validation. All three operations fail with an IAM permission denied error. The pipeline aborts, leaving no evaluation … Read more

Milvus index rebuild failure in air‑gapped deployment missing binary dependencies

by
A red kite soars against a green forest

Problem In an air‑gapped deployment of Milvus (v2.3.x), attempts to rebuild a vector index on an existing collection fail with errors indicating missing binary dependencies and permission denials. The failure occurs during the milvusctl index rebuild operation and leaves the collection without a usable index. Typical Symptoms CLI returns non‑zero exit code: milvusctl index rebuild … Read more

Milvus ingestion fails with tokenizer encoding error in GitHub Actions

by
brown and black bird flying under white clouds during daytime

Problem: Milvus ingestion fails with tokenizer encoding error in GitHub Actions During automated integration tests run on GitHub Actions (or similar CI/CD runners), the data‑loading stage aborts with an exception originating from the tokenizer used to generate text embeddings. Typical log excerpts look like: UnicodeEncodeError: ‘utf-8’ codec can’t encode character ‘\ud83d’ in position 45 Traceback … Read more

Milvus vector index loading fails with intermittent DNS resolution timeouts

by
brown and black bird flying under white clouds during daytime

Milvus Vector Index Loading Fails with Intermittent DNS Resolution Timeouts Problem Statement In edge deployments where Milvus runs on resource‑constrained nodes (Raspberry Pi, Jetson, etc.), the vector index loading phase intermittently hangs. The underlying symptom is a DNS resolution failure for the Etcd and MinIO service endpoints, which manifests as: Log entries such as 2023/07/15 10:12:34 … Read more

Milvus node init fails due to port conflict in multi-region deployment

by
a bird of prey is flying in the air

Problem: Milvus node initialization fails due to port conflict in a multi‑region Kubernetes deployment When deploying Milvus across several AWS regions with Amazon EKS, the Init phase of the Milvus pods repeatedly aborts with errors such as: Error: listen tcp 0.0.0.0:19530: bind: address already in use grpc server start failed: failed to listen on port … Read more

Milvus cloud service returns 429 error during high query load

by
photo of brown hawk flying

Problem – Milvus Cloud Service Returns HTTP 429 Under High Query Load During a traffic spike the application receives responses such as: HTTP/1.1 429 Too Many Requests Content-Type: application/json { “code”: 429, “message”: “Rate limit exceeded”, “details”: “RPS limit of 1000 exceeded” } or the Milvus Python SDK throws: milvus.exceptions.ServerException: status=429, message=’Rate limit exceeded, please retry … Read more

Milvus ReplicaSet scaling failure during high concurrency benchmarking

by
A red kite soars against a green forest.

Problem – Milvus ReplicaSet Scaling Failure Under High‑Concurrency Benchmark During a distributed performance benchmark of Milvus 2.x, the team attempted to increase the replicaCount of the QueryNode (and optionally the Proxy) to meet a target query throughput of ~10 k QPS. The scaling operation consistently failed: New QueryNode pods remained in Pending or entered CrashLoopBackOff. Existing … Read more