Data Infrastructure

Kafka topic creation fails with TopicExistsException after docker‑compose up

by
white concrete building under blue sky during daytime

Problem: Kafka topic creation fails with TopicExistsException after docker‑compose up During integration testing of an AI model inference pipeline, the test harness attempts to create a Kafka topic named inference-logs via the AdminClient. The call consistently returns an error similar to: org.apache.kafka.common.errors.TopicExistsException: Topic ‘inference-logs’ already exists. Despite the exception, the topic is not visible in … Read more

RAG decomposition service throws DecompositionFailedException on complex queries

by
Black cables and wires connected to the back of modular LED display panels

Problem: DecompositionFailedException in RAG Query Decomposition Service The RAG decomposition microservice consumes user queries from the rag.requests exchange, splits them into sub‑queries, and republishes each sub‑query to topic queues such as rag.subqueries.*. When a complex query (e.g., nested JSON, multi‑entity request, or very long prompt) is processed, the service throws DecompositionFailedException. The exception propagates, the … Read more

PostgreSQL SSL handshake failure after certificate expiration in Docker dev

by
Rows of black server racks with white logos in a data center

PostgreSQL SSL Handshake Failure After Certificate Expiration in Docker Development Problem Description During a local AI model training run, the Python script that connects to the PostgreSQL service via psycopg2 started throwing: psycopg2.OperationalError: SSL SYSCALL error: certificate verify failed PostgreSQL container logs showed the corresponding server‑side error: 2026-09-12 10:45:23.456 UTC [1] LOG: SSL error: certificate … Read more

LLM‑generated SQL function call syntax error on AWS RDS

by
Black cables and wires connected to the back of modular LED display panels

LLM‑generated SQL function call syntax error on AWS RDS Problem When an application uses a large language model (LLM) to generate SQL for invoking PostgreSQL stored procedures, the resulting statements often fail with errors such as: ERROR: syntax error at or near ‘(‘ ERROR: function my_schema.my_func(integer, text) does not exist ERROR: function my_func(integer, text) does … Read more

etcd state divergence after rolling upgrade leads to stale Redis config

by
a rack of servers in a server room

Problem Description During a rolling upgrade of an on‑premises Kubernetes‑managed etcd cluster (v3.5.0 → v3.5.7) the AI model‑serving stack began loading outdated model parameters. The model‑config microservice reads configuration from etcd, writes the current config_version to a Redis key (model:config), and caches the full parameter payload with a 12‑hour TTL. After the upgrade the following … Read more

Elasticsearch data node PVC fails with storage class not found in staging

by
a rack of servers in a server room

Problem – Elasticsearch data node PVC fails with “storage class not found” in staging In a staging Kubernetes cluster the Elasticsearch Helm release creates a StatefulSet for data nodes. The pods never start because their persistent volume claims remain in Pending with errors such as: persistentvolumeclaims “elasticsearch-data-0” not bound: storageclass.storage.k8s.io “standard” not found Consequences: Data‑node … Read more

PostgreSQL index rebuild lock timeout after Kubernetes pod restart

by
Rows of black server racks with white logos in a data center

Problem Description During rolling restarts or horizontal pod autoscaling of a PostgreSQL StatefulSet, repeated REINDEX CONCURRENTLY operations abort with lock timeouts or out‑of‑memory errors. The symptoms observed in production are: Log entries such as: 2024-07-12 14:03:21.123 UTC [12345] LOG: index build (concurrently) aborted: could not obtain exclusive lock on relation “orders_pkey” 2024-07-12 14:03:21.124 UTC [12345] … Read more

Inference API timeout connecting to PostgreSQL after firewall rule update

by
Server rack with blinking green lights

Problem Description The inference API service, built with FastAPI and using psycopg2/SQLAlchemy to query a PostgreSQL instance, started returning HTTP 504 timeouts after a recent firewall rule change. The API logs contain repeated connection errors such as: 2026-08-31 14:02:13,842 ERROR inference.api.handlers – could not connect to server: Connection timed out (0.00s) Is the server running … Read more

Canary Elasticsearch pods timing out after Calico network policy change

by
photo of computer cables

Problem – Canary Elasticsearch Pods Time Out After Calico NetworkPolicy Change During a canary rollout of a new Elasticsearch version, pods labeled app=es-canary were unable to reach the primary elasticsearch-master service. All HTTP requests to port 9200 and transport requests to port 9300 resulted in connection‑timeout errors, causing indexing and query failures and halting the … Read more

Intermittent Redis Pub/Sub streaming interruptions in AI inference staging

by
a rack of servers in a server room

Problem: Intermittent Redis Pub/Sub Streaming Interruptions in AI Inference Staging The AI inference service streams tokenized responses to clients via a Redis Pub/Sub channel. In the staging environment the stream is occasionally truncated, causing the client to receive incomplete answers or a sudden “connection closed” error. The symptom manifests as: Log entry: socket.timeout: timed out … Read more