Jordan Lee

RAG pipeline dimension error after updating embedding model in Docker Compose

by
A computer monitor sitting on top of a desk next to a plant

Problem The Retrieval‑Augmented Generation (RAG) service, deployed with Docker Compose, fails during inference with a dimension mismatch error. The query encoder produces vectors of a different size than the document encoder, causing the retrieval component to raise an exception and abort the request. 2026-10-31 14:12:07,842 ERROR rag_service.inference – RuntimeError: Expected embedding dimension 768 but got … Read more

MLflow tracking API unreachable during batch job execution

by
Black cables and wires connected to the back of modular LED display panels

Problem – MLflow Tracking API Unreachable During Batch Job Execution Batch jobs that run model training or data preprocessing attempt to log parameters, metrics, and artifacts to an MLflow tracking server. In production the jobs fail with errors such as: 2026-10-30 14:12:03,721 ERROR mlflow.tracking.client: Failed to log metric: Traceback (most recent call last): File “/usr/local/lib/python3.11/site-packages/mlflow/tracking/client.py”, … Read more

MLflow tracking 400 Bad Request after JSON schema validation failure

by
a rack of servers in a server room

MLflow Tracking — 400 Bad Request After JSON Schema Validation Failure Problem When logging experiment metadata (e.g., mlflow.log_param, mlflow.log_metric, or mlflow.set_tag) to a managed MLflow tracking endpoint (Azure ML, AWS SageMaker, GCP AI Platform), the client receives: HTTP/1.1 400 Bad Request Content-Type: application/json { “error_code”: “INVALID_PARAMETER_VALUE”, “message”: “JSON schema validation failed: request payload does not conform to the expected … Read more

etcd cluster inconsistency causing Elasticsearch index metadata divergence in staging

by
Black cables and wires connected to the back of modular LED display panels

Problem In the staging environment the Elasticsearch service reports index_not_found_exception and vector‑search queries return empty results. Simultaneously the Elasticsearch logs contain messages such as: [2024-10-15T14:22:31.842Z] [WARN ][o.e.b.IndexMetaData] [es-node-1] failed to load index metadata for index [products] from cluster state: index not found [2024-10-15T14:22:32.017Z] [ERROR][o.e.c.s.MasterService] [es-node-2] master not discovered, aborting request Investigation revealed that the underlying … Read more

Weaviate Docker image pull failure on GitLab Runner during CI

by
photo of computer cables

Problem A GitLab CI job that sets up integration tests fails during the docker pull step for the semitechnologies/weaviate image. The runner aborts with a non‑zero exit code, preventing the pipeline from progressing. Typical CI log excerpt: Running with gitlab-runner 16.2.0 (c8b1c8c5) on docker-auto-scale 1234567890abcdef Preparing the “docker” executor Using Docker image sha256:7c3b… for build … Read more

Container crash loop on AMD GPU during preprocessing stage

by
black and green digital device

Problem A container that runs the data‑pipeline preprocessing and model‑inference stages crashes repeatedly when the node is equipped with an AMD GPU. The pod enters a CrashLoopBackOff after the preprocessing step completes, preventing the downstream inference from ever starting. Symptoms and Impact Pod status: CrashLoopBackOff after 0/1 containers ready. Container logs end with a stack … Read more

Pod placement failure for PyTorch job on kind after node selector change

by
a purple and black background with a purple and black logo

Problem: Pod placement failure for a PyTorch job on a kind cluster after changing the node selector When launching a distributed PyTorch training job on a local kind (Kubernetes IN Docker) cluster, the scheduler repeatedly reports that it cannot place the training pods. The job remains in Pending state even though the single node appears … Read more

ChromaDB connection timeout during RAG inference serving

by
Black cables and wires connected to the back of modular LED display panels

Problem – ChromaDB Connection Timeout During RAG Inference Serving When deploying a Retrieval‑Augmented Generation (RAG) service, the inference container repeatedly fails to retrieve documents from the ChromaDB vector store. The request handler logs a ConnectionTimeoutError after the client’s requests call exceeds the configured 5 second deadline, and the service returns a 504 response to the downstream … Read more

LangChain RAG metadata filter dropping relevant chunks after config tweak

by
a rack of servers in a server room

Problem After adjusting the metadata_filter parameters in a LangChain Retrieval‑Augmented Generation (RAG) pipeline, the system began returning dramatically fewer relevant document chunks. Queries that previously produced multiple supporting passages now return only one or none, leading to low recall and incomplete answers. Root Cause The metadata filter is applied during the Retriever stage to prune … Read more

DeepSeek token refresh failure during rolling update of inference pods

by
A person holding a cell phone in their hand

Problem – Token Refresh Failure During Rolling Update of DeepSeek Inference Pods During a rolling update of the DeepSeek inference service in a Kubernetes cluster, newly scheduled pods attempt to obtain a fresh OAuth2 access token from the internal token‑issuer service. The request fails, resulting in authentication errors (HTTP 401) and dropped inference requests. The issue … Read more