DeepSeek Collection Creation Fails With Runtime Exception After Rolling Deployment

Q: 2. Apply the migration before traffic switch

Modify the Helm chart to run a preUpgrade hook that executes Flyway against the DB:

Q: 3. Increase HikariCP pool size and enable graceful shutdown

Update application.yml (or ConfigMap) with the following settings:

Q: 4. Redeploy with zero‑downtime strategy

# Apply the updated Helm chart helm upgrade deepseek ./chart \ --set image.tag=1.4.2 \ --set resources.limits.cpu=500m \ --set resources.limits.memory=1Gi The pre‑upgrade hook runs the migration, then the rolling update proceeds with sufficient DB connections, eliminating the runtime exception.

In this article

Table of Contents

Problem Description

During a rolling deployment of DeepSeek on a 5‑node GKE cluster, API calls to /v1/collections started failing with a runtime exception. The failure manifested as:

java.lang.RuntimeException: Failed to write collection to PostgreSQL
    at com.deepseek.service.CollectionService.create(CollectionService.java:112)
Caused by: org.postgresql.util.PSQLException: Connection is closed
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:247)

Additional logs showed HikariCP pool timeouts:

HikariPool-1 - Connection is not available, request timed out after 30000ms.

Impact:

New data collections could not be created, breaking ingestion pipelines.
Existing collections remained readable, so read‑only traffic was unaffected.
During the incident, the error rate spiked to 85 % for collection‑create requests.

Root Cause Analysis

Three inter‑related factors triggered the exception:

Missing Flyway migration for the collections table. The deployment introduced a new metadata column (see Collections API reference), but the migration script for version V12__add_metadata_to_collections.sql was not packaged in the Docker image. Without the column, PostgreSQL returned PSQLException: column "metadata" does not exist, which was wrapped as a generic RuntimeException by the service layer.
Connection‑pool exhaustion during the rolling update. The DeepSeek deployment uses HikariCP with a default max pool size of 10 (see PostgreSQL Backend Configuration). When the rolling restart replaced pods, each new pod attempted to warm its pool while the old pods were still processing traffic, temporarily exceeding the total allowed connections on the PostgreSQL instance. This produced the Connection is closed and timeout messages observed in the logs.
Zero‑downtime deployment pattern without a pre‑flight migration step. The blue‑green strategy in the Kubernetes integration guide assumes that schema migrations are applied before traffic is switched. In this rollout, the migration step was omitted, so the new pods hit the stale schema immediately.

Investigation and Debugging Steps

The following checklist reproduces the diagnostic path used in the incident:

Collect recent pod logs.
```
kubectl logs -l app=deepseek -c deepseek -n production --since=5m
```
Look for the stack trace shown above and any HikariCP warnings.

Verify database schema version.

kubectl exec -it $(kubectl get pod -l app=deepseek-db -n production -o jsonpath="{.items[0].metadata.name}") -n production -- psql -U deepseek -d deepseek -c "SELECT version FROM flyway_schema_history ORDER BY installed_rank DESC LIMIT 1;"

If the latest version is V11 while the application expects V12, a migration is missing.

Check HikariCP pool metrics. Use Prometheus query:
```
hikari_pool_connections_acquired_total{job="deepseek"} - hikari_pool_connections_idle{job="deepseek"}
```
A sudden drop to zero idle connections during the rollout confirms exhaustion.
Reproduce the failure locally. Spin up a single‑node Kubernetes sandbox with the same image tag and run:
```
curl -X POST https://deepseek.example.com/v1/collections \
  -H "Content-Type: application/json" \
  -d '{"name":"test","metadata":{}}' -k -v
```
The response should contain the same RuntimeException if the migration is absent.
Inspect the Docker image for migration scripts.
```
docker run --rm deepseek:latest ls /app/migrations
```
If V12__add_metadata_to_collections.sql is missing, the image is incomplete.

Resolution

The fix required two coordinated actions: applying the missing migration and adjusting the connection‑pool configuration to survive rolling updates.

1. Add the missing Flyway migration

Before rebuilding the image, create the migration script:

-- V12__add_metadata_to_collections.sql
ALTER TABLE collections ADD COLUMN metadata JSONB NOT NULL DEFAULT '{}'::jsonb;

Update pom.xml (or Gradle build) to include the migrations directory in the final JAR, then rebuild:

# Build the new image
docker build -t deepseek:1.4.2 .
docker push deepseek:1.4.2

2. Apply the migration before traffic switch

Modify the Helm chart to run a preUpgrade hook that executes Flyway against the DB:

apiVersion: batch/v1
kind: Job
metadata:
  name: deepseek-db-migrate
  annotations:
    "helm.sh/hook": pre-upgrade
spec:
  template:
    spec:
      containers:
      - name: flyway
        image: deepseek:1.4.2
        command: ["java", "-jar", "flyway.jar", "migrate"]
        envFrom:
        - secretRef:
            name: deepseek-db-secret
      restartPolicy: OnFailure

3. Increase HikariCP pool size and enable graceful shutdown

Update application.yml (or ConfigMap) with the following settings:

Parameter	Before	After
maximumPoolSize	10	25
connectionTimeout	30000ms	60000ms
idleTimeout	600000ms	300000ms
maxLifetime	1800000ms	1800000ms

Also enable spring.lifecycle.timeout-per-shutdown-phase=60s so pods release connections before termination.

4. Redeploy with zero‑downtime strategy

# Apply the updated Helm chart
helm upgrade deepseek ./chart \
  --set image.tag=1.4.2 \
  --set resources.limits.cpu=500m \
  --set resources.limits.memory=1Gi

The pre‑upgrade hook runs the migration, then the rolling update proceeds with sufficient DB connections, eliminating the runtime exception.

Validation

After the rollout, perform the following checks:

Confirm migration applied.
```
psql -U deepseek -d deepseek -c "\d+ collections"
```
Output should list the metadata column.
Verify pool health. In Prometheus:
```
hikari_pool_connections_idle{job="deepseek"} > 5
```
Idle connections should stay above the threshold.

Test collection creation.

curl -X POST https://deepseek.example.com/v1/collections \
  -H "Content-Type: application/json" \
  -d '{"name":"prod‑test","metadata":{"source":"ci"}}' -k -v

Response should be 201 Created with a JSON body containing the new collection ID.

Run a health‑check endpoint.
```
curl -s http://deepseek-service/healthz | jq .
```
All checks should report "status":"UP".

Prevention and Best Practices

Enforce migration continuity. Integrate Flyway as a mandatory Helm hook (pre‑upgrade) and fail the release if migrations are pending.
Monitor connection‑pool metrics. Alert when hikari_pool_connections_idle falls below 20 % of maximumPoolSize for more than 30 seconds.
Version‑lock migration scripts. Store migration files in a separate Git submodule to avoid accidental omission from the build artifact.
Graceful pod termination. Set terminationGracePeriodSeconds to at least 60 seconds and configure Spring Boot’s shutdown timeout to release DB connections before SIGTERM.
Load‑test rolling updates. Use a tool like k6 to simulate collection‑create traffic during a staged rollout and verify that the system maintains p99 latency < 200 ms.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the error appear only after a rolling deployment? The new pods expect the latest schema (including the metadata column). If the migration hasn’t run, the first request that touches the table triggers a PSQLException, which is wrapped as a RuntimeException. Existing pods continue using the old schema until they are terminated.
Can I disable the pre‑upgrade migration hook? Disabling the hook is unsafe; it removes the guarantee that the database schema matches the application version. If you must skip it, you must manually run Flyway against the DB before upgrading.
How do I determine the appropriate HikariCP pool size for my workload? Start with max(2 * number_of_app_pods, 10) and adjust based on observed maxConnections in PostgreSQL (SHOW max_connections;) and the average concurrent request rate. Keep the total connections across all pods below the DB limit.
What if the migration fails during the hook? Helm will roll back the release automatically. Inspect the Job logs:
```
kubectl logs job/deepseek-db-migrate -n production
```
Fix the migration script and re‑run the upgrade.
Is there a way to make collection creation idempotent during deployments? Yes. Use a client‑generated UUID for the collection ID and handle duplicate key errors gracefully by treating them as a successful “already exists” case. This mitigates race conditions when multiple pods attempt to create the same collection concurrently.