Compute Engine index rebuild failure during nightly batch on AI service

Problem

During the nightly batch window a search index rebuild executed by the AI model‑serving workload on Google Compute Engine (GCE) VMs fails repeatedly. The failure prevents the latest documents from being searchable until the next successful run, causing stale results for downstream applications.

Typical symptoms observed on the affected instances:

• Batch job exits with non‑zero status (exit code 1).
• Log entry: ERROR 2024-10-19T02:15:42Z index_rebuilder.go:112: index rebuild failed: transaction aborted
• Search service health endpoint returns 503 Service Unavailable after the batch completes.
• CPU usage spikes to 90 % during the rebuild, then drops abruptly.

Root Cause

The index rebuild process runs inside a Docker container that writes to a /mnt/data/index directory mounted from a regional persistent disk. The nightly batch runs concurrently with the autoscaling controller, which may start new VM instances while the rebuild is still in progress. When a new instance is added, the autoscaler triggers a gcloud compute instances stop on the older VM to rebalance the instance group. The stop operation sends a SIGTERM to the container runtime, which aborts the ongoing transaction in the index library (a RocksDB‑based store). Because the index library does not automatically roll back incomplete writes, the on‑disk state becomes corrupted, and the next rebuild attempt aborts with the “transaction aborted” error.

Key factors:

• Shared persistent disk is attached to a single VM at a time; stop‑and‑start cycles break the exclusive lock.
• Autoscaling policies (CPU‑based target 70 %) cause scale‑in events during the batch window.
• The index library’s default write‑ahead log (WAL) directory is on the same disk, lacking a separate durable journal.

Debug

Step‑by‑step investigation that reproduced the failure:

1. Inspect instance‑group activity during the batch window.

gcloud compute instance-groups managed list-instances my-igm \
    --region=us-central1 \
    --filter="status:RUNNING" \
    --format="table(name, currentAction, status)"

Output showed a STOPPING action on ai-worker-01 at 02:15:40 UTC, exactly when the index rebuild log entry appeared.

2. Check container logs for termination signals.

docker logs ai-index-rebuilder --since "2024-10-19T02:10:00Z"

Log excerpt:

2024-10-19T02:15:41Z INFO  index_rebuilder.go:98: received SIGTERM, shutting down
2024-10-19T02:15:42Z ERROR index_rebuilder.go:112: index rebuild failed: transaction aborted

3. Verify disk attachment state.

lsblk -f | grep /dev/sdb
# /dev/sdb1 ext4 rw,relatime,data=ordered /mnt/data

After the VM stopped, /dev/sdb1 was detached, confirming the disk was no longer available to the rebuild process.

4. Reproduce the race condition in a test environment.

Created a minimal script that starts the rebuild and, after a 30‑second delay, issues a gcloud compute instances stop on the same VM. The script consistently produced the same “transaction aborted” error.

Solution

The fix consists of two coordinated changes:

1. Make the index rebuild resilient to host termination

Configure the index library to use a separate journal disk for its write‑ahead log and enable automatic rollback on incomplete transactions.

Before:

# index_config.yaml
wal_path: "/mnt/data/index/wal"
auto_rollback: false

After:

# index_config.yaml
wal_path: "/mnt/journal/wal"
auto_rollback: true
journal_disk: "projects/my-project/zones/us-central1-a/disks/journal-disk"

Deploy the updated configuration via the CI pipeline and restart the container.

2. Prevent autoscaler‑induced scale‑in during the batch window

Adjust the managed instance group (MIG) autoscaling policy to disable scale‑in between 02:00 UTC and 04:00 UTC.

gcloud compute instance-groups managed set-autoscaling my-igm \
    --region=us-central1 \
    --min-num-replicas=3 \
    --max-num-replicas=10 \
    --target-cpu-utilization=0.70 \
    --cool-down-period=300 \
    --custom-metric-utilization=metric=custom.googleapis.com/scale_in_block,utilization-target=0,utilization-target-type=GAUGE \
    --autoscaling-policy-schedule="02:00-04:00=DISABLE_SCALE_IN"

Alternatively, add a scaleInControl block to the MIG’s autoscaling configuration to enforce a maxScaleInRate of 0 during the batch window.

3. Update the batch orchestration script to wait for a stable instance set

Insert a pre‑flight check that verifies no STOPPING actions are pending.

#!/usr/bin/env bash
# wait_for_stable_instances.sh
MAX_RETRIES=10
INTERVAL=30
for i in $(seq 1 $MAX_RETRIES); do
  pending=$(gcloud compute instance-groups managed list-instances my-igm \
    --region=us-central1 \
    --filter="currentAction:STOPPING" \
    --format="value(name)" | wc -l)
  if [[ $pending -eq 0 ]]; then
    echo "Instance group stable, proceeding with index rebuild."
    exit 0
  fi
  echo "Waiting for $pending instance(s) to stop..."
  sleep $INTERVAL
done
echo "Timeout waiting for stable instance group."
exit 1

Invoke this script at the start of the nightly job; abort the job if the check fails.

Verify

After applying the changes, perform the following validation steps:

1. Trigger a manual rebuild during the protected window.
2. Confirm the job exits with status 0 and logs contain:

2024-10-19T02:45:12Z INFO index_rebuilder.go:150: index rebuild completed successfully

3. Check that the search service health endpoint returns 200 OK and that newly indexed documents appear in query results.

curl -s -o /dev/null -w "%{http_code}" http://ai-service.internal/health
200

4. Verify no STOPPING actions were recorded during the run:

gcloud compute instance-groups managed list-instances my-igm \
    --region=us-central1 \
    --filter="creationTimestamp>2024-10-19T02:00:00Z" \
    --format="table(name, currentAction)"

All rows should show RUNNING or NONE.

Prevent

Operational safeguards to avoid recurrence:

• Scheduled autoscaling windows: Use autoscalingPolicy.scheduledScalingPolicy to lock scale‑in during any maintenance or batch jobs.
• Separate WAL storage: Keep write‑ahead logs on a dedicated, high‑IOPS SSD that is not detached during VM lifecycle events.
• Health‑check‑driven graceful shutdown: Configure the instance group’s instanceTerminationAction to send a SIGTERM to the container and wait for a shutdown script to complete before detaching disks.
• Monitoring alerts: Create a Cloud Monitoring alert on the metric custom.googleapis.com/index_rebuild_failure with a threshold of >0 within a 5‑minute window.
• Automated rollback testing: Include a nightly smoke test that intentionally aborts a rebuild (e.g., by sending SIGTERM) and verifies the index library rolls back cleanly.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

1. Why does the index rebuild succeed when I run it manually outside the batch window?

   Manual runs typically occur on a stable instance set with no concurrent scale‑in actions, so the persistent disk remains attached for the entire duration of the transaction.

2. Can I keep the original autoscaling policy and just add a grace period?

   Yes. Adding a scaleInControl.maxScaleInRate of 0 for the specific time window achieves the same effect without altering min/max replica counts.

3. Is a separate journal disk mandatory?

   It is not strictly mandatory, but it eliminates the single point of failure where disk detachment corrupts the WAL. If you cannot provision a separate disk, enable the index library’s auto_rollback and ensure the WAL directory is on a resilient filesystem (e.g., tmpfs) that survives VM termination.

4. What if the batch job must run longer than the autoscaler’s cool‑down period?

   Increase the cool‑down period in the autoscaling policy or extend the scheduled “no‑scale‑in” window to cover the full expected runtime plus a safety margin.

5. How can I detect a corrupted index after a failed rebuild?

   Run the library’s integrity check command (e.g., indexctl verify --path /mnt/data/index). A non‑zero exit code indicates corruption, prompting a full re‑index from source data.