Kubernetes Pod Stuck After TensorFlow Model Update

Q: Problem Description

A Kubernetes Deployment that runs tensorflow_model_server was updated to serve a new model version. After the rollout the pods entered a Running state but never became Ready. The following symptoms were observed:

Q: 1. Examine pod status and events

kubectl describe pod my-model-7c9f5d8f5b-9kz2l Key excerpts:

Q: 2. Inspect container logs

kubectl logs my-model-7c9f5d8f5b-9kz2l -c tensorflow-serving 3. Verify model directory inside the container kubectl exec -it my-model-7c9f5d8f5b-9kz2l -c tensorflow-serving -- ls -R /models/my_model Typical output for a broken deployment:

Q: 4. Check resource usage and OOM events

kubectl top pod my-model-7c9f5d8f5b-9kz2l And inspect the termination reason:

Q: 5. Validate image permissions

kubectl exec -it my-model-7c9f5d8f5b-9kz2l -c tensorflow-serving -- stat -c "%A %n" /usr/bin/tensorflow_model_server Expected output: -rwxr-xr-x /usr/bin/tensorflow_model_server. A missing execute bit triggers the permission denied error.

In this article

Table of Contents

Problem Description

A Kubernetes Deployment that runs tensorflow_model_server was updated to serve a new model version. After the rollout the pods entered a Running state but never became Ready. The following symptoms were observed:

Readiness probe failures:

Readiness probe failed: Get http://localhost:8501/v1/models/my_model/metadata: dial tcp 127.0.0.1:8501: connect: connection refused

Container logs show the server exiting immediately:

2026-07-30T10:12:45.123Z I tensorflow_serving/model_servers.cc:123] Starting TensorFlow ModelServer...
2026-07-30T10:12:45.124Z E tensorflow_serving/model_servers.cc:128] Failed to load model from /models/my_model/2: OSError: [Errno 2] No such file or directory: '/models/my_model/2/saved_model.pb'
2026-07-30T10:12:45.124Z I tensorflow_serving/model_servers.cc:135] Exiting.

kubectl reports CrashLoopBackOff after a few seconds:

$ kubectl get pod -l app=my-model -o wide
NAME                         READY   STATUS             RESTARTS   AGE
my-model-7c9f5d8f5b-9kz2l    0/1     CrashLoopBackOff   4          2m

In one incident the pod remained in Terminating for >5 minutes after a model bump because the container hit an OOM kill:
```
State:          Terminated
Reason:         OOMKilled
Exit Code:      137
```

Root Cause Analysis

The issue stems from a combination of TensorFlow Serving expectations and Kubernetes pod lifecycle handling:

Model path mismatch – TensorFlow Serving requires the model directory to contain a versioned sub‑directory (e.g., /models/my_model/2/) with a saved_model.pb. The updated CI pipeline copied only the saved_model.pb without the version folder, causing the server to exit with No such file or directory (see TensorFlow Serving documentation).
Readiness probe timing – The Deployment’s readiness probe hits /v1/models/.../metadata before the server has successfully loaded the model. When the server exits, the probe repeatedly fails, keeping the pod in Ready=0 (Kubernetes rolling update docs).
Resource limit violation – The new model size grew from ~500 MiB to ~2 GiB, exceeding the pod’s memory request/limit. The kernel OOM killer terminated the container, leaving the pod in a Terminating state (Kubernetes resource limits docs).
Image incompatibility – In a separate environment the TensorFlow Serving image was rebuilt with a newer glibc version. The binary /usr/bin/tensorflow_model_server lost executable permission, resulting in:
```
Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/usr/bin/tensorflow_model_server": permission denied
```
This matches the pattern reported in GitHub issue #102345.

Investigation and Debugging

The following step‑by‑step investigation reproduced the failure and isolated the root cause.

1. Examine pod status and events

kubectl describe pod my-model-7c9f5d8f5b-9kz2l

Key excerpts:

Events show FailedCreatePodSandBox only when the image was rebuilt with wrong permissions.
Readiness probe failure messages as shown above.

2. Inspect container logs

kubectl logs my-model-7c9f5d8f5b-9kz2l -c tensorflow-serving

3. Verify model directory inside the container

kubectl exec -it my-model-7c9f5d8f5b-9kz2l -c tensorflow-serving -- ls -R /models/my_model

Typical output for a broken deployment:

/models/my_model:
saved_model.pb
variables/

Correct layout should be:

/models/my_model:
2/
   saved_model.pb
   variables/

4. Check resource usage and OOM events

kubectl top pod my-model-7c9f5d8f5b-9kz2l

And inspect the termination reason:

kubectl get pod my-model-7c9f5d8f5b-9kz2l -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}'

5. Validate image permissions

kubectl exec -it my-model-7c9f5d8f5b-9kz2l -c tensorflow-serving -- stat -c "%A %n" /usr/bin/tensorflow_model_server

Expected output: -rwxr-xr-x /usr/bin/tensorflow_model_server. A missing execute bit triggers the permission denied error.

Resolution

Three independent fixes were applied depending on the observed symptom.

Fix A – Correct Model Directory Layout

Update the CI step that stages the model to create a versioned subdirectory.

# Before (broken)
cp -r model_dir/* /mnt/models/my_model/

# After (fixed)
VERSION=2
mkdir -p /mnt/models/my_model/${VERSION}
cp -r model_dir/* /mnt/models/my_model/${VERSION}/

Re‑apply the Deployment to trigger a new rollout:

kubectl rollout restart deployment/my-model

Fix B – Adjust Resource Requests/Limits

Increase memory limits to accommodate the larger model. Example patch:

# Before
resources:
  requests:
    cpu: "500m"
    memory: "512Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

# After
resources:
  requests:
    cpu: "500m"
    memory: "2Gi"
  limits:
    cpu: "2"
    memory: "4Gi"

Apply the change:

kubectl apply -f deployment.yaml

Fix C – Restore Executable Permission on the Server Binary

If the image is built from source, ensure the binary is installed with execute permissions:

# Dockerfile snippet (before)
COPY tensorflow_model_server /usr/bin/

# Dockerfile snippet (after)
COPY tensorflow_model_server /usr/bin/
RUN chmod +x /usr/bin/tensorflow_model_server

Re‑build and push the image, then update the Deployment image tag.

Validation

After applying the appropriate fix, perform the following checks:

Confirm pod reaches Ready:

kubectl get pod -l app=my-model -o wide
NAME                         READY   STATUS    RESTARTS   AGE
my-model-7c9f5d8f5b-9kz2l    1/1     Running   0          30s

Validate the readiness endpoint:

curl -s http://$(kubectl get pod -l app=my-model -o jsonpath='{.items[0].status.podIP}'):8501/v1/models/my_model/metadata | jq .

Check that no OOM events are recorded:

kubectl describe pod my-model-7c9f5d8f5b-9kz2l | grep -i OOMKilled || echo "No OOMKilled events"

Verify the model version is correctly reported:

curl -s http://localhost:8501/v1/models/my_model | jq .model_version_status

Prevention and Best Practices

Model versioning contract: Always publish models under a numeric version directory. Automate the creation of the version folder as part of the CI pipeline.
Resource sizing validation: Use a pre‑deployment script that parses the model size (e.g., du -sh) and fails the build if it exceeds a configurable threshold.
Readiness probe tuning: Add an initial initialDelaySeconds longer than the expected model load time, and use failureThreshold to avoid premature rollout failures.
Image reproducibility: Pin the base image and explicitly set file permissions in the Dockerfile to avoid silent permission regressions.
Observability: Export TensorFlow Serving metrics (e.g., tensorflow/serving/loaded_model_bytes) to Prometheus and set alerts for sudden drops in model_ready status.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

Why does the pod stay in Running but never become Ready after a model update?
Because the readiness probe contacts the TensorFlow Serving HTTP endpoint before the server successfully loads the new model. If the model path is incorrect or the server crashes, the probe keeps failing, leaving the pod in Ready=0.
How can I detect that a model version is missing the required saved_model.pb file?
Add a pre‑deployment validation step that runs test -f /models/$MODEL_NAME/$VERSION/saved_model.pb inside a temporary container. Fail the CI job if the test returns non‑zero.
What is the recommended way to avoid OOMKilled pods when model size grows?
Configure requests and limits based on the actual model footprint, and enable vertical-pod-autoscaler or HPA with custom metrics (e.g., tensorflow/serving/loaded_model_bytes).
Can I force a rolling update to wait for the model to be fully loaded?
Yes. Set spec.strategy.rollingUpdate.maxUnavailable to 0 and increase spec.minReadySeconds to a value larger than the model load time. This ensures the new pod must become Ready before the old pod is terminated.
Why did the pod get stuck in Terminating after the image was rebuilt?
The rebuilt image lacked execute permission on /usr/bin/tensorflow_model_server, causing the container runtime to fail with “permission denied”. Kubernetes then waited for the graceful termination period to expire, leaving the pod in Terminating until the timeout elapsed.