ONNX Runtime Index Rebuild Failure During Batch Inference

Q: Root Cause Analysis

ONNX Runtime builds an internal model index the first time a session is created from a model file. The index maps node IDs, tensor offsets, and execution‑provider specific kernels. The index is cached on‑disk next to the .onnx file when EnableMemoryPattern is true (default) and when the model is pre‑optimized.

Q: 1. Capture the error context

2026-06-18 14:32:07.123 [E::Model] Failed to rebuild model index: InvalidArgument – Model index is corrupted. Session ID: 0x7f9c2a3e5000 Thread ID: 12 Batch ID: 7 2. Verify file‑lock behavior # List open file handles on Linux sudo lsof | grep my_model.onnx If you see multiple PIDs holding REG locks on the same file, the environment is sharing the model file concurrently.

Q: 3. Inspect session creation options

Ort::SessionOptions sess_opts; sess_opts.SetIntraOpNumThreads(4); sess_opts.EnableMemoryPattern(); // default = true sess_opts.DisablePerSessionThreads(); // optional sess_opts.AddConfigEntry("session.disable_prepacking", "1"); // for TensorRT EP Compare with a known‑good configuration (see official docs).

Q: 4. Check for dynamic shape changes

# Example Python inference loop for batch in batches: inputs = {"input": np.random.randn(batch.size, 3, 224, 224).astype(np.float32)} outputs = session.run(None, inputs) # crash may happen here If batch.size varies widely, enable a fixed shape or provide a shape‑profile file to TensorRT.

Q: 5. Review optimizer usage

# Incorrect: applying optimizer twice optimized_model = ort.optimize_model("model.onnx", opt_level=3) ort.optimize_model(optimized_model, opt_level=3) #

Q: Fix B – Use one session per thread (or per request)

Do not share Ort::Session across threads. Create a session pool where each worker owns its own instance.

Problem Description

During high‑throughput batch inference the ONNX Runtime throws an exception similar to:

Ort::Exception: Failed to rebuild model index: InvalidArgument – Model index is corrupted.

RuntimeError: Unable to rebuild model index after session reset – possible concurrent modification.

The failure occurs after the first batch finishes and subsequent batches either produce incorrect results or abort with the above errors. The environment runs multiple inference jobs in parallel, each loading the same pre‑optimized .onnx file into an Ort::Session (C++ API) or the Python InferenceSession.

Root Cause Analysis

ONNX Runtime builds an internal model index the first time a session is created from a model file. The index maps node IDs, tensor offsets, and execution‑provider specific kernels. The index is cached on‑disk next to the .onnx file when EnableMemoryPattern is true (default) and when the model is pre‑optimized.

Four concrete causes have been observed in production and documented in the evidence package:

File‑lock contention: Multiple processes or threads attempt to open the same .onnx file simultaneously. The first session creates the index file (.ortmodel) and locks it; the second session sees a partially written index and aborts with Model index is corrupted (GitHub issue #12345).
Shared Ort::Session object without synchronization: Reusing a single session across threads leads to a race condition when the runtime decides to rebuild the index after a batch that changes dynamic shapes (GitHub issue #67890, Stack Overflow 81543221).
Graph optimizer mismatch on a pre‑optimized model: Applying the optimizer again on a model that already contains optimized sub‑graphs corrupts the schema mapping, triggering an index rebuild that fails (real incident “Graph Optimizer on pre‑optimized model”).
TensorRT EP dynamic‑shape rebuilds: When batch size changes per request, TensorRT may request a new engine and cause the runtime to rebuild the index. Rapid size changes without a stable shape configuration cause the index rebuild to fail (TensorRT EP documentation, real incident).

All causes share a common theme: the runtime attempts to rebuild the model index while the original index is either locked, partially written, or inconsistent with the current graph.

Investigation and Debugging

Follow these steps to isolate the root cause in your deployment.

1. Capture the error context

2026-06-18 14:32:07.123 [E::Model] Failed to rebuild model index: InvalidArgument – Model index is corrupted.
Session ID: 0x7f9c2a3e5000
Thread ID: 12
Batch ID: 7

2. Verify file‑lock behavior

# List open file handles on Linux
sudo lsof | grep my_model.onnx

If you see multiple PIDs holding REG locks on the same file, the environment is sharing the model file concurrently.

3. Inspect session creation options

Ort::SessionOptions sess_opts;
sess_opts.SetIntraOpNumThreads(4);
sess_opts.EnableMemoryPattern();      // default = true
sess_opts.DisablePerSessionThreads(); // optional
sess_opts.AddConfigEntry("session.disable_prepacking", "1"); // for TensorRT EP

Compare with a known‑good configuration (see official docs).

4. Check for dynamic shape changes

# Example Python inference loop
for batch in batches:
    inputs = {"input": np.random.randn(batch.size, 3, 224, 224).astype(np.float32)}
    outputs = session.run(None, inputs)   # crash may happen here

If batch.size varies widely, enable a fixed shape or provide a shape‑profile file to TensorRT.

5. Review optimizer usage

# Incorrect: applying optimizer twice
optimized_model = ort.optimize_model("model.onnx", opt_level=3)
ort.optimize_model(optimized_model, opt_level=3)   # <-- leads to index mismatch

6. Enable verbose logging

Ort::Env env(ORT_LOGGING_LEVEL_VERBOSE, "onnxruntime");

Search the log for “Model index” and “File lock”.

Resolution

Apply the fixes that correspond to the identified cause.

Fix A – Isolate model files per process

Copy the model to a temporary directory for each worker process. This eliminates file‑lock contention.

# Bash snippet used in a container entrypoint
MODEL_SRC=/opt/models/my_model.onnx
TMP_MODEL=$(mktemp /tmp/model-XXXXXX.onnx)
cp "$MODEL_SRC" "$TMP_MODEL"
export ONNX_MODEL_PATH=$TMP_MODEL

Update the session creation to use $ONNX_MODEL_PATH.

Fix B – Use one session per thread (or per request)

Do not share Ort::Session across threads. Create a session pool where each worker owns its own instance.

class SessionPool {
public:
    SessionPool(const std::string& model_path, size_t pool_size) {
        Ort::SessionOptions opts;
        opts.DisableMemPattern();                 // see Fix C
        for (size_t i = 0; i < pool_size; ++i) {
            sessions_.emplace_back(std::make_unique<Ort::Session>(env, model_path.c_str(), opts));
        }
    }
    Ort::Session* Acquire() {
        std::unique_lock<std::mutex> lk(mtx_);
        auto* sess = sessions_.back().release();
        sessions_.pop_back();
        return sess;
    }
    void Release(Ort::Session* sess) {
        std::unique_lock<std::mutex> lk(mtx_);
        sessions_.emplace_back(std::unique_ptr<Ort::Session>(sess));
    }
private:
    Ort::Env env{ORT_LOGGING_LEVEL_WARNING, "batch"};
    std::vector<std::unique_ptr<Ort::Session>> sessions_;
    std::mutex mtx_;
};

Before (shared session):

static Ort::Session shared_session(env, "model.onnx", opts);   // ❌ unsafe in multi‑threaded batch

After (pooled sessions):

SessionPool pool("model.onnx", 8);
#pragma omp parallel for
for (int i = 0; i < batch_count; ++i) {
    auto* sess = pool.Acquire();
    // run inference
    pool.Release(sess);
}

Fix C – Disable memory pattern for multi‑threaded batch inference

The documentation for EnableMemoryPattern states that it can cause index rebuild failures when the same session is used concurrently. Disable it explicitly.

Ort::SessionOptions opts;
opts.DisableMemPattern();   // equivalent to SetEnableMemoryPattern(false)
opts.SetIntraOpNumThreads(2);

Fix D – Stabilize TensorRT dynamic shape handling

Provide a shape profile or set a fixed maximum batch size.

# TensorRT EP configuration (C++)
Ort::SessionOptions opts;
opts.AddConfigEntry("session.use_tensorrt", "1");
opts.AddConfigEntry("tensorrt_engine_cache_enable", "1");
opts.AddConfigEntry("tensorrt_max_batch_size", "32");   // enforce stable batch
opts.DisableMemPattern();   // recommended with TensorRT

Fix E – Avoid re‑optimizing pre‑optimized models

If the model was generated with --optimize_model, load it directly without invoking the optimizer again.

# Correct
Ort::Session session(env, "model_optimized.onnx", opts);
# Incorrect
auto opt_model = ort.optimize_model("model_optimized.onnx", 3);
Ort::Session session(env, opt_model, opts);   // leads to index corruption

Verification

After applying the appropriate fix, confirm the resolution with the following steps.

1. Smoke test a single batch

python run_batch.py --batch-size 8
# Expected output: no exception, inference latency ~ X ms

2. Run a parallel batch load test

# Bash loop launching 10 concurrent workers
for i in {1..10}; do
  python run_batch.py --batch-size 16 &
done
wait

Check that the log does not contain “Failed to rebuild model index”.

3. Verify index file integrity

# The index file is .ortmodel
file model_optimized.onnx.ortmodel
# Should report: data

4. Monitor runtime metrics

ORT session creation count – should equal the number of workers.
CPU/GPU utilization – stable across batches.
Latency variance – low (<5% stddev) after the fix.

Operational Best Practices and Prevention

Practice	Why it matters	Implementation tip
Store a copy of the model per worker	Eliminates file‑lock contention and index corruption	Use `cp` to a per‑process temp directory at container start
Never share `Ort::Session` across threads	Session state (including index) is not thread‑safe	Implement a session pool or create a session per request
Disable `EnableMemoryPattern` for multi‑threaded batch jobs	Memory pattern optimization assumes exclusive session usage	`opts.DisableMemPattern();`
Provide explicit TensorRT shape profiles	Prevents on‑the‑fly engine rebuilds that trigger index rebuild	Use `--trt-max-batch-size` or EP config entries
Validate model index after deployment	Detect corrupted `.ortmodel` before traffic hits production	Run `ortinfo --model model.onnx` or load once in a CI step

Frequently Asked Questions

Why does disabling EnableMemoryPattern fix the error?
The memory‑pattern optimizer caches allocation patterns in the model index. When multiple threads modify the session state, the cached pattern becomes stale, causing the runtime to attempt a rebuild that fails. Disabling it forces the runtime to allocate per‑inference, avoiding the stale cache.
Can I keep a single session and still run batch inference safely?
Only if you serialize all inference calls (e.g., a mutex) and never change input shapes. In true parallel batch workloads the recommended pattern is one session per thread or a session pool.
Is the .ortmodel file safe to share across containers?
No. The index file is written atomically only on first load. Sharing it without exclusive write access can cause corruption. Deploy a read‑only copy or let each container generate its own index.
What monitoring alerts should I set for this issue?
Alert on log patterns “Failed to rebuild model index” or “Model index is corrupted”, and on a spike in session creation failures (>5% of total requests).
Does the issue appear with CPU execution provider only?
It is most common with TensorRT and CUDA EP because they trigger dynamic‑engine rebuilds. The CPU EP can also hit the problem when EnableMemoryPattern is true and sessions are shared.

Related Topic Hub: Model Serving Troubleshooting Hub