Mistral training loop runtime error: temperature value out of 0‑1 range

Problem – Runtime error: temperature value out of 0‑1 range

During a fine‑tuning run of a Mistral model the training script aborts after a few epochs with an exception similar to:

ValueError: temperature must be in the range [0.0, 1.0], got 1.3

Typical symptoms observed in the logs:

• Trainer initialization succeeds, then validate_sampling_params raises.
• Training stops before the first checkpoint is written.
• CI pipelines report “Training loop failed: temperature out of allowed range”.
• In production pipelines the crash appears after the 3rd epoch when a dynamically computed temperature exceeds 1.0.

Root Cause Analysis

The Mistral SDK enforces the sampling hyper‑parameter contract defined in the official Sampling Configuration documentation: temperature must be a float between 0.0 and 1.0 inclusive. Starting with v2.3.0, the Trainer class performs an explicit validation in mistral.trainer.validate_sampling_params. If the value is outside the allowed range a ValueError (or RuntimeError in older releases) is raised, causing the entire training loop to terminate.

Common ways the out‑of‑range value is introduced:

Source	Typical Value	Why it exceeds 1.0
Static config file	temperature: 1.2	Copied from a previous experiment without sanitisation.
Dynamic schedule (e.g., temperature = lr * scale)	temperature: 1.05	Learning‑rate scaling logic not clamped.
Restored checkpoint	temperature: 1.05	Older checkpoint stored a value that was legal in pre‑v2.3 releases.
Parameter sweep	temperature: 2.0	Search space defined without upper bound.

In all cases the underlying assumption that “temperature is always within [0,1]” is violated, and the validation layer rightfully aborts execution.

Investigation and Debugging

1. Reproduce the failure locally

python train.py --config configs/ft.yaml

Typical stack trace (excerpt):

Traceback (most recent call last):
  File "train.py", line 42, in <module>
    trainer = Trainer(**cfg)
  File ".../mistral/trainer.py", line 215, in __init__
    self.validate_sampling_params()
  File ".../mistral/trainer.py", line 237, in validate_sampling_params
    raise ValueError(f"temperature must be in the range [0.0, 1.0], got {self.temperature}")
ValueError: temperature must be in the range [0.0, 1.0], got 1.3

2. Inspect the configuration source

# configs/ft.yaml
sampling:
  temperature: 1.3   # ← suspect value
  top_k: 50
  top_p: 0.95

3. Verify runtime values before validation

import logging
logging.basicConfig(level=logging.DEBUG)

def debug_temperature(cfg):
    logging.debug("Effective temperature: %s", cfg.sampling.temperature)

debug_temperature(cfg)

Output shows the exact float that triggers the error.

4. Check for dynamic overrides

# In training loop
if args.dynamic_temp:
    cfg.sampling.temperature = args.base_temp * epoch / max_epochs
    # No clamping → may exceed 1.0

5. Review checkpoint metadata

python -c "import torch; ckpt = torch.load('ckpt_last.pt'); print(ckpt['sampling']['temperature'])"
# Output: 1.05

Resolution – Bringing temperature back into the allowed range

Static configuration fix

Update the YAML (or JSON) file so that temperature respects the 0‑1 bound.

# Before (invalid)
sampling:
  temperature: 1.3
  top_k: 50
  top_p: 0.95

# After (valid)
sampling:
  temperature: 0.9   # choose a value appropriate for your generation style
  top_k: 50
  top_p: 0.95

Dynamic computation safeguard

def compute_temperature(base, epoch, max_epochs):
    raw = base * epoch / max_epochs
    # Clamp to the legal interval
    return max(0.0, min(1.0, raw))

# Usage
cfg.sampling.temperature = compute_temperature(args.base_temp, epoch, max_epochs)

Checkpoint migration script

import torch

def migrate_ckpt(path):
    ckpt = torch.load(path)
    temp = ckpt.get('sampling', {}).get('temperature', 0.7)
    if temp > 1.0:
        ckpt['sampling']['temperature'] = 1.0
        torch.save(ckpt, path)
        print(f"Clamped temperature from {temp} to 1.0 in {path}")

migrate_ckpt('ckpt_last.pt')

Parameter sweep constraint

When using tools like Optuna or Ray Tune, enforce the search space:

search_space = {
    "sampling.temperature": tune.uniform(0.0, 1.0),  # upper bound inclusive
    "learning_rate": tune.loguniform(1e-5, 1e-3),
}

Verification – Confirming the fix works

• Rerun the training job; the stack trace should disappear.
• Check the first few log lines for the effective temperature:

2026-08-23 10:12:01,123 DEBUG Effective temperature: 0.9

Validate that a checkpoint is written after the first epoch.

Run a quick inference pass to ensure the model respects the new temperature.

Prevention – Guardrails for future runs

• Schema validation: Use a JSON/YAML schema (e.g., jsonschema) that marks temperature with minimum: 0.0 and maximum: 1.0. Integrate the check into CI.
• Runtime assertions: Keep the clamping helper (compute_temperature) in a shared utilities module.
• Checkpoint compatibility layer: On trainer start, invoke mistral.trainer.validate_sampling_params inside a try/except block and auto‑fix legacy checkpoints.
• Monitoring: Emit a custom metric mistral.sampling.temperature to Prometheus; set an alert if the value exceeds 1.0.
• Documentation reminder: Add a comment next to the temperature field in config files referencing the official docs (Mistral Sampling Configuration).

FAQ – Common follow‑up questions

1. Why does the trainer accept temperature = 1.0 but reject 1.0001?
   The validation uses a closed interval [0.0, 1.0]. Floating‑point rounding errors can push a computed value just above 1.0, so clamping is required before the check.
2. Can I disable the temperature validation?
   Not recommended. The check is hard‑coded in Trainer.validate_sampling_params. Overriding it would require patching the SDK, which defeats the purpose of the safety guard.
3. Is the same range enforced for other sampling parameters (top_k, top_p)?
   Yes. top_p must be in (0, 1] and top_k must be a positive integer. Validation errors follow the same pattern.
4. My checkpoint was created with temperature = 1.05 on Mistral 2.2.1; can I reuse it?
   Run a migration script (see the example above) to clamp the stored temperature to 1.0 before loading the checkpoint with v2.3.0 or later.
5. How do I debug a situation where the temperature is computed from a learning‑rate schedule?
   Print the intermediate value before clamping, e.g., logging.debug("raw temperature: %s", raw), and verify the schedule never exceeds the 1.0 ceiling. Adjust the scaling factor accordingly.

Related Topic Hub: LLM Systems Troubleshooting Hub