AWS EC2 GPU token limit exceeded during language model evaluation

Problem — Token‑limit failures during language‑model evaluation on AWS EC2 GPU instances

When running a Hugging Face transformers evaluation pipeline on an EC2 GPU instance (e.g., p3.2xlarge, g5.12xlarge), the script aborts with errors such as:

ValueError: Token indices sequence length is longer than the model maximum sequence length (max_length=2048).

or

RuntimeError: CUDA out of memory. Tried to allocate 12.3 GiB on GPU 0.
... (subsequent handling) ...
Error: token limit exceeded – input length 4096 exceeds max allowed 2048 tokens (model: Llama‑2‑7B).

Symptoms observed in production:

• Evaluation jobs terminate after a few minutes.
• GPU memory usage spikes to 100 % just before the crash.
• Logs contain both CUDA out of memory and token limit exceeded messages.
• Increasing batch size or adding more instances does not resolve the failure.

Root Cause Analysis

The failure is a combination of two tightly coupled constraints:

1. Model‑defined maximum position embeddings. Every transformer model has a hard‑coded max_position_embeddings (e.g., 2048 for Llama‑2‑7B, 512 for BERT‑large). The tokenizer will produce a token sequence whose length must not exceed this value, otherwise a ValueError is raised.
2. GPU memory required for the attention matrix. For a sequence of length L, the self‑attention operation allocates an L × L matrix per layer. On a p3.2xlarge (16 GiB GPU) a 2048‑token sequence already consumes >10 GiB of memory. Any increase (e.g., 3000 tokens) exceeds the available memory, triggering a CUDA OOM. The OOM is often caught by the evaluation harness, which then reports a generic “token limit exceeded” error.

Evidence from the official AWS EC2 Instance Types shows that p3.2xlarge provides 16 GiB GPU memory, while g5.12xlarge provides 48 GiB. Even the larger p4d.24xlarge (1.1 TiB) can hit the positional‑embedding ceiling when evaluating very long documents (>3000 tokens) as reported in the Reddit discussion.

Thus, the root cause is input sequences longer than the model’s positional embedding limit, combined with insufficient GPU memory to allocate the required attention tensors for those sequences.

Investigation & Debugging Steps

1. Confirm the exact token length of the failing input.

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
text = open("sample.txt").read()
tokens = tokenizer.encode(text, return_tensors="pt")
print("Token count:", tokens.shape[1])

Expected output (example): Token count: 4096

2. Check model configuration for max_position_embeddings.

from transformers import AutoConfig
config = AutoConfig.from_pretrained("meta-llama/Llama-2-7b-hf")
print("max_position_embeddings:", config.max_position_embeddings)

Typical output: max_position_embeddings: 2048

3. Inspect GPU memory usage before the failure.

nvidia-smi --query-gpu=memory.used,memory.total --format=csv
# Example output:
# 12234 MiB, 16384 MiB

4. Search logs for OOM and token‑limit messages.

journalctl -u evaluation.service -n 200 | grep -E "CUDA out of memory|token limit"
# Sample excerpt:
# RuntimeError: CUDA out of memory. Tried to allocate 12.3 GiB on GPU 0.
# Error: token limit exceeded – input length 4096 exceeds max allowed 2048 tokens

5. Validate that the EC2 instance type matches the expected GPU memory.

curl -s http://169.254.169.254/latest/meta-data/instance-type
# Returns: p3.2xlarge

Solution – Bringing the Evaluation Pipeline Within Limits

The fix consists of three coordinated actions:

1. Truncate or chunk inputs to respect max_position_embeddings

def safe_tokenize(text, tokenizer, max_len):
    tokens = tokenizer.encode(text, add_special_tokens=True)
    if len(tokens) > max_len:
        # Simple truncation
        tokens = tokens[:max_len]
    return torch.tensor(tokens).unsqueeze(0)

# Usage
max_len = config.max_position_embeddings
input_ids = safe_tokenize(text, tokenizer, max_len)

2. Enable memory‑efficient attention or off‑loading

For PyTorch 2.x, torch.compile with attention="flash" reduces the attention matrix footprint. Alternatively, DeepSpeed ZeRO‑2 offloads optimizer states and activations to host RAM.

# Using flash attention (requires CUDA 11.8+)
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-7b-hf",
    torch_dtype=torch.float16,
    device_map="auto",
    attn_implementation="flash_attention_2"
)

Or with DeepSpeed:

# deepspeed_config.json
{
  "zero_optimization": {
    "stage": 2,
    "offload_param": {"device": "cpu"},
    "offload_optimizer": {"device": "cpu"}
  },
  "gradient_checkpointing": true
}

3. Scale up the GPU instance or use Elastic Inference

If the workload must process sequences near the positional limit (e.g., 3000 tokens for research), move to a larger instance (p4d.24xlarge) or attach AWS Elastic Inference to a smaller instance to increase effective memory for inference.

Before & After Comparison

Aspect	Before	After
Input handling	Raw text → tokenizer → OOM → token‑limit error	Safe tokenization with truncation to max_position_embeddings
Attention implementation	Standard dense attention (L × L matrix)	Flash attention (reduced memory) or DeepSpeed ZeRO‑2
Instance type	p3.2xlarge (16 GiB)	p4d.24xlarge (1.1 TiB) or p3.2xlarge + Elastic Inference

Verification – Confirming the Fix

1. Run the same evaluation script with a known‑long input.
2. Check that nvidia-smi reports no OOM and GPU memory stays below 80 %.
3. Confirm that the log no longer contains “token limit exceeded”.

# Expected log excerpt
INFO:root:Processing document 1/10 – token count 2048
INFO:root:Inference completed in 0.87 s, GPU memory used 12.1 GiB

Run a quick sanity test:

python evaluate.py --input sample.txt --max-length 2048
# Should exit with code 0
echo $?
# 0

Prevention – Operational Guardrails

• Input validation layer. Add a pre‑processor that rejects or automatically chunks any request exceeding model.config.max_position_embeddings.
• Monitoring. Export torch.cuda.memory_allocated and torch.cuda.memory_reserved as Prometheus metrics; alert when usage exceeds 85 %.
• Instance‑type policy. Enforce a minimum GPU memory threshold (e.g., 32 GiB) for any evaluation job that uses models with >2 K positional embeddings.
• Version pinning. Use the Deep Learning AMI version that matches the CUDA driver required for flash attention (see AWS Deep Learning AMI guide).
• Batch size & dynamic padding. Keep batch_size low (1–2) for long sequences and enable padding=False to avoid unnecessary memory allocation.

Related Topic Hub: Cloud Infrastructure Troubleshooting Hub

FAQ

1. Why does the error say “token limit exceeded” after a CUDA OOM?
   The evaluation harness catches the OOM, then re‑raises a higher‑level ValueError that reports the root cause (input length > max_position_embeddings). Both conditions are present, but the generic message masks the underlying memory pressure.
2. Can I increase max_position_embeddings without retraining?
   Only by fine‑tuning the model with a larger positional embedding matrix (e.g., using model.resize_position_embeddings(new_len)) and re‑initializing the embeddings. This requires additional GPU memory and is not a quick fix for production pipelines.
3. Is Elastic Inference suitable for large language models?
   EI can off‑load matrix multiplications for inference, reducing GPU memory usage, but it is limited to models ≤ 2 B parameters. For 7 B+ models, scaling the GPU instance or using model parallelism (e.g., torch.distributed) is recommended.
4. How do I know which attention implementation is active?
   Inspect model.config.attn_implementation at runtime. It should report flash_attention_2 when flash attention is enabled. If it shows eager, the memory‑heavy dense attention is still in use.
5. What monitoring metric best predicts an upcoming token‑limit failure?
   Track the maximum token length observed per request (e.g., max_input_tokens) alongside GPU memory usage. A sudden spike in max_input_tokens that approaches model.config.max_position_embeddings is a strong predictor.