Google Gemini Token Limit Exceeded During Multi-GPU Distributed Training

In this article

Table of Contents

Google Gemini Token Limit Exceeded During Multi‑GPU Distributed Training

Problem Description

When launching a Distributed Data Parallel (DDP) or torch.distributed job that trains a Gemini model across multiple GPUs, the training loop aborts with errors similar to the following:


Error 400: Token limit exceeded – request exceeds maximum allowed tokens of 8192
gemini_token_limit_exceeded: batch_id=42, tokens=10240, limit=8192
GeminiError: Token limit exceeded during batch processing
Request failed: token limit exceeded (max_output_tokens=2048)

These failures appear after a few epochs or intermittently when the batch size or sequence padding changes. The symptom is a 400 HTTP response from the Gemini API or a raised GeminiError in the Python SDK, causing the entire distributed job to terminate.

Root Cause Analysis

The Gemini service enforces strict per‑request token caps for both input and output tokens. According to the Google Gemini API Reference – Token Limits section, the default maximum is 8192 input tokens and 2048 output tokens per request. In a multi‑GPU setting the following interactions cause the limit to be exceeded:

Sequence Sharding & Padding – The Google Gemini Distributed Training Guide describes that each GPU receives a shard of the full sequence. When the trainer pads each shard to the longest sequence in the batch, the effective token count becomes the sum of shard_length × num_gpus. If the original sequence length is near the limit, padding can push the total over 8192.
Gradient Accumulation – Community reports (GitHub issue in googleapis/python-gemini and internal benchmark) show that accumulating gradients across steps doubles the effective sequence length sent to the model for the backward pass, effectively multiplying the token count.
Dynamic Batch Concatenation – In Vertex AI pipelines, prompts are concatenated across GPUs for batch inference. The Limits and quotas page notes that token limits apply to the *combined* batch, not per‑GPU, so dynamic padding can cause occasional spikes above the ceiling.
Mis‑configured SDK Parameters – The Gemini Python SDK exposes max_input_tokens and max_output_tokens. If these are left at defaults while the training script increases batch_size or seq_len, the SDK will reject the request before it reaches the API, emitting the “Token limit exceeded during batch processing” error.

In short, the distributed training pipeline unintentionally aggregates token counts across GPUs, violating the per‑request token caps defined by the Gemini service.

Investigation and Debugging Steps

Inspect Training Logs for Token Counts


2024-07-12 14:03:21,874 INFO  gemini_training - batch_id=42 tokens=10240 limit=8192
2024-07-12 14:03:22,001 ERROR gemini_training - GeminiError: Token limit exceeded during batch processing

Look for the tokens= field and compare against the documented limit.

Validate SDK Configuration


import gemini
client = gemini.GeminiClient(
    max_input_tokens=8192,   # default
    max_output_tokens=2048   # default
)
print(client.config)

If max_input_tokens is lower than the effective token count, increase it only up to the service limit.

Capture Per‑GPU Sequence Lengths using torch.distributed utilities:


import torch.distributed as dist
local_len = input_ids.size(1)
all_lens = [torch.zeros_like(local_len) for _ in range(dist.get_world_size())]
dist.all_gather(all_lens, local_len)
print("Per‑GPU lengths:", all_lens)
print("Total tokens:", sum(all_lens))

If sum(all_lens) > 8192, the request will be rejected.

Check Padding Strategy – Verify that torch.nn.utils.rnn.pad_sequence is not padding to the maximum length of the *entire* batch across GPUs. Example log:
```
Padding to length 2560 (max across 4 GPUs) → effective tokens = 2560 * 4 = 10240
```
Network Trace (optional) – Use tcpdump to capture the outbound HTTP request and confirm the Content-Length header reflects the oversized payload:
```
sudo tcpdump -i eth0 -s 0 -w gemini_req.pcap port 443 and host gemini.googleapis.com
```

Resolution

The fix consists of three coordinated changes: limit the effective token budget per request, adjust padding/accumulation logic, and enforce SDK limits.

1. Enforce a Global Token Budget

Calculate the maximum allowable per‑GPU sequence length before sharding:


MAX_GLOBAL_TOKENS = 8192
NUM_GPUS = torch.cuda.device_count()
MAX_PER_GPU = MAX_GLOBAL_TOKENS // NUM_GPUS   # integer division

Clamp the input sequence on the host side:


def truncate_to_budget(input_ids):
    if input_ids.size(1) > MAX_PER_GPU:
        return input_ids[:, :MAX_PER_GPU]
    return input_ids

2. Switch to Length‑Aware Padding

Instead of padding to the longest shard across all GPUs, pad each shard to MAX_PER_GPU or the local maximum, whichever is smaller.


def pad_shard(shard):
    target_len = min(shard.size(1), MAX_PER_GPU)
    return torch.nn.functional.pad(
        shard,
        (0, target_len - shard.size(1)),
        value=tokenizer.pad_token_id
    )

3. Adjust Gradient Accumulation

If using gradient_accumulation_steps, ensure that the effective token count does not exceed the limit by resetting the accumulation buffer after each sub‑step:


accum_steps = 4
for step, batch in enumerate(dataloader):
    outputs = model(**batch)
    loss = outputs.loss / accum_steps
    loss.backward()
    if (step + 1) % accum_steps == 0:
        optimizer.step()
        optimizer.zero_grad()

This keeps each backward pass within the token budget.

4. Update SDK Parameters Explicitly

Set the SDK limits to match the calculated budget:


client = gemini.GeminiClient(
    max_input_tokens=MAX_GLOBAL_TOKENS,
    max_output_tokens=2048   # keep within service cap
)

Before / After Comparison

Aspect	Before	After
Per‑GPU sequence length	2560 (4 GPUs → 10240 tokens)	2048 (4 GPUs → 8192 tokens)
Padding strategy	Global max across GPUs	Length‑aware per shard
Gradient accumulation	Accumulated over 8 steps → effective 2× length	Accumulator reset every 4 steps
SDK config	Defaults (8192/2048) but not enforced	Explicit max_input_tokens=8192

Verification

Run a Single‑GPU Smoke Test with the same batch size to confirm the request succeeds.
Check Log Output for the token count field:
```
2024-07-12 14:45:10,112 INFO  gemini_training - batch_id=57 tokens=8192 limit=8192
```
No “Token limit exceeded” entry should appear.
Validate API Response – A successful HTTP 200 with a JSON payload containing "status":"ok" confirms the request passed the token check.
Monitor GPU Utilization – Ensure that the new padding does not cause severe under‑utilization; target >70% memory usage.

Operational Best Practices and Prevention

Automated Token Budget Check – Insert a pre‑flight hook that computes total_tokens = seq_len * world_size and aborts early if it exceeds the service limit.
Dynamic Batch Sizing – Adjust batch_size or seq_len on the fly based on observed token usage metrics.
Metric Alerting – Emit a custom Prometheus metric gemini_token_budget_exceeded_total and set an alert threshold of >0.
Version Pinning – Use the Gemini SDK version referenced in the Distributed Training Guide (e.g., gemini==2.3.1) to avoid regressions in token handling.
Documentation Alignment – Keep the training script’s max_input_tokens and max_output_tokens values synchronized with the limits documented on the Limits and quotas page.

FAQ

Why does the error appear only after a few epochs?
Sequence lengths can grow during curriculum learning or when data augmentation adds tokens. The token budget is static, so later epochs may cross the threshold.
Can I increase the token limit by requesting a higher quota?
No. The Gemini service enforces a hard cap (8192 input, 2048 output) per request regardless of quota. You must stay within these bounds.
Is the limit applied per GPU or per request?
The limit applies to the *combined* request sent to the Gemini API. In DDP each shard is concatenated into a single request, so the sum across GPUs must stay under the cap.
How do I know the exact token count sent to the API?
Enable SDK debug logging (export GEMINI_LOG_LEVEL=DEBUG) which prints the tokens= field before each request, as shown in the training logs.
Does pipeline parallelism affect the token limit?
Yes. When upstream stages emit longer token streams than downstream stages expect, the downstream stage may receive a batch that exceeds max_input_tokens. Align the max_input_tokens configuration across all pipeline stages.

Related Topic Hub: LLM Systems Troubleshooting Hub