the google logo is displayed in front of a black background

Google Gemini function calling JSON parse error after offline deployment

by

Problem – Malformed Function‑Calling JSON from Gemini in an Air‑Gapped Deployment

When calling a Gemini model deployed on‑premises behind an air‑gap, the client receives a response that cannot be parsed as JSON. Typical error messages include:

  • JSONDecodeError: Expecting ',' delimiter – trailing commas or missing quotes.
  • JSONDecodeError: Unexpected token '<' at position 0 – HTML error page returned.
  • Invalid character '\x0a' at position 0 – protobuf bytes or stray newline before the opening brace.

These errors prevent the application from invoking the intended function and cause downstream failures such as request timeouts or incorrect business logic execution.

Root Cause – Why the JSON Payload Is Invalid

Three independent factors combine to produce malformed JSON in an offline Gemini deployment:

  1. Outdated function‑calling schema bundled with the on‑prem package. The financial services incident showed that an older function_call_schema.json emitted an extra trailing comma after the last argument, violating the schema defined in the official Function Calling reference.
  2. Server output mode mismatch. If the Gemini service is started without the --enable-json-output flag, it returns raw protobuf bytes with Content‑Type: application/octet-stream. This triggers the “Invalid content‑type” error described in the community GitHub issue “JSONDecodeError on local endpoint – response contains protobuf delimiters”.
  3. Network‑layer transformations. A custom reverse proxy used to expose the local endpoint added a health‑check banner (a newline before the JSON object). The healthcare provider case demonstrated that the leading \n broke strict parsers that expect the payload to start with {.

In addition, the default max_response_size of 4 KB on the on‑prem server truncates large function‑call payloads, leading to incomplete JSON (manufacturing plant incident).

Investigation – Debugging Steps

Follow these steps to isolate the exact failure mode.

1. Capture the raw HTTP response

curl -i -X POST https://gemini-local.example.com/v1beta/models/gemini-1.5/functionCall \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"Call getUserInfo with id=42","function_calling":true}'

Typical problematic output:

HTTP/1.1 200 OK
Content-Type: application/octet-stream
Content-Length: 112

\x0a\x0a\x0a...binary protobuf...

Or with a stray newline:

HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 215

\n{"functionCall":{"name":"getUserInfo","arguments":{"id":42,}}}

2. Verify server configuration flags

# Check the Gemini service startup command
ps -ef | grep gemini

# Expected flag
--enable-json-output

If the flag is missing, the server defaults to protobuf output.

3. Inspect the schema file version

cat /opt/gemini/schema/function_call_schema.json | grep '"$schema"'
# Expected version string (e.g., "v2.1")

An older version will contain a trailing comma pattern.

4. Look for proxy‑injected banners

journalctl -u reverse-proxy.service | grep "banner"
# Example line:
# Adding health‑check banner: "\n--- Gemini Service Ready ---\n"

5. Check max response size limits

grep max_response_size /etc/gemini/server.conf
# Default: 4096

Resolution – Fixing the Malformed JSON

1. Enable JSON output on the Gemini server

Update the service unit or startup script to include the flag.

# Before
ExecStart=/opt/gemini/bin/gemini_server --config /etc/gemini/server.conf

# After
ExecStart=/opt/gemini/bin/gemini_server --config /etc/gemini/server.conf --enable-json-output

Restart the service:

systemctl daemon-reload
systemctl restart gemini-server

2. Upgrade the function‑calling schema

Replace the outdated schema with the version shipped in the latest on‑prem release (v2.1 as of 2024‑09).

# Backup old schema
mv /opt/gemini/schema/function_call_schema.json /opt/gemini/schema/function_call_schema.json.bak

# Copy new schema from the release bundle
cp /opt/gemini/releases/2024-09/schema/function_call_schema.json /opt/gemini/schema/

3. Remove proxy‑added newline/banner

Adjust the reverse‑proxy configuration to disable any response‑body modifications.

# Example Nginx snippet before
add_before_body "\n--- Gemini Service Ready ---\n";

# After – comment out
# add_before_body "\n--- Gemini Service Ready ---\n";

Reload the proxy:

nginx -s reload

4. Increase the maximum response size (if needed)

# /etc/gemini/server.conf
max_response_size = 16384  # 16 KB

# Apply changes
systemctl restart gemini-server

5. Explicitly request JSON response format (client side)

Some SDKs default to streaming mode, which can emit partial protobuf fragments. Set the request parameter as shown in the Stack Overflow answer.

from vertexai.preview import generative_models

model = generative_models.GenerativeModel(
    "gemini-1.5-pro",
    generation_config={"response_format": "JSON"}  # forces JSON output
)

response = model.generate_content(
    ["Call getUserInfo with id=42"],
    generation_config={"function_calling": True}
)

Verification – Confirming the Fix

  1. Re‑run the curl command. The response should start with { and have Content-Type: application/json:
    2. HTTP/1.1 200 OK
Content-Type: application/json
Content-Length: 182

{"functionCall":{"name":"getUserInfo","arguments":{"id":42}}}
  2. Parse the payload with the SDK; no JSONDecodeError should be raised.
    3. import json
payload = response.json()
print(payload["functionCall"]["name"])  # => getUserInfo
  3. Check monitoring metrics for gemini_function_call_json_errors (custom counter) – it should be zero.

Prevention – Operational Guardrails

  • Configuration management. Store the Gemini startup flags and schema files in version‑controlled IaC (e.g., Terraform or Ansible). Enforce --enable-json-output via a validation script before service rollout.
  • Schema validation CI step. Run a JSON‑schema lint against function_call_schema.json on every release to catch trailing commas.
  • Health‑check proxy hygiene. Ensure reverse proxies do not modify response bodies. Use proxy_set_header Content-Type $upstream_http_content_type; to preserve the original header.
  • Response size monitoring. Alert when gemini_server_payload_truncated exceeds a threshold, indicating the max_response_size may need adjustment.
  • Content‑type enforcement. In the client SDK, assert that response.headers["Content-Type"] == "application/json" and fail fast with a clear message if not.

FAQ – Common Follow‑Up Questions

  1. Why does the error appear only in the air‑gapped environment?
    Because the on‑prem package bundles an older schema and defaults to protobuf output, while the cloud‑hosted version always returns JSON.
  2. Can I keep streaming mode and still get valid JSON?
    Streaming mode concatenates partial chunks; in the offline server it may emit protobuf delimiters. Disable streaming or set response_format=JSON to receive a single well‑formed JSON object.
  3. What header should I check to ensure the payload is JSON?
    Verify Content-Type: application/json. If you see application/octet-stream, the server is sending protobuf.
  4. How do I increase the function‑call payload limit safely?
    Adjust max_response_size in /etc/gemini/server.conf and monitor the gemini_server_payload_truncated metric. Do not set it arbitrarily high; choose a value that covers the largest expected argument set.
  5. Is there a way to automatically strip a leading newline added by a proxy?
    Yes. In Nginx, disable any add_before_body directives or use proxy_buffering off; to prevent body manipulation. Alternatively, preprocess the response client‑side with response.text.lstrip() before JSON parsing, though fixing the proxy is preferred.

Related Topic Hub: LLM Systems Troubleshooting Hub