Google Gemini function call format error from on-prem Kubernetes via VPN

Problem

When a service running in an on‑premises Kubernetes cluster invokes the Google Gemini Function Calling API over a site‑to‑site VPN, the request fails with a 400 Bad Request and the response body contains:

{
  "error": {
    "code": 400,
    "message": "Function call format error: payload does not match expected schema",
    "status": "INVALID_ARGUMENT"
  }
}

The same code works flawlessly from a GKE workload that resides in the same Google Cloud project, which indicates the issue is not a code‑level bug but something introduced by the hybrid network path or environment‑specific configuration.

Root Cause

The Gemini Function Calling API expects a JSON payload that conforms exactly to the FunctionCallRequest schema defined in the public OpenAPI spec. In the on‑prem environment the request body is being altered by an NGINX ingress controller that terminates TLS and re‑serialises the JSON using its default application/json formatter. The formatter strips null fields and collapses empty objects, producing a payload that no longer matches the strict schema required by Gemini.

Additionally, the VPN MTU is set to 1400 bytes. The Gemini client library adds a Content‑Length header that reflects the original payload size (≈ 1500 bytes). When the packet is fragmented, the downstream Google load balancer rejects the request because the Content‑Length does not match the reassembled body size, causing the same “format error” response.

Debug

1. Capture the outbound request from the pod

# Exec into the pod and run tcpdump on the outbound interface
kubectl exec -n prod svc/my-service-abc-123 -- \
  tcpdump -i any -s 0 -w /tmp/request.pcap host api.google.com and port 443

Extract the HTTP payload with tshark:

# On the host
tshark -r request.pcap -Y http.request -V | grep -A5 "Content-Type"

Result (truncated):


Content-Type: application/json
Content-Length: 1523

{
  "functionCalls": [
    {
      "name": "getUserInfo",
      "arguments": {
        "userId": "12345"
      }
    }
  ],
  "metadata": null
}

2. Compare with the request received by Gemini (captured via Cloud Logging)

# In Cloud Logging, filter on the requestId
resource.type="http_load_balancer"
jsonPayload.requestId="abcd-1234"

Logged payload:


{
  "functionCalls": [
    {
      "name": "getUserInfo",
      "arguments": {}
    }
  ]
}

Notice that the userId field is missing and metadata has been stripped.

3. Verify MTU fragmentation

# From the pod, send a test request with a known size
curl -s -o /dev/null -w "%{size_upload}\n" -X POST \
  -H "Content-Type: application/json" \
  -d @large_payload.json \
  https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=API_KEY

Output shows 1523 bytes uploaded, exceeding the VPN MTU.

4. Identify the ingress configuration

# Show the NGINX ingress config map
kubectl -n ingress-nginx get configmap nginx-configuration -o yaml

Relevant snippet:


data:
  proxy-body-size: "1m"
  http-snippet: |
    proxy_set_header Content-Type "application/json";
    proxy_pass_request_body on;

Solution

1. Preserve the original JSON payload

Configure the ingress to disable JSON re‑serialization by passing the request body verbatim:

# Update the ConfigMap
kubectl -n ingress-nginx edit configmap nginx-configuration

Change the http-snippet to:


http-snippet: |
  # Preserve original body; do not modify JSON
  proxy_set_header Content-Type $http_content_type;
  proxy_pass_request_body on;
  proxy_set_body $request_body;

Reload NGINX:

kubectl -n ingress-nginx rollout restart deployment/ingress-nginx-controller

2. Align MTU and Content‑Length

Set the VPN MTU to 1500 (standard Ethernet) or lower the client payload size. The simplest change is to add a --max-body-size flag to the Gemini client library configuration:

# Before (payload > 1500 bytes)
client = GeminiClient(api_key=API_KEY)

# After (truncate optional fields)
client = GeminiClient(
    api_key=API_KEY,
    max_payload_bytes=1400  # ensures Content‑Length ≤ MTU
)

Alternatively, adjust the VPN device on the on‑prem gateway:

# On the VPN appliance (example for strongSwan)
ip link set dev eth0 mtu 1500

3. Validate the request schema locally

Run the Gemini SDK’s built‑in validator before sending the request:

from google.generativeai import schema

def validate_call(payload):
    try:
        schema.validate_function_call(payload)
    except schema.SchemaError as e:
        raise ValueError(f"Invalid payload: {e}")

validate_call(my_payload)  # raises early if mismatched

Verify

1. Re‑capture the outbound request

# Repeat the tcpdump steps; inspect the payload
tshark -r request.pcap -Y http.request -V | grep -A6 "Content-Type"

Expected payload (unchanged):


{
  "functionCalls": [
    {
      "name": "getUserInfo",
      "arguments": {
        "userId": "12345"
      }
    }
  ],
  "metadata": null
}

2. Confirm successful API response

# Run the request from the pod
curl -s -X POST \
  -H "Content-Type: application/json" \
  -d @payload.json \
  "https://generativelanguage.googleapis.com/v1beta/models/gemini-pro:generateContent?key=API_KEY"

Expected response:

{
  "candidates": [
    {
      "content": {
        "parts": [
          {"text": "User 12345 is John Doe, active since 2020."}
        ]
      }
    }
  ]
}

3. Monitor Cloud Logging for the requestId

The log entry should now show the full functionCalls array with the userId argument present and no schema errors.

Prevent

  • Ingress hygiene: Disable any body‑modifying filters (e.g., proxy_set_body, sub_filter) for traffic destined to external APIs that enforce strict JSON schemas.
  • MTU consistency: Standardise the VPN MTU to 1500 bytes across all sites or enforce payload size limits in the client library.
  • Schema validation CI step: Add a unit test that runs schema.validate_function_call against every generated request payload.
  • Observability: Create a Cloud Logging metric that counts INVALID_ARGUMENT responses from Gemini; alert on spikes.
  • Version pinning: Keep the Gemini SDK version aligned with the OpenAPI spec used by the service; mismatched versions can introduce subtle field‑name changes.

Related Topic Hub: LLM Systems Troubleshooting Hub

FAQ

  1. Why does the same code work from GKE but fail from on‑prem?
    Because GKE traffic bypasses the on‑prem NGINX ingress, so the JSON payload is sent unchanged and the MTU matches Google’s network defaults.
  2. Can I disable the payload size check instead of fixing MTU?
    The Gemini API validates the JSON schema regardless of size; truncating fields to fit the MTU is safe, but disabling the check would cause the API to reject the request anyway.
  3. Is there a way to make NGINX forward the raw body without any transformation?
    Yes. Set proxy_pass_request_body on; and avoid any proxy_set_body or sub_filter directives that rewrite the request.
  4. Do I need to regenerate client libraries after changing the ingress?
    No. The client library remains unchanged; the fix only ensures the request reaches Google unchanged.
  5. What monitoring should I add to catch similar issues early?
    Create a log‑based metric for error.message:"Function call format error" and set an alert threshold of >0 per 5 minutes.