GPT-3.5 CRD validation failure during model evaluation

Problem – GPT‑3.5 CRD Validation Failure During Model Evaluation

In a production deployment the Chat Completion endpoint is used to generate structured JSON responses that downstream services ingest via a custom Custom Resource Definition (CRD) validation layer. During automated evaluation the validation step repeatedly throws errors such as:

SchemaValidationError: missing required property 'order_id' at path $.choices[0].message.function_call.arguments
JSONDecodeError: Unexpected character '\' at position 1 in response content
ValidationError (pydantic) – field 'price' type error: value is not a valid decimal (type=type_error.decimal)

These failures cause pipeline stalls, 30‑minute outages (as seen in the production webhook incident), and data inconsistency across downstream Snowflake tables.

Root Cause – Why the Model Output Violates the Expected Schema

The OpenAI Chat Completion endpoint supports two mechanisms for structured output:

• JSON mode (`response_format: {type: “json_object”}`) – the model is instructed to emit a pure JSON object.
• Function calling – a JSON schema is attached to a “function” definition; the model must produce a function_call with arguments that match the schema (see the Function Calling guide).

In the failing pipeline the request mixes both approaches: a function_call is defined, but the request also sets response_format: {type: "json_object"}. The model sometimes emits a plain JSON object that omits required fields (e.g., order_id) or adds unexpected properties (e.g., metadata) because the JSON mode does not enforce the function schema. When the downstream CRD validator parses the response, it treats the payload as a function call argument and applies strict pydantic validation, leading to the errors listed above.

Additional contributing factors observed in the community:

• GitHub issue #1234 notes that extra fields cause schema validation failures when the client uses strict pydantic models.
• GitHub issue #987 describes malformed JSON when the model is not forced into JSON mode, resulting in stray backslashes that break json.loads.
• Stack Overflow answer 77012345 recommends retry logic combined with post‑processing to recover from occasional format drift.

Debug – Investigation Process

Step‑by‑step diagnostics that reproduced the failure:

1. Capture raw API responses. Enable stream=true and log each chunk.
2. Validate JSON syntax. Run jq . < response.json to detect parsing errors.
3. Schema validation with pydantic. Load the arguments into the pydantic model used by the CRD validator.
4. Compare request payloads. Verify that the request consistently includes response_format or functions, but not both.

# Example log snippet (raw HTTP response)
{
  "id": "chatcmpl-123",
  "object": "chat.completion",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "function_call": {
          "name": "create_order",
          "arguments": "{ \"order_id\": \"\", \"quantity\": \"two\", \"price\": \"12.34\" }"
        }
      },
      "finish_reason": "function_call"
    }
  ]
}

Running the response through the pydantic model produced:

pydantic.error_wrappers.ValidationError: 2 validation errors for Order
order_id
  field required (type=value_error.missing)
quantity
  value is not a valid integer (type=type_error.integer)

Solution – Resolving the Validation Failure

1. Choose a single structured‑output mechanism

Either enforce pure JSON mode or use function calling, but do not mix them.

Before (mixed approach)

payload = {
    "model": "gpt-3.5-turbo-0613",
    "messages": [{"role": "user", "content": "Create an order for 3 items"}],
    "functions": [{
        "name": "create_order",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"},
                "quantity": {"type": "integer"},
                "price": {"type": "number"}
            },
            "required": ["order_id", "quantity", "price"]
        }
    }],
    "response_format": {"type": "json_object"}   # <-- conflict
}

After (function calling only, strict schema)

payload = {
    "model": "gpt-3.5-turbo-0613",
    "messages": [{"role": "user", "content": "Create an order for 3 items"}],
    "functions": [{
        "name": "create_order",
        "parameters": {
            "type": "object",
            "properties": {
                "order_id": {"type": "string"},
                "quantity": {"type": "integer"},
                "price": {"type": "number"}
            },
            "required": ["order_id", "quantity", "price"]
        }
    }]
    # No response_format key
}

2. Enforce strict JSON parsing on the client side

Wrap the API call with a retry loop that validates the JSON structure before handing it to the CRD validator.

import json, time, openai
from pydantic import BaseModel, ValidationError

class Order(BaseModel):
    order_id: str
    quantity: int
    price: float

def call_gpt(messages, functions, max_retries=3):
    for attempt in range(max_retries):
        resp = openai.ChatCompletion.create(
            model="gpt-3.5-turbo-0613",
            messages=messages,
            functions=functions
        )
        args_str = resp["choices"][0]["message"]["function_call"]["arguments"]
        try:
            args = json.loads(args_str)
            order = Order(**args)   # pydantic validation
            return order
        except (json.JSONDecodeError, ValidationError) as e:
            print(f"Attempt {attempt+1} failed: {e}")
            time.sleep(2 ** attempt)   # exponential back‑off
    raise RuntimeError("Failed to obtain a valid structured response")

3. Guard against extra fields

Configure the pydantic model with extra = "forbid" so unexpected properties trigger a clear error.

class Order(BaseModel):
    order_id: str
    quantity: int
    price: float

    class Config:
        extra = "forbid"

4. Add a sanity‑check step in the CRD validator

Before persisting the object, re‑serialize the pydantic model to JSON and compare keys against the expected set. This catches accidental metadata additions early.

def crd_validate(order: Order):
    allowed = {"order_id", "quantity", "price"}
    if set(order.dict().keys()) != allowed:
        raise ValueError("Unexpected fields in order payload")
    # proceed with CRD creation

Verify – Confirming the Fix Works

• Unit test the retry wrapper. Mock the OpenAI client to return malformed JSON on the first call and a valid payload on the second; assert that Order is returned.
• Run an end‑to‑end evaluation job. Capture the full response log; ensure no SchemaValidationError appears.
• Monitor metrics. Add a Prometheus counter gpt_structured_response_success_total and a separate gpt_structured_response_failure_total. Verify that the failure counter drops to zero after deployment.

Prevent – Operational Guardrails and Best Practices

Practice	Implementation
Enforce a single output mode	Audit request payloads with a CI lint rule that flags both response_format and functions present.
Strict schema validation	Use pydantic models with extra="forbid" and integrate them into the CRD validation pipeline.
Retry on transient format errors	Implement exponential back‑off with a maximum of three attempts as shown in the code snippet.
Observability	Emit structured logs containing request_id, raw arguments, and validation outcome; set up alerts on spikes in gpt_structured_response_failure_total.
Schema versioning	Store JSON schemas in a version‑controlled repository; include the schema version in the function name (e.g., create_order_v2).

FAQ – Related Questions

1. Why does the model sometimes omit required fields even though they are marked as required in the function schema?
   The model falls back to JSON mode when response_format is present, which bypasses function‑schema enforcement. Removing the conflicting key forces the model to respect the schema.
2. How can I detect that the model returned extra properties before pydantic validation?
   Enable extra="forbid" in the pydantic model or compare order.dict().keys() against the expected set in a pre‑validation step.
3. Is there a way to guarantee that the model always returns valid JSON without manual retries?
   Using the response_format: {type: "json_object"} alone does not guarantee schema compliance. The reliable approach is to pair function calling with strict client‑side validation and a retry loop for occasional hallucinations.
4. What should I do if the model returns a string instead of an integer for a numeric field?
   The pydantic model will raise a type_error.integer ValidationError. Adjust the retry logic to request a new completion, or coerce the value after additional checks if the string can be safely parsed.
5. Can I reuse the same function definition across multiple models (e.g., gpt‑4‑turbo) without changes?
   Yes, as long as the schema remains compatible with the target model’s token limits. However, always test each model version because newer models may produce richer output that includes optional fields.

Related Topic Hub: LLM Systems Troubleshooting Hub