Structured output: Force LLM to return JSON matching schema

1. Overview

When integrating LLM into applications, you often don’t need a paragraph of text. You need structured data to save to DB, call APIs, or make decisions. This article shows how to force LLM to return JSON matching the correct schema in Spring Boot 3.x using Spring AI. After reading, you’ll be able to map a model’s response directly to a Java record, enable OpenAI’s native JSON Schema mode, validate results, retry when the model returns wrong data, and write tests for the parsing logic.

2. Why “just return JSON” is not enough

The simplest way is to add “Return as JSON” to the prompt. This approach breaks in multiple ways:

• The model wraps JSON in markdown fence, or adds a line like “Here’s the result:” at the beginning.
• Field names change between calls, sometimes quantity, sometimes qty.
• Data types are wrong, for example "quantity": "two" instead of 2.
• Enum values are made up, e.g., "speed": "super fast".

To be stable, you need three layers of protection:

1. Clear schema sent with the prompt, or better yet, sent via API as a constraint.
2. Strict parsing into a Java type.
3. Business validation after parsing, then retry if needed.

3. Setup

This article uses Spring AI 1.0 with OpenAI. First, add the BOM and starter:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>1.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-validation</artifactId>
    </dependency>
</dependencies>

Then configure the API key and model:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o-mini
spring.ai.openai.chat.options.temperature=0

temperature=0 reduces the model’s “creativity”. For data extraction tasks, that’s what you want.

4. Map response to a record with ChatClient.entity()

The example problem: extract order information from a customer’s email. Start by defining the target data type:

public enum DeliverySpeed { STANDARD, EXPRESS }

public record OrderRequest(
        @JsonProperty(required = true)
        @JsonPropertyDescription("Customer name")
        String customerName,

        @JsonProperty(required = true)
        @JsonPropertyDescription("Product code, format SP-xxxx")
        String productCode,

        @JsonProperty(required = true)
        @JsonPropertyDescription("Quantity, positive integer")
        int quantity,

        @JsonProperty(required = true)
        DeliverySpeed speed
) {}

The Jackson annotations here serve two purposes:

• @JsonPropertyDescription is included in the JSON Schema. The model reads this description to understand what each field means.
• @JsonProperty(required = true) marks the field as required in the schema.
• The enum DeliverySpeed generates "enum": ["STANDARD", "EXPRESS"], so the model is limited to these two values.

Next is the service that calls the model:

@Service
public class OrderExtractionService {

    private final ChatClient chatClient;

    public OrderExtractionService(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultSystem("You extract order information from customer emails.")
                .build();
    }

    public OrderRequest extract(String email) {
        return chatClient.prompt()
                .user(u -> u.text("Email:\n{email}").param("email", email))
                .call()
                .entity(OrderRequest.class);
    }
}

The .entity(OrderRequest.class) call does three things:

1. Generates a JSON Schema from OrderRequest.
2. Appends format instructions along with the schema to the end of the prompt.
3. Parses the response using Jackson into OrderRequest.

Try it with this email:

Hi shop, I'm Tran Thi Lan. I want to order 3 units of SP-1024, express shipping please.

Result received:

OrderRequest[customerName=Tran Thi Lan, productCode=SP-1024, quantity=3, speed=EXPRESS]

The model understands “express” corresponds to EXPRESS, because the enum limits it to two values.

4.1. Under the hood: BeanOutputConverter

.entity() uses BeanOutputConverter behind the scenes. You can use this class directly to see exactly what’s being sent:

var converter = new BeanOutputConverter<>(OrderRequest.class);
System.out.println(converter.getJsonSchema());

Output (abbreviated):

{
  "$schema" : "https://json-schema.org/draft/2020-12/schema",
  "type" : "object",
  "properties" : {
    "customerName" : { "type" : "string", "description" : "Customer name" },
    "productCode" : { "type" : "string", "description" : "Product code, format SP-xxxx" },
    "quantity" : { "type" : "integer", "description" : "Quantity, positive integer" },
    "speed" : { "type" : "string", "enum" : [ "STANDARD", "EXPRESS" ] }
  },
  "required" : [ "customerName", "productCode", "quantity", "speed" ],
  "additionalProperties" : false
}

The getFormat() method returns the instruction snippet appended to the prompt. This snippet asks the model to return only JSON per RFC 8259, with no explanation, no markdown wrapper, and includes the schema above. When parsing, convert() also strips markdown fence if the model adds it anyway.

4.2. Generic types: list of orders

If an email contains multiple orders, use ParameterizedTypeReference:

public List<OrderRequest> extractAll(String email) {
    return chatClient.prompt()
            .user(u -> u.text("List all orders in the email:\n{email}").param("email", email))
            .call()
            .entity(new ParameterizedTypeReference<List<OrderRequest>>() {});
}

Passing List.class won’t work. Generic information is erased at runtime, so Spring AI doesn’t know what type the elements are.

5. OpenAI’s native structured output

The approach in section 4 is just a request in the prompt. The model can still ignore it. Recent OpenAI models support response_format of type json_schema in strict mode. In strict mode, token generation is constrained by the schema, so the returned JSON is always valid in syntax and structure.

public OrderRequest extractStrict(String email) {
    var converter = new BeanOutputConverter<>(OrderRequest.class);

    var options = OpenAiChatOptions.builder()
            .responseFormat(new ResponseFormat(
                    ResponseFormat.Type.JSON_SCHEMA, converter.getJsonSchema()))
            .build();

    String json = chatClient.prompt()
            .user(u -> u.text("Email:\n{email}").param("email", email))
            .options(options)
            .call()
            .content();

    return converter.convert(json);
}

This code works as follows:

• Still uses BeanOutputConverter, but only to get the schema (getJsonSchema()) and parse the result (convert()).
• The schema is sent via API parameter instead of embedded in the prompt.
• .content() returns the raw JSON string, which we then convert.

Note: OpenAI’s strict mode has its own limitations. Every field must be in required, and the object must have additionalProperties: false. Some keywords like pattern or minimum may not be supported, depending on the model. If a field is optional, declare its type to allow null instead of leaving it out of required.

Native structured output guarantees structure, not correct content. The model can still return quantity: 0 or make up a productCode. That’s why the next section is needed.

6. Validate and retry

Add Bean Validation constraints to the record:

public record OrderRequest(
        @JsonProperty(required = true) @NotBlank
        String customerName,

        @JsonProperty(required = true) @Pattern(regexp = "SP-\\d{4}")
        String productCode,

        @JsonProperty(required = true) @Positive
        int quantity,

        @JsonProperty(required = true) @NotNull
        DeliverySpeed speed
) {}

Then validate the result. If the data is wrong, send the errors back to the model and ask it to fix them:

@Service
public class ValidatedOrderExtractor {

    private static final int MAX_ATTEMPTS = 3;

    private final ChatClient chatClient;
    private final Validator validator;

    public ValidatedOrderExtractor(ChatClient.Builder builder, Validator validator) {
        this.chatClient = builder.build();
        this.validator = validator;
    }

    public OrderRequest extract(String email) {
        String feedback = "";
        for (int attempt = 1; attempt <= MAX_ATTEMPTS; attempt++) {
            String prompt = "Extract order information from email:\n" + email + feedback;
            try {
                OrderRequest order = chatClient.prompt().user(prompt).call().entity(OrderRequest.class);
                Set<ConstraintViolation<OrderRequest>> violations = validator.validate(order);
                if (violations.isEmpty()) {
                    return order;
                }
                feedback = "\n\nLast time you returned incorrect data: " + describe(violations) + ". Please fix it.";
            } catch (RuntimeException e) {
                feedback = "\n\nLast time the response couldn't be parsed: " + e.getMessage();
            }
        }
        throw new IllegalStateException("Could not extract order after " + MAX_ATTEMPTS + " attempts");
    }

    private String describe(Set<ConstraintViolation<OrderRequest>> violations) {
        return violations.stream()
                .map(v -> v.getPropertyPath() + " " + v.getMessage())
                .collect(Collectors.joining("; "));
    }
}

Key points:

• validator.validate(order) checks business constraints that the schema can’t fully express, like a regex pattern for a product code.
• Error messages like quantity must be greater than 0 are appended to the next prompt. The model fixes errors better when it knows what went wrong.
• The catch block handles malformed JSON. This is rare with strict mode, but can happen with other models.
• A limit on retry attempts prevents wasting money on infinite loops.

Note: If an email truly doesn’t contain a quantity, retry only makes the model make up a number to pass validation. In this case, allow the field to accept null, then handle it at the business logic layer, e.g., ask the customer.

7. Writing tests

You shouldn’t call the real LLM in unit tests, because it’s slow, expensive, and results are unstable. What you should test is the deterministic logic: schema is generated correctly, parsing is correct, validation is correct.

class OrderRequestConversionTest {

    private final BeanOutputConverter<OrderRequest> converter =
            new BeanOutputConverter<>(OrderRequest.class);

    private final Validator validator =
            Validation.buildDefaultValidatorFactory().getValidator();

    @Test
    void schemaContainsAllFieldsAndEnumValues() {
        String schema = converter.getJsonSchema();

        assertThat(schema)
                .contains("\"customerName\"", "\"productCode\"", "\"quantity\"")
                .contains("\"STANDARD\"", "\"EXPRESS\"");
    }

    @Test
    void convertsValidJsonToRecord() {
        String json = """
                {"customerName":"Tran Thi Lan","productCode":"SP-1024","quantity":3,"speed":"EXPRESS"}
                """;

        OrderRequest order = converter.convert(json);

        assertThat(order.quantity()).isEqualTo(3);
        assertThat(order.speed()).isEqualTo(DeliverySpeed.EXPRESS);
        assertThat(validator.validate(order)).isEmpty();
    }

    @Test
    void flagsInvalidBusinessValues() {
        String json = """
                {"customerName":"Lan","productCode":"ABC","quantity":0,"speed":"STANDARD"}
                """;

        OrderRequest order = converter.convert(json);

        assertThat(validator.validate(order))
                .extracting(v -> v.getPropertyPath().toString())
                .containsExactlyInAnyOrder("productCode", "quantity");
    }
}

The third test clearly shows the separation of concerns. This JSON has the right structure to parse successfully, but fails business validation. Bean Validation catches both fields with errors.

For checking prompt quality with a real model, create a separate integration test suite. This suite runs on schedule or manually, uses a set of sample emails, and measures the extraction success rate.

8. Common errors and when not to use

Some common errors:

• Using .entity() with models that don’t follow instructions well. Small local models often return JSON missing fields. Use the model’s JSON mode or grammar constraint if available.
• Schema too deep or too wide. Dozens of nested fields increase error rates and consume tokens. Better to split into multiple smaller calls.
• Missing @JsonPropertyDescription. Field names like code or type are vague to the model. One line of description significantly improves accuracy.
• Relying entirely on strict mode. Strict mode only guarantees the shape of JSON. It doesn’t prevent made-up data.
• Keeping temperature high. Extraction results will vary between runs.
• Putting user data in the prompt without clear separation. An email could contain a line like “ignore previous instructions”. Always separate the system prompt and validate the output.

Don’t use structured output when:

• The output is naturally free-form text, like a summary or chat reply. Forcing text into JSON only degrades quality.
• Data can be extracted with regex or a standard parser. That’s cheaper, faster, and more deterministic.
• You need absolute accuracy, like transaction amounts, without human confirmation.

9. Conclusion

This article covered ways to force LLM to return JSON matching the correct schema in Spring Boot:

• ChatClient.entity() with records and enums lets you map the response to a Java type with just one line.
• BeanOutputConverter lets you see and control the generated schema.
• OpenAI’s ResponseFormat.Type.JSON_SCHEMA guarantees structure as the model generates tokens.
• Bean Validation combined with retry feedback handles constraints that schema can’t express.

Core principle: schema preserves the shape of data, validation preserves the meaning. Production applications need both layers, along with unit tests for the parsing logic and a separate evaluation suite for prompt quality.