Streaming Responses from LLM APIs with Java 21 and Spring Boot

1. Overview

A long answer from an LLM can take 10–30 seconds to generate. If you wait for the entire response before returning it, users stare at a blank screen that whole time. Streaming lets you receive each token as the model generates it, so the first characters usually appear in under a second.

This article shows you how to read a stream from an OpenAI Chat Completions-compatible API using java.net.http.HttpClient, then using Spring’s WebClient. Next, you’ll push the stream to the browser via Server-Sent Events (SSE) and write tests for the entire flow.

2. LLM API Stream Format

When you send a request with "stream": true, the API doesn’t return a single JSON object. Instead, it returns Content-Type: text/event-stream, with each event on a line starting with data::

data: {"id":"c1","choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}

data: {"id":"c1","choices":[{"index":0,"delta":{"content":"Hello"}}]}

data: {"id":"c1","choices":[{"index":0,"delta":{"content":" there"}}]}

data: {"id":"c1","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Three key points to remember:

• The text lives in choices[0].delta.content. Some chunks don’t have this field—for example, the first chunk might only have role, or the last chunk only finish_reason.
• Events are separated by a blank line.
• data: [DONE] signals the end of the stream. This value is not JSON, so don’t feed it into Jackson.

3. Separate Parsing Logic into Its Own Class

Parsing logic should live in its own class for testability. This class needs only Jackson, which is already included in every Spring Boot starter web:

package dev.nonla.chat;

import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.util.Optional;

public class ChatChunkParser {

    public static final String DONE = "[DONE]";
    private final ObjectMapper mapper = new ObjectMapper();

    /** Receives an entire SSE line, e.g.: data: {...} */
    public Optional<String> parseLine(String line) {
        if (!line.startsWith("data:")) {
            return Optional.empty();
        }
        return parsePayload(line.substring(5).trim());
    }

    /** Receives the payload part after "data:" */
    public Optional<String> parsePayload(String payload) {
        if (payload.isEmpty() || DONE.equals(payload)) {
            return Optional.empty();
        }
        try {
            JsonNode content = mapper.readTree(payload)
                    .path("choices").path(0).path("delta").path("content");
            return content.isTextual() && !content.asText().isEmpty()
                    ? Optional.of(content.asText())
                    : Optional.empty();
        } catch (JsonProcessingException e) {
            throw new IllegalStateException("Invalid chunk: " + payload, e);
        }
    }
}

How this class works:

• parseLine skips blank lines and lines like event: or : keep-alive.
• Jackson’s path(...) doesn’t throw an exception if a node is missing. If a chunk has no content, the result is Optional.empty().
• [DONE] is filtered out before calling readTree, so it doesn’t cause a parse error.

4. Streaming with java.net.http.HttpClient

If you only need a CLI tool or a simple job, the HttpClient built into the JDK is enough. BodyHandlers.ofLines() returns a Stream<String> that reads gradually as data arrives from the network, without waiting for the entire body to download:

package dev.nonla.chat;

import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.net.http.HttpResponse.BodyHandlers;
import java.time.Duration;
import java.util.Optional;
import java.util.stream.Collectors;
import java.util.stream.Stream;

public class StreamingChatDemo {

    public static void main(String[] args) throws Exception {
        String baseUrl = System.getenv("LLM_BASE_URL");
        String apiKey = System.getenv("LLM_API_KEY");

        String body = """
            {"model": "gpt-4o-mini", "stream": true,
             "messages": [{"role": "user",
                           "content": "Explain the relationship between Order and Customer in 3 sentences"}]}
            """;

        HttpRequest request = HttpRequest.newBuilder(URI.create(baseUrl + "/v1/chat/completions"))
                .header("Authorization", "Bearer " + apiKey)
                .header("Content-Type", "application/json")
                .header("Accept", "text/event-stream")
                .timeout(Duration.ofSeconds(60))
                .POST(HttpRequest.BodyPublishers.ofString(body))
                .build();

        HttpClient client = HttpClient.newHttpClient();
        HttpResponse<Stream<String>> response = client.send(request, BodyHandlers.ofLines());

        if (response.statusCode() != 200) {
            String error = response.body().collect(Collectors.joining("\n"));
            throw new IllegalStateException("HTTP " + response.statusCode() + ": " + error);
        }

        ChatChunkParser parser = new ChatChunkParser();
        try (Stream<String> lines = response.body()) {
            lines.takeWhile(line -> !line.equals("data: [DONE]"))
                 .map(parser::parseLine)
                 .flatMap(Optional::stream)
                 .forEach(token -> {
                     System.out.print(token);
                     System.out.flush();
                 });
        }
        System.out.println();
    }
}

Explanation of the key lines:

• Status code must be checked first. When the API returns an error (401, 429…), the body is error JSON, not SSE.
• takeWhile stops the stream when it encounters [DONE]. The try-with-resources block closes the stream, and the connection is released.
• System.out.flush() makes text appear immediately. Without it, the terminal might buffer the output before printing.

The result is text appearing gradually on the terminal:

An Order belongs to exactly one Customer, while a Customer can have multiple Orders...

Note: The timeout(...) on HttpRequest only applies until the response headers arrive. It doesn’t limit how long you read the body. If the model “hangs” partway through, the loop will wait forever. This is one reason to switch to WebClient in a real application.

5. Streaming with Spring WebClient

In a Spring Boot service, WebClient gives you a Flux<String> with support for backpressure, timeout, and cancellation. You need to add a dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>

Connection configuration goes in application.properties:

llm.base-url=${LLM_BASE_URL}
llm.api-key=${LLM_API_KEY}
llm.model=gpt-4o-mini

Next, declare a WebClient bean:

@Configuration
class LlmClientConfig {

    @Bean
    WebClient llmWebClient(@Value("${llm.base-url}") String baseUrl,
                           @Value("${llm.api-key}") String apiKey) {
        return WebClient.builder()
                .baseUrl(baseUrl)
                .defaultHeader(HttpHeaders.AUTHORIZATION, "Bearer " + apiKey)
                .build();
    }
}

The service calls the API and returns a Flux containing tokens:

@Service
public class ChatStreamService {

    private static final Logger log = LoggerFactory.getLogger(ChatStreamService.class);
    private static final ParameterizedTypeReference<ServerSentEvent<String>> SSE_TYPE =
            new ParameterizedTypeReference<>() {};

    private final WebClient llmWebClient;
    private final String model;
    private final ChatChunkParser parser = new ChatChunkParser();

    public ChatStreamService(WebClient llmWebClient, @Value("${llm.model}") String model) {
        this.llmWebClient = llmWebClient;
        this.model = model;
    }

    public Flux<String> stream(String question) {
        Map<String, Object> body = Map.of(
                "model", model,
                "stream", true,
                "messages", List.of(Map.of("role", "user", "content", question)));

        return llmWebClient.post()
                .uri("/v1/chat/completions")
                .contentType(MediaType.APPLICATION_JSON)
                .accept(MediaType.TEXT_EVENT_STREAM)
                .bodyValue(body)
                .retrieve()
                .bodyToFlux(SSE_TYPE)
                .mapNotNull(ServerSentEvent::data)
                .takeWhile(data -> !ChatChunkParser.DONE.equals(data))
                .mapNotNull(data -> parser.parsePayload(data).orElse(null))
                .timeout(Duration.ofSeconds(30))
                .doOnCancel(() -> log.info("Client cancelled stream for question: {}", question));
    }
}

Each step in the processing chain:

• bodyToFlux(SSE_TYPE) lets Spring automatically split events by blank lines and strip the data: prefix. You don’t handle buffering yourself.
• takeWhile completes the Flux when it encounters [DONE].
• timeout(Duration.ofSeconds(30)) applies to the gap between consecutive elements, including the first one. If no token arrives within 30 seconds, the Flux raises TimeoutException. This is what HttpClient in section 4 lacks.
• When the API returns 4xx/5xx, retrieve() throws WebClientResponseException before any tokens arrive.

5.1. Save the complete response after streaming finishes

You often need to save the full response for history or cost calculation. The StringBuilder must be created fresh for each subscription. Wrap the processing chain in Flux.defer:

public Flux<String> streamAndRecord(String question) {
    return Flux.defer(() -> {
        StringBuilder full = new StringBuilder();
        return stream(question)
                .doOnNext(full::append)
                .doOnComplete(() -> log.info("Complete response ({} characters): {}",
                        full.length(), full));
    });
}

If StringBuilder were declared outside defer, two concurrent requests would write to the same buffer. This bug is hard to spot in manual testing because you usually only send one request at a time.

6. Push the stream to the browser

A controller returns a Flux<ServerSentEvent<...>> with produces = text/event-stream:

public record ChatChunk(String text) {}

@RestController
class ChatController {

    private final ChatStreamService chatStreamService;

    ChatController(ChatStreamService chatStreamService) {
        this.chatStreamService = chatStreamService;
    }

    @GetMapping(path = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    Flux<ServerSentEvent<ChatChunk>> stream(@RequestParam("q") String question) {
        return chatStreamService.streamAndRecord(question)
                .map(token -> ServerSentEvent.builder(new ChatChunk(token)).event("token").build())
                .concatWith(Mono.just(ServerSentEvent.builder(new ChatChunk("")).event("done").build()))
                .onErrorResume(ex -> Mono.just(ServerSentEvent
                        .builder(new ChatChunk("Model is busy, please try again."))
                        .event("error").build()));
    }
}

You can test the endpoint with curl -N. The -N flag disables curl’s buffering:

curl -N "http://localhost:8080/api/chat/stream?q=What%20is%20an%20Order"

event:token
data:{"text":"Order"}

event:token
data:{"text":" is"}

event:token
data:{"text":" a"}

event:done
data:{"text":""}

Three design decisions stand out:

• Tokens are wrapped in JSON (ChatChunk), not sent as plain strings. According to the SSE spec, the receiver discards one space right after data:. Sending is as plain text becomes is in the browser, and words get stuck together. Tokens containing \n also split into multiple data: lines. Wrapping in JSON avoids both problems.
• Errors are sent as an error event. Once the first byte goes out, status 200 is in the headers. You can’t switch to 500. The client must listen for the error event.
• A done event is sent explicitly. The browser’s EventSource auto-reconnects when a stream closes. The client needs the done event to know the response is finished and call close() itself.

When a user closes the tab, WebFlux cancels the subscription. The cancellation signal flows back up to WebClient, Reactor Netty closes the connection to the LLM API, and the provider stops generating tokens. This way you don’t pay for response text nobody read.

7. Write tests

7.1. Test the parser

class ChatChunkParserTest {

    private final ChatChunkParser parser = new ChatChunkParser();

    @Test
    void givenContentChunk_whenParseLine_thenReturnsToken() {
        String line = "data: {\"choices\":[{\"index\":0,\"delta\":{\"content\":\" hello\"}}]}";

        assertThat(parser.parseLine(line)).contains(" hello");
    }

    @Test
    void givenDoneOrRoleOnlyChunk_whenParseLine_thenReturnsEmpty() {
        assertThat(parser.parseLine("data: [DONE]")).isEmpty();
        assertThat(parser.parseLine(
                "data: {\"choices\":[{\"delta\":{\"role\":\"assistant\"}}]}")).isEmpty();
        assertThat(parser.parseLine(": keep-alive")).isEmpty();
    }

    @Test
    void givenBrokenJson_whenParseLine_thenThrows() {
        assertThatThrownBy(() -> parser.parseLine("data: {oops"))
                .isInstanceOf(IllegalStateException.class)
                .hasMessageContaining("Invalid chunk");
    }
}

The first test confirms that leading whitespace in a token is preserved. This is the trickiest detail in the entire flow.

7.2. Test the controller with WebTestClient

Mock ChatStreamService so the test doesn’t call the real API. @MockitoBean has been available since Spring Boot 3.4. On older versions, use @MockBean.

@WebFluxTest(ChatController.class)
class ChatControllerTest {

    @Autowired
    WebTestClient webTestClient;

    @MockitoBean
    ChatStreamService chatStreamService;

    @Test
    void whenStreaming_thenSendsTokensThenDone() {
        when(chatStreamService.streamAndRecord("hi"))
                .thenReturn(Flux.just("Hello", " there", "!"));

        List<ServerSentEvent<String>> events = webTestClient.get()
                .uri("/api/chat/stream?q=hi")
                .accept(MediaType.TEXT_EVENT_STREAM)
                .exchange()
                .expectStatus().isOk()
                .returnResult(new ParameterizedTypeReference<ServerSentEvent<String>>() {})
                .getResponseBody()
                .collectList()
                .block(Duration.ofSeconds(5));

        assertThat(events).extracting(ServerSentEvent::event)
                .containsExactly("token", "token", "token", "done");
        assertThat(events.get(1).data()).isEqualTo("{\"text\":\" there\"}");
    }

    @Test
    void whenUpstreamFails_thenSendsErrorEvent() {
        when(chatStreamService.streamAndRecord("hi"))
                .thenReturn(Flux.concat(Flux.just("Hello"),
                        Flux.error(new TimeoutException())));

        List<ServerSentEvent<String>> events = webTestClient.get()
                .uri("/api/chat/stream?q=hi")
                .accept(MediaType.TEXT_EVENT_STREAM)
                .exchange()
                .returnResult(new ParameterizedTypeReference<ServerSentEvent<String>>() {})
                .getResponseBody()
                .collectList()
                .block(Duration.ofSeconds(5));

        assertThat(events).extracting(ServerSentEvent::event)
                .containsExactly("token", "error");
    }
}

The second test simulates a failure midway through. The client receives the tokens already generated, then an error event, and the response still carries status 200.

8. Common pitfalls and when not to stream

• Using bodyToMono(String.class). This buffers the entire body before returning, so you lose all the benefits of streaming.
• Reverse proxy buffering the response. Nginx buffers responses by default, and the client receives all tokens at once at the end. You need to disable proxy_buffering for this route, or add the X-Accel-Buffering: no header. Gzip compression at the proxy layer can also cause buffering.
• Using .block() in a Spring MVC controller. If your project uses spring-boot-starter-web (Servlet), you can still return a Flux or use SseEmitter. Don’t block() and write piece by piece—each request would hold a thread for 30 seconds.
• Displaying structured output before you have all of it. If you ask the model to return JSON to parse, or to invoke tools, incomplete chunks won’t parse. You can still stream in this case, but you must collect enough before validating. If you don’t need gradual display, a regular (non-streaming) request is simpler.
• Computing token usage. By default, streamed chunks don’t include a usage field. Many APIs support an option like "stream_options": {"include_usage": true} to send an extra chunk at the end with usage and empty choices. The parser from section 3 safely skips this chunk because path(0) doesn’t exist.

Note: Don’t auto-retry once you’ve sent part of a token to the user. Retrying generates a different response, which concatenates with the old part into garbled content. Only retry if the error happens before the first token, like when retrieve() gets 429 or 503.

9. Conclusion

Streaming responses from an LLM API is essentially reading Server-Sent Events. You read each data: line, extract choices[0].delta.content, and stop when you hit [DONE]. The JDK’s HttpClient is enough for small tools. In a Spring Boot service, WebClient combined with Flux gives you timeouts between tokens and the ability to cancel all the way upstream.

When pushing the stream to the browser, wrap tokens in JSON to preserve spaces and newlines. Send explicit done and error events, because the status code is fixed the moment the first byte goes out. Finally, check your reverse proxy to ensure it’s not silently buffering the entire response.