GPT-Live Listen and Speak — How It Works and Java Integration

1. Overview

“GPT-Live” is a common term for LLM models capable of listening and speaking simultaneously through a single audio streaming channel, rather than running sequentially Speech-to-Text → LLM → Text-to-Speech. This article explains how that full-duplex mechanism works at the protocol level, then builds a Spring Boot example connecting over WebSocket, sending audio, receiving streaming responses, and handling scenarios where users interrupt (barge-in). After reading, you’ll understand enough to write your own Java client calling a real-time voice API and know the pitfalls of bringing this feature to production.

2. Why the Old Pipeline (STT → LLM → TTS) Isn’t “Live”

Previously, voice assistants typically combined 3 separate models:

Mic → [Speech-to-Text] → text → [LLM] → text → [Text-to-Speech] → Speaker

Each step waits for the previous one to complete before running. Cumulative latency (typically 2–5 seconds), and users cannot interrupt during the response because the system isn’t “listening” while speaking.

GPT-Live merges all three roles into a single multimodal model, receiving audio frames directly and producing audio frames through a continuous bidirectional (duplex) connection. There’s no “wait for the sentence to finish” step — the model processes each small audio chunk as soon as it arrives, and can begin producing a response before the user finishes their sentence completely if semantic confidence is high enough.

Note: “Live” here doesn’t mean zero latency, but rather latency distributed across chunks (typically 20–100ms) instead of concentrated at the end of the pipeline.

3. Full-Duplex Architecture: Event Stream Over WebSocket

Unlike REST API (single request-response), real-time APIs use WebSocket to keep the connection open and exchange events in JSON format. Two data streams run concurrently:

• Client → server: audio chunk (base64 PCM16), control events (start/end of turn).
• Server → client: response audio chunk, partial transcript, event indicating “model detected you’re speaking” (to self-interrupt output).

Client                         Server (GPT-Live)
  |--- audio.append (chunk1) -->|
  |--- audio.append (chunk2) -->|
  |                             |-- transcript.delta (partial) -->
  |--- audio.append (chunk3) -->|
  |<------ response.audio.delta -|  (model has started responding)
  |--- audio.append (interrupt) ->|
  |<------ response.cancelled ---|  (server cancels response)

Having two audio streams run in parallel on a single socket is the essence of “listening and speaking simultaneously.”

4. Setting Up WebSocket Connection Dependencies in Spring Boot

Spring Boot has built-in WebSocket client support via spring-websocket. Add the dependencies:

<dependency>
    <groupId>org.springframework</groupId>
    <artifactId>spring-websocket</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-json</artifactId>
</dependency>

Create configuration to connect to a real-time endpoint (example endpoint wss://api.gptlive.example/v1/voice):

@Configuration
public class VoiceSessionConfig {

    @Bean
    public StandardWebSocketClient webSocketClient() {
        return new StandardWebSocketClient();
    }
}

StandardWebSocketClient is Spring’s standard implementation, sufficient for client-side use without needing a servlet container.

5. Sending Audio Chunks and Receiving Streaming Responses

Core idea: each time the microphone captures an audio segment (e.g., 20ms), immediately send an audio.append event to the server without waiting for a complete sentence.

public class VoiceSessionHandler extends TextWebSocketHandler {

    private final ObjectMapper mapper = new ObjectMapper();

    public void sendAudioChunk(WebSocketSession session, byte[] pcmChunk) throws IOException {
        String base64Audio = Base64.getEncoder().encodeToString(pcmChunk);
        Map<String, Object> event = Map.of(
            "type", "input_audio.append",
            "audio", base64Audio
        );
        session.sendMessage(new TextMessage(mapper.writeValueAsString(event)));
    }

    @Override
    protected void handleTextMessage(WebSocketSession session, TextMessage message) throws Exception {
        JsonNode event = mapper.readTree(message.getPayload());
        String type = event.get("type").asText();

        switch (type) {
            case "response.audio.delta" -> playAudioChunk(event.get("audio").asText());
            case "response.text.delta" -> System.out.print(event.get("delta").asText());
            case "input_audio.speech_started" -> cancelOngoingPlayback();
            default -> { /* skip events we don't need to handle */ }
        }
    }

    private void playAudioChunk(String base64Audio) {
        byte[] pcm = Base64.getDecoder().decode(base64Audio);
        // push pcm into the speaker's audio output buffer
    }

    private void cancelOngoingPlayback() {
        // stop playing audio — this is the barge-in mechanism
    }
}

Explanation:

• sendAudioChunk is called continuously from the mic recording thread (e.g., every 20ms), without batching to keep latency low.
• handleTextMessage is the callback receiving events from the server; response.audio.delta means the model is speaking, while input_audio.speech_started means the server detected the user starting to speak — at this point the client must stop the speaker immediately.
• event.get("type").asText() uses Jackson’s JsonNode to read the type field without needing to define a rigid DTO — useful because the event schema may expand with new types.

6. Barge-in Mechanism: Who Gets to Speak and When?

Barge-in is the ability for users to interrupt the AI mid-sentence. To enable this, the server needs a voice activity detector (Voice Activity Detection—VAD) running continuously on the incoming audio stream, independent of whether the server is currently playing back audio.

public class TurnStateMachine {

    public enum State { IDLE, USER_SPEAKING, MODEL_SPEAKING }

    private volatile State state = State.IDLE;

    public void onServerEvent(String eventType) {
        state = switch (eventType) {
            case "input_audio.speech_started" -> {
                // user speaking while model is speaking → interrupt
                if (state == State.MODEL_SPEAKING) {
                    stopPlaybackImmediately();
                }
                yield State.USER_SPEAKING;
            }
            case "input_audio.speech_stopped" -> State.IDLE;
            case "response.audio.delta" -> State.MODEL_SPEAKING;
            case "response.done" -> State.IDLE;
            default -> state;
        };
    }

    private void stopPlaybackImmediately() {
        System.out.println("Stop playback, send response.cancel to server");
    }
}

Explanation: TurnStateMachine tracks who currently “has the mic” with 3 states. When input_audio.speech_started arrives during MODEL_SPEAKING, that means the user has interrupted — the client must stop the speaker immediately and notify the server to cancel the response being generated (send a response.cancel event) to prevent old and new audio from overlapping.

Note: If you don’t handle barge-in, the AI will speak over the user — the worst experience for a voice assistant. This is often overlooked in quick demos but mandatory for production.

7. Managing Session and Context in Voice Conversations

Because the connection is stateful (persisted throughout the call), you need a session object to store conversation history and voice configuration, separate from transport logic.

public class VoiceSession {

    private final String sessionId;
    private final List<ConversationTurn> history = new ArrayList<>();
    private final String voice; // e.g., "alloy", "verse"

    public VoiceSession(String sessionId, String voice) {
        this.sessionId = sessionId;
        this.voice = voice;
    }

    public void appendTurn(String role, String transcript) {
        history.add(new ConversationTurn(role, transcript, Instant.now()));
    }

    public record ConversationTurn(String role, String transcript, Instant at) {}
}

When initializing a session, you send a session.update event to set the voice and system instructions before starting audio recording:

Map<String, Object> sessionUpdate = Map.of(
    "type", "session.update",
    "session", Map.of(
        "voice", "alloy",
        "instructions", "You are a scheduling assistant, keep responses concise.",
        "turn_detection", Map.of("type", "server_vad")
    )
);

turn_detection: server_vad means you delegate detection of when the user finishes speaking to the server using VAD, instead of the client guessing from silence — this approach is more accurate because the server can use semantic signals too (e.g., a grammatically incomplete sentence isn’t yet considered finished).

8. Unit Testing Turn-Taking Logic with JUnit

The transport layer (actual WebSocket) is hard to unit test, but the state machine controlling turns can be tested completely independently.

class TurnStateMachineTest {

    @Test
    void whenModelIsSpeakingAndUserStartsSpeaking_thenMustStopPlayback() {
        TurnStateMachine machine = new TurnStateMachine();

        machine.onServerEvent("response.audio.delta");
        assertThat(machine.getState()).isEqualTo(TurnStateMachine.State.MODEL_SPEAKING);

        machine.onServerEvent("input_audio.speech_started");
        assertThat(machine.getState()).isEqualTo(TurnStateMachine.State.USER_SPEAKING);
    }

    @Test
    void whenNobodyIsSpeaking_thenStateIsIdle() {
        TurnStateMachine machine = new TurnStateMachine();

        machine.onServerEvent("response.done");

        assertThat(machine.getState()).isEqualTo(TurnStateMachine.State.IDLE);
    }
}

(Assuming you add a getState() getter to TurnStateMachine.) The first test confirms state transitions happen in the right order when there’s an interruption; the second test confirms the system returns to IDLE after the response completes — this is the condition for the next conversation loop to start cleanly.

9. Common Mistakes and When Not to Use This

• Sending audio in large batches (e.g., once every 2 seconds): eliminates all streaming benefits, latency becomes like the old pipeline again. Always send small chunks (20–100ms).
• Not handling the response.cancel event: if you only stop playback on the client without notifying the server to cancel, the server keeps generating extra audio, wastes costs and can desynchronize session state.
• Using client-side VAD to decide when the user finishes: easy to cut off sentences too early or too late in noisy environments; should let server VAD handle it if the API supports it, unless you need offline operation.
• Applying this to use cases that don’t need real-time interaction (e.g., generating reports, summarizing long documents): real-time voice models are much more expensive and complex than traditional STT/LLM/TTS pipelines, not worth trading off for low latency you don’t need.
• Running the WebSocket client on the main UI thread without separating recording/playback threads: risks stuttering and blocking when JSON parsing blocks the audio stream.

Note: Full-duplex voice consumes continuous bandwidth and API costs throughout the call, not just measured in text tokens. Consider limiting session duration or using timeouts when there’s no activity.

10. Conclusion

GPT-Live operates based on a persistent WebSocket connection transmitting audio chunks bidirectionally and using VAD to determine who has the turn — that’s the mechanism enabling “listen and speak simultaneously” instead of waiting sequentially like the old STT/LLM/TTS pipeline. When integrating into a Java/Spring Boot application, your main task is to manage the turn-taking state machine, handle barge-in correctly, and separate session logic from transport for easier testing. If your use case doesn’t need real-time responses, the traditional pipeline remains the simpler and cheaper choice.