How LLMs Work: tokens, context window and sampling

1. Overview

When calling an LLM API, you’ll encounter parameters like max_tokens, temperature, top_p, and errors like “context length exceeded”. These parameters become clear once you understand how the model generates text.

This article explains three foundational concepts: token, context window, and sampling. For each concept, you’ll get runnable Java 21 code. After reading, you’ll be able to count tokens before calling the API, trim chat history to fit within the context window, and choose the right temperature or top_p for your use case.

2. The core loop: predicting the next token

LLMs don’t generate an entire response at once. They run a simple loop:

1. Take an input sequence of tokens.
2. Compute a logit (raw score) for every token in the vocabulary, typically tens of thousands to over a hundred thousand tokens.
3. Convert logits to probabilities, then choose a token. This choice step is called sampling.
4. Append the chosen token to the input and go back to step 1.

The loop stops when the model generates an end token or reaches the max_tokens limit. This loop leads to three key insights:

• The model sees text as tokens, not characters or words.
• Everything the model “remembers” in one call must fit in the input. This input is bounded by the context window.
• The response is the result of many controlled random choices. So the same prompt can produce different results.

3. Tokens

3.1. What is a token?

A tokenizer splits text into small pieces and maps each piece to an integer. Most modern models use a variant of Byte Pair Encoding (BPE). Character sequences that appear frequently in training data are merged into a single token. Rare sequences are split into smaller tokens, sometimes down to individual bytes.

So a common English word like hello typically costs just one token. Conversely, Vietnamese with diacritical marks typically costs more tokens for the same amount of content, because combinations like “ượ” or “ững” appear less frequently in the tokenizer’s training data.

3.2. Counting tokens with JTokkit

To count tokens in Java, you can use JTokkit. This library implements OpenAI’s BPE encodings like cl100k_base and o200k_base:

<dependency>
    <groupId>com.knuddels</groupId>
    <artifactId>jtokkit</artifactId>
    <version>1.1.0</version>
</dependency>

The following code compares token counts for English and Vietnamese sentences:

import com.knuddels.jtokkit.Encodings;
import com.knuddels.jtokkit.api.Encoding;
import com.knuddels.jtokkit.api.EncodingType;
import com.knuddels.jtokkit.api.IntArrayList;

public class TokenCounterDemo {

    public static void main(String[] args) {
        Encoding encoding = Encodings.newDefaultEncodingRegistry()
                .getEncoding(EncodingType.CL100K_BASE);

        String english = "Hello world";
        String vietnamese = "Đơn hàng của bạn đã được giao thành công";

        IntArrayList ids = encoding.encode(english);
        System.out.println(english + " -> " + ids.size() + " tokens: " + ids);

        int viTokens = encoding.countTokens(vietnamese);
        System.out.printf("%s -> %d tokens, %d characters%n",
                vietnamese, viTokens, vietnamese.length());
    }
}

Explanation:

• getEncoding(EncodingType.CL100K_BASE) retrieves the vocabulary that early GPT-4 models use.
• encode(...) returns a list of token IDs. countTokens(...) returns only the count, so it’s faster when you don’t need the IDs.
• For "Hello world", the result is 2 tokens: Hello and world. The space is merged into the following token.

Output looks like:

Hello world -> 2 tokens: [9906, 1917]
Đơn hàng của bạn đã được giao thành công -> ... tokens, 40 characters

Run it yourself and see how the Vietnamese sentence costs significantly more tokens than its number of words. This is important to account for when estimating costs, since providers charge by token.

Note: Each model family uses its own tokenizer. Claude, Gemini, Llama, and GPT don’t split tokens the same way. JTokkit is accurate only for OpenAI’s encoding. For other models, use the provider’s tokenizer or token-counting endpoint, or treat JTokkit numbers as estimates and add a safety margin.

4. Context window

4.1. What the context window contains

The context window is the maximum number of tokens a model can process in one call. This number counts both input and output:

system prompt + chat history + attached documents + question + response ≤ context window

There are three common misunderstandings:

• “The model remembers previous conversations.” Wrong. The API doesn’t maintain state between calls. Chat applications must resend the entire history each time, so each subsequent call costs more tokens than the previous one.
• “Longer context is always better.” Not necessarily. Cost and latency increase with input tokens. Models also tend to miss information in the middle of very long contexts, a phenomenon often called “lost in the middle”.
• “Fitting input within context window is enough.” Not quite. You also need to reserve space for the output. If input takes up almost all of it, the response will be truncated.

4.2. Trimming chat history to fit a token budget

A simple and common strategy is: always keep the system prompt, then take messages from newest to oldest until you run out of budget.

import com.knuddels.jtokkit.api.Encoding;

import java.util.ArrayDeque;
import java.util.Deque;
import java.util.List;

public record ChatMessage(String role, String content) {}

public class ContextWindowManager {

    // Each message costs a few extra tokens for role and separators
    private static final int PER_MESSAGE_OVERHEAD = 4;

    private final Encoding encoding;
    private final int contextWindow;
    private final int reservedForOutput;

    public ContextWindowManager(Encoding encoding, int contextWindow, int reservedForOutput) {
        this.encoding = encoding;
        this.contextWindow = contextWindow;
        this.reservedForOutput = reservedForOutput;
    }

    public int countTokens(ChatMessage message) {
        return encoding.countTokens(message.content()) + PER_MESSAGE_OVERHEAD;
    }

    public List<ChatMessage> fit(ChatMessage systemPrompt, List<ChatMessage> history) {
        int budget = contextWindow - reservedForOutput - countTokens(systemPrompt);
        if (budget < 0) {
            throw new IllegalStateException("System prompt is too long for the context window");
        }

        Deque<ChatMessage> kept = new ArrayDeque<>();
        for (int i = history.size() - 1; i >= 0; i--) {
            int cost = countTokens(history.get(i));
            if (cost > budget) {
                break;
            }
            kept.addFirst(history.get(i));
            budget -= cost;
        }
        kept.addFirst(systemPrompt);
        return List.copyOf(kept);
    }
}

Explanation:

• reservedForOutput is space reserved for the response. This value should match the max_tokens you send to the API.
• The loop goes backward from the end of the list, because recent messages are usually most relevant to the current question.
• When a message doesn’t fit, stop immediately with break. Don’t skip it to find older ones. If you skip it, the conversation becomes fragmented and the model may misunderstand the context.
• addFirst preserves chronological order in the returned result.

In practice, you can combine two more techniques: summarizing older messages into a short passage, or using RAG to include only document excerpts relevant to the question instead of entire documents.

5. Sampling

5.1. From logits to probabilities: softmax and temperature

Suppose the model needs to complete “The capital of Vietnam is…” and has four candidates with these logits: Hanoi = 4.0, Hue = 2.5, Da Nang = 2.0, Saigon = 1.0.

To convert logits to probabilities, the model uses softmax. Before computing softmax, logits are divided by temperature (T):

• T < 1 makes the distribution “sharper”. High-scoring tokens are much more likely to be chosen.
• T > 1 makes the distribution “flatter”. Low-scoring tokens get better chances.

Here’s a class that implements softmax, temperature, top-k, top-p, and token selection:

import java.util.*;

public record Candidate(String token, double logit) {}

public class TokenSampler {

    private final Random random;

    public TokenSampler(long seed) {
        this.random = new Random(seed);
    }

    public static Map<String, Double> softmax(List<Candidate> candidates, double temperature) {
        if (temperature <= 0) {
            throw new IllegalArgumentException("temperature must be > 0, use greedy() for T = 0");
        }
        double max = candidates.stream().mapToDouble(Candidate::logit).max().orElseThrow();
        Map<String, Double> exps = new LinkedHashMap<>();
        double sum = 0;
        for (Candidate c : candidates) {
            double e = Math.exp((c.logit() - max) / temperature);
            exps.put(c.token(), e);
            sum += e;
        }
        Map<String, Double> probs = new LinkedHashMap<>();
        for (var entry : exps.entrySet()) {
            probs.put(entry.getKey(), entry.getValue() / sum);
        }
        return sortDesc(probs);
    }

    public static Map<String, Double> topK(Map<String, Double> probs, int k) {
        Map<String, Double> kept = new LinkedHashMap<>();
        sortDesc(probs).entrySet().stream().limit(k)
                .forEach(e -> kept.put(e.getKey(), e.getValue()));
        return normalize(kept);
    }

    public static Map<String, Double> topP(Map<String, Double> probs, double p) {
        Map<String, Double> kept = new LinkedHashMap<>();
        double cumulative = 0;
        for (var entry : sortDesc(probs).entrySet()) {
            kept.put(entry.getKey(), entry.getValue());
            cumulative += entry.getValue();
            if (cumulative >= p) {
                break;
            }
        }
        return normalize(kept);
    }

    public static String greedy(List<Candidate> candidates) {
        return candidates.stream()
                .max(Comparator.comparingDouble(Candidate::logit))
                .orElseThrow().token();
    }

    public String sample(Map<String, Double> probs) {
        double r = random.nextDouble();
        double cumulative = 0;
        String last = null;
        for (var entry : probs.entrySet()) {
            cumulative += entry.getValue();
            last = entry.getKey();
            if (r < cumulative) {
                return last;
            }
        }
        return last; // guard against floating-point rounding
    }

    private static Map<String, Double> normalize(Map<String, Double> probs) {
        double sum = probs.values().stream().mapToDouble(Double::doubleValue).sum();
        Map<String, Double> result = new LinkedHashMap<>();
        probs.forEach((token, prob) -> result.put(token, prob / sum));
        return result;
    }

    private static Map<String, Double> sortDesc(Map<String, Double> probs) {
        Map<String, Double> sorted = new LinkedHashMap<>();
        probs.entrySet().stream()
                .sorted(Map.Entry.<String, Double>comparingByValue().reversed())
                .forEach(e -> sorted.put(e.getKey(), e.getValue()));
        return sorted;
    }
}

Key parts explained:

• In softmax, each logit is first subtracted by the maximum to avoid overflow when calling Math.exp. The mathematical result is unchanged, but it prevents overflow for large logits.
• Both topK and topP renormalize probabilities to sum to 1 after removing candidates.
• sample draws a random number in [0, 1), then accumulates probabilities. The candidate that makes the cumulative sum exceed this number gets chosen.

Try printing probabilities at three temperature levels:

var candidates = List.of(
        new Candidate("Hanoi", 4.0), new Candidate("Hue", 2.5),
        new Candidate("Da Nang", 2.0), new Candidate("Saigon", 1.0));

for (double t : new double[]{0.5, 1.0, 2.0}) {
    System.out.println("T=" + t + " " + TokenSampler.softmax(candidates, t));
}

Output (rounded):

T=0.5 {Hanoi=0.934, Hue=0.047, Da Nang=0.017, Saigon=0.002}
T=1.0 {Hanoi=0.710, Hue=0.158, Da Nang=0.096, Saigon=0.035}
T=2.0 {Hanoi=0.485, Hue=0.229, Da Nang=0.178, Saigon=0.108}

At T = 2.0, about 1 in 9 times the model will answer “Saigon”. For a factual question, that’s a wrong answer. This example shows why high temperature easily leads to hallucination.

Temperature = 0 is commonly understood as greedy decoding: always pick the token with the highest logit, equivalent to the greedy() function.

5.2. Top-k and top-p

Both techniques trim the “long tail” of low-probability tokens before selection:

• Top-k: keep only the k tokens with highest probability. With k = 2, only Hanoi and Hue remain.
• Top-p (nucleus sampling): keep the smallest set of tokens whose cumulative probability ≥ p. At T = 1.0 and p = 0.9, cumulative probabilities are 0.710, then 0.868, then 0.964. So top-p keeps 3 tokens and excludes Saigon.

Top-p is more flexible than top-k. When the model is very confident, top-p keeps only a few tokens. When it’s uncertain, it keeps more. Top-k always keeps exactly k tokens regardless of the distribution shape.

Note: Many providers recommend tuning only one of temperature or top_p, not both at the same time. The reason is their effects compound and results are hard to predict.

5.3. Testing sampling behavior

Dependencies for testing (Spring Boot 3.x already includes this in spring-boot-starter-test):

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>

import org.junit.jupiter.api.Test;

import java.util.List;
import java.util.stream.IntStream;

import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.within;

class TokenSamplerTest {

    private final List<Candidate> candidates = List.of(
            new Candidate("Hanoi", 4.0), new Candidate("Hue", 2.5),
            new Candidate("Da Nang", 2.0), new Candidate("Saigon", 1.0));

    @Test
    void lowerTemperatureMakesTopTokenMoreLikely() {
        double cold = TokenSampler.softmax(candidates, 0.5).get("Hanoi");
        double hot = TokenSampler.softmax(candidates, 2.0).get("Hanoi");

        assertThat(cold).isCloseTo(0.934, within(0.001));
        assertThat(cold).isGreaterThan(hot);
    }

    @Test
    void topPKeepsSmallestNucleus() {
        var nucleus = TokenSampler.topP(TokenSampler.softmax(candidates, 1.0), 0.9);

        assertThat(nucleus).containsOnlyKeys("Hanoi", "Hue", "Da Nang");
        assertThat(nucleus.values().stream().mapToDouble(Double::doubleValue).sum())
                .isCloseTo(1.0, within(1e-9));
    }

    @Test
    void sameSeedGivesSameSequence() {
        var probs = TokenSampler.softmax(candidates, 1.0);
        var first = new TokenSampler(42L);
        var second = new TokenSampler(42L);

        List<String> a = IntStream.range(0, 20).mapToObj(i -> first.sample(probs)).toList();
        List<String> b = IntStream.range(0, 20).mapToObj(i -> second.sample(probs)).toList();

        assertThat(a).isEqualTo(b);
    }

    @Test
    void greedyAlwaysPicksHighestLogit() {
        assertThat(TokenSampler.greedy(candidates)).isEqualTo("Hanoi");
    }
}

The last test makes an important point: the same seed produces the same sequence of results. Some APIs have a seed parameter, but most only offer “best-effort” reproducibility. Parallel computation on GPUs and batching multiple requests can cause slight variations in logits. So even with temperature = 0, you should not write tests asserting exact word-for-word matches with real LLM output.

6. Choosing parameters in practice

Use case	Recommendation
Data extraction, classification, JSON generation	temperature 0–0.2
Question answering, code generation	temperature 0.2–0.5
Creative writing, brainstorming	temperature 0.7–1.0 or top_p 0.9–0.95

Some common pitfalls:

• Setting max_tokens too low: JSON gets truncated mid-structure and won’t parse. Check the stopping reason in the response (often finish_reason or stop_reason) before processing.
• Not counting tokens before sending: requests get rejected as chat history grows over time.
• Using high temperature for accuracy-critical tasks: the model “makes up” facts that don’t exist.
• Assuming temperature = 0 guarantees correctness: greedy picks only the highest-probability token. If the model misunderstands from the start, it will be consistently wrong.

7. Conclusion

This article covered three foundational LLM concepts. Tokens are the units the model reads and the units providers charge for. Vietnamese typically costs more tokens than English for equivalent content. Context window limits the total tokens of input and output in one call, so you must actively count and trim history. Sampling determines how to choose the next token from a probability distribution. temperature changes the “sharpness” of the distribution, while top-k and top-p trim the low-probability tail.

Once you understand the “predict token, choose token, append to input” loop, you can explain most LLM behaviors you encounter when working with APIs, from truncated responses to varying results between calls.