Calling LLM APIs: temperature, top_p, and max_tokens

1. Overview

When calling an LLM API, besides the prompt, you also send along several text generation parameters. The three most commonly used parameters are temperature, top_p, and max_tokens. They determine whether the answer is stable or creative, and how long it is.

This article explains how each parameter works using a small Java simulation. Next, you’ll configure them with Spring AI and detect truncated responses. By the end, you’ll know which values to choose for different types of tasks.

2. How the Model Selects Tokens

At each step, the LLM computes a score (logit) for every possible next token. The softmax function converts these logits into probabilities. The model then samples a token according to that distribution.

temperature and top_p don’t change the model’s “knowledge.” They only intervene in this sampling step.

2.1. Temperature: Making the Distribution Sharper or Flatter

The softmax formula with temperature T:

p_i = exp(z_i / T) / Σ exp(z_j / T)

The following code simulates the model selecting the next token for the phrase “The conical hat color is …”:

import java.util.*;

public class SamplingDemo {

    static final List<String> TOKENS = List.of("green", "red", "yellow", "purple");
    static final double[] LOGITS = {2.0, 1.0, 0.5, 0.1};

    static double[] softmax(double[] logits, double temperature) {
        double[] exp = new double[logits.length];
        double sum = 0;
        for (int i = 0; i < logits.length; i++) {
            exp[i] = Math.exp(logits[i] / temperature);
            sum += exp[i];
        }
        for (int i = 0; i < exp.length; i++) {
            exp[i] /= sum;
        }
        return exp;
    }

    public static void main(String[] args) {
        for (double t : new double[]{0.5, 1.0, 1.5}) {
            double[] p = softmax(LOGITS, t);
            StringBuilder line = new StringBuilder("T=" + t + ": ");
            for (int i = 0; i < p.length; i++) {
                line.append(String.format(Locale.ROOT, "%s=%.2f ", TOKENS.get(i), p[i]));
            }
            System.out.println(line);
        }
    }
}

The softmax function divides each logit by temperature before taking exp. The main function prints the distribution with three temperature levels:

T=0.5: green=0.83 red=0.11 yellow=0.04 purple=0.02
T=1.0: green=0.57 red=0.21 yellow=0.13 purple=0.09
T=1.5: green=0.46 red=0.24 yellow=0.17 purple=0.13

The output clearly shows the effect of temperature:

• Low T (0.5): the “green” token accounts for 83%. The model almost always picks the most popular option. Answers are stable and predictable.
• High T (1.5): probabilities spread out more. “purple” rises from 2% to 13%. Answers are more diverse, but easier to go off-topic.
• T = 0: in theory, always picks the highest-probability token (greedy).

Note: temperature = 0 does not guarantee identical results across multiple calls. Floating-point arithmetic on GPUs and how the provider batches requests can still create minor differences. Don’t write tests that compare strings exactly with real LLM output.

2.2. Top_p: Trimming the Low-Probability Tail

top_p (nucleus sampling) works differently. The model sorts tokens by descending probability. It keeps the smallest group whose cumulative probability reaches threshold p, then only samples from that group.

Add the following method to SamplingDemo:

static List<String> nucleus(double[] probs, double topP) {
    Integer[] idx = {0, 1, 2, 3};
    Arrays.sort(idx, (a, b) -> Double.compare(probs[b], probs[a]));

    List<String> kept = new ArrayList<>();
    double cumulative = 0;
    for (int i : idx) {
        kept.add(TOKENS.get(i));
        cumulative += probs[i];
        if (cumulative >= topP) {
            break;
        }
    }
    return kept;
}

Calling nucleus(softmax(LOGITS, 1.0), 0.8) returns:

[green, red, yellow]

The sum of “green” and “red” is 0.785, not quite reaching 0.8. So “yellow” is added and the total becomes 0.914. The token “purple” is excluded entirely, no matter how high the temperature.

The key difference between the two parameters:

• temperature changes the shape of the distribution.
• top_p changes the set of candidates allowed to be chosen.

Note: Most providers recommend tuning only one of these parameters and leaving the other at its default (top_p = 1.0). Tuning both at once makes it hard to know which change caused the result.

3. Max Tokens: Limiting Output Length

max_tokens is the maximum number of tokens the model is allowed to generate in its response. It’s not a target length. The model doesn’t try to “write up to” that number. When it hits the limit, it stops immediately, even mid-sentence.

There are three points you need to understand:

1. Tokens are not words. With English, a token is approximately 3/4 of a word. Vietnamese with diacritics typically uses more tokens for the same amount of content.
2. Cost is calculated per token. max_tokens is a cap on both cost and latency.
3. The context window includes both input and output. If your prompt has already used most of the context window, very little remains for output.

When truncated, the API returns finish_reason as length instead of stop. A simplified response from an OpenAI-style API looks like:

{
  "choices": [{
    "message": { "role": "assistant", "content": "Huế conical hats are made from young palm fronds, sun-dried" },
    "finish_reason": "length"
  }],
  "usage": { "prompt_tokens": 42, "completion_tokens": 20, "total_tokens": 62 }
}

The content stops mid-way because max_tokens = 20. If your code only reads content, you’ll show the user a truncated sentence without even knowing it.

4. Configuration with Spring AI

This section uses Spring AI to call OpenAI from Spring Boot 3.x.

4.1. Dependency and Default Configuration

Add the BOM and starter to pom.xml:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-bom</artifactId>
            <version>1.0.0</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency>
    </dependencies>
</dependencyManagement>

<dependencies>
    <dependency>
        <groupId>org.springframework.ai</groupId>
        <artifactId>spring-ai-starter-model-openai</artifactId>
    </dependency>
</dependencies>

Default values for the entire application go in application.properties:

spring.ai.openai.api-key=${OPENAI_API_KEY}
spring.ai.openai.chat.options.model=gpt-4o-mini
spring.ai.openai.chat.options.temperature=0.3
spring.ai.openai.chat.options.max-tokens=300

Any request that doesn’t override these will use temperature = 0.3 and a maximum of 300 output tokens. The API key is taken from an environment variable, not hard-coded into the file.

4.2. Parameter Profiles for Different Task Types

Each task type needs a different set of parameters. Instead of scattering magic numbers throughout your code, group them in an enum:

import org.springframework.ai.openai.OpenAiChatOptions;

public enum GenerationProfile {

    EXTRACTION(0.0, 1.0, 200),      // data extraction, classification
    SUPPORT_REPLY(0.4, 1.0, 400),   // customer support replies
    MARKETING_COPY(0.9, 1.0, 600);  // product descriptions, slogans

    private final double temperature;
    private final double topP;
    private final int maxTokens;

    GenerationProfile(double temperature, double topP, int maxTokens) {
        this.temperature = temperature;
        this.topP = topP;
        this.maxTokens = maxTokens;
    }

    public OpenAiChatOptions toOptions() {
        return OpenAiChatOptions.builder()
                .temperature(temperature)
                .topP(topP)
                .maxTokens(maxTokens)
                .build();
    }
}

Each constant describes a task type. EXTRACTION needs consistent results, so it uses temperature 0. MARKETING_COPY needs diversity, so it uses 0.9. All three keep top_p = 1.0, meaning we only tune temperature as recommended in section 2.2.

4.3. Overriding Parameters per Request and Detecting Truncation

Spring Boot auto-configures the ChatClient.Builder bean. Inject it into your service:

import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.chat.model.ChatResponse;
import org.springframework.ai.chat.model.Generation;
import org.springframework.stereotype.Service;

@Service
public class ProductDescriptionService {

    private final ChatClient chatClient;

    public ProductDescriptionService(ChatClient.Builder builder) {
        this.chatClient = builder
                .defaultSystem("You are a copywriter for Nón Lá Shop. Write at most 3 sentences.")
                .build();
    }

    public String describe(String productName) {
        ChatResponse response = chatClient.prompt()
                .user(u -> u.text("Write a description for the product: {name}")
                            .param("name", productName))
                .options(GenerationProfile.MARKETING_COPY.toOptions())
                .call()
                .chatResponse();

        Generation result = response.getResult();
        String finishReason = result.getMetadata().getFinishReason();

        if ("length".equalsIgnoreCase(finishReason)) {
            throw new TruncatedResponseException(
                "Description was truncated after " + response.getMetadata().getUsage().getCompletionTokens() + " tokens");
        }
        return result.getOutput().getText();
    }
}

Let’s break this down:

• .options(...) overrides values in application.properties, only for this request.
• .chatResponse() returns the full response with metadata, instead of just the string like .content().
• getFinishReason() tells you why generation stopped. Spring AI might return LENGTH in uppercase, so the code uses equalsIgnoreCase.
• getUsage().getCompletionTokens() tells you the actual number of output tokens. This number is useful for logging and cost tracking.

The exception is defined simply as:

public class TruncatedResponseException extends RuntimeException {
    public TruncatedResponseException(String message) {
        super(message);
    }
}

When you encounter this exception, you have two options. First, you can increase max_tokens and retry. Second, you can tighten your prompt to ask for shorter responses. The second approach is usually cheaper and gives more concise results.

5. Testing

You shouldn’t test real LLM output by comparing strings. What you should test is your own logic: the parameter profiles and simulation algorithms.

import org.junit.jupiter.api.Test;
import org.springframework.ai.openai.OpenAiChatOptions;

import static org.assertj.core.api.Assertions.assertThat;

class GenerationProfileTest {

    @Test
    void extractionProfile_shouldBeDeterministicAndShort() {
        OpenAiChatOptions options = GenerationProfile.EXTRACTION.toOptions();

        assertThat(options.getTemperature()).isZero();
        assertThat(options.getTopP()).isEqualTo(1.0);
        assertThat(options.getMaxTokens()).isEqualTo(200);
    }

    @Test
    void marketingProfile_shouldBeMoreCreativeThanSupport() {
        assertThat(GenerationProfile.MARKETING_COPY.toOptions().getTemperature())
                .isGreaterThan(GenerationProfile.SUPPORT_REPLY.toOptions().getTemperature());
    }

    @Test
    void lowerTemperature_shouldConcentrateProbabilityOnTopToken() {
        double[] cold = SamplingDemo.softmax(SamplingDemo.LOGITS, 0.5);
        double[] hot = SamplingDemo.softmax(SamplingDemo.LOGITS, 1.5);

        assertThat(cold[0]).isGreaterThan(hot[0]);
        assertThat(cold[3]).isLessThan(hot[3]);
    }

    @Test
    void nucleus_shouldDropLowProbabilityTail() {
        double[] probs = SamplingDemo.softmax(SamplingDemo.LOGITS, 1.0);

        assertThat(SamplingDemo.nucleus(probs, 0.8))
                .containsExactly("green", "red", "yellow");
    }
}

The first two tests ensure no one accidentally changes parameters of an important profile. The last two tests affirm the behaviors described in section 2: low temperature concentrates probability on the top token, and top_p = 0.8 excludes the “purple” token.

6. Common Mistakes and When Not to Use

Using high temperature for tasks requiring accuracy. JSON extraction, ticket classification, and SQL generation need consistent results. Temperature 0.8 and above increase the risk of format errors and hallucinations.

Thinking low temperature prevents “hallucinations.” Low temperature just makes the model pick what it thinks is most likely. If the model doesn’t know the answer, it can still confidently give a wrong one. To reduce hallucination, you need to provide the right context (RAG) and verify the output.

Setting max_tokens too low to save money. You don’t save anything if the response gets truncated and you have to retry. Limit length with your prompt (“at most 3 sentences”), and use max_tokens only as a safety net.

Not checking finish_reason. This is the most common mistake. Users see a truncated sentence, but your logs show nothing.

Carrying parameters across to a different model. Each provider has its own conventions:

• Temperature scale differs. OpenAI uses 0–2, Anthropic uses 0–1, so the same value of 1.0 means different things.
• Reasoning models (like OpenAI’s o-series) may reject temperature and top_p.
• Some models require max_completion_tokens instead of max_tokens. Spring AI supports this via OpenAiChatOptions.builder().maxCompletionTokens(...).

Note: With reasoning models, internal “thinking” tokens count toward the output limit. If you set the limit too low, you might get empty content with finish_reason = length.

7. Conclusion

This article has covered three core parameters when calling LLM APIs:

• temperature changes how sharp or flat the probability distribution is. Low values give stable results, high values give diverse results.
• top_p limits the set of candidate tokens. You should adjust one of these two parameters, not both at once.
• max_tokens is the ceiling for both length and cost. Always check finish_reason to detect truncated responses.

With Spring AI, set defaults in application.properties and override per-request with .options(...). Grouping parameter sets in an enum like GenerationProfile makes code more readable, testable, and easier to adjust when you switch models.