Optimizing LLM Call Costs: Prompt Caching, Small Models, and Batch

1. Overview

When an LLM-powered feature moves from demo to production, cost typically becomes the first problem to solve. Bills grow with the number of tokens, and tokens grow with the number of users, prompt length, and agent call rounds.

This article presents three popular techniques to reduce costs: prompt caching, routing to smaller models, and batch processing. Each technique includes Java 21 / Spring Boot 3.x examples. After reading, you’ll know how to structure prompts to leverage cache, how to choose models for different tasks, and where to use batch processing. The article also shows when not to apply each technique.

2. Where LLM Costs Come From

Most providers charge by token, divided into three categories:

• Input tokens: system prompt, conversation history, RAG documents, tool definitions.
• Cached input tokens: the part of input read from cache, usually cheaper than regular input.
• Output tokens: content generated by the model, usually more expensive than input.

In many applications, input makes up the bulk of the cost. The reason is that each request sends the same long system prompt, the same set of tools, the same piece of documentation. We model the cost like this:

public record TokenUsage(long inputTokens, long cachedInputTokens, long outputTokens) {}

public record ModelPricing(double inputPerMillion, double cachedInputPerMillion, double outputPerMillion) {

    public double cost(TokenUsage u) {
        long uncached = u.inputTokens() - u.cachedInputTokens();
        return (uncached * inputPerMillion
                + u.cachedInputTokens() * cachedInputPerMillion
                + u.outputTokens() * outputPerMillion) / 1_000_000.0;
    }

    public static void main(String[] args) {
        // Illustrative prices in arbitrary units, NOT actual provider pricing
        var pricing = new ModelPricing(10, 1, 30);
        System.out.printf("No cache: %.3f%n", pricing.cost(new TokenUsage(20_000, 0, 500)));
        System.out.printf("With cache:    %.3f%n", pricing.cost(new TokenUsage(20_000, 18_000, 500)));
    }
}

TokenUsage contains the token count that the provider returns in each response. cost() separates regular input from cached input, then multiplies each by its unit price. With the hypothetical prices above, the result is:

No cache: 0.215
With cache:    0.053

The numbers here are for illustration only. The key takeaway is the structure: when most of the prompt repeats across requests, cache directly impacts your biggest expense. You should put the provider’s actual pricing into configuration and calculate the cost for each request in your logs.

3. Prompt Caching

3.1. Concept

Prompt caching is a mechanism on the provider’s side. When processing a prompt for the first time, the provider saves the computed state for the beginning (prefix) of that prompt. Subsequent requests with a prefix that matches exactly, character for character will reuse this state, making them cheaper and usually faster too.

Three characteristics you need to remember:

• Cache matches by prefix. If even one character differs at the beginning, the entire rest of the prompt can’t use the cache.
• Cache has a short lifetime and usually requires the prefix to be long enough. The exact threshold varies by provider.
• Some providers cache automatically. Some require you to mark where the cache cut-off is in the request.

3.2. Structuring Prompts: Static Content First, Dynamic Content Last

The most important rule is to put stable content at the beginning and changing content at the end:

import java.time.LocalDate;
import java.util.List;

public record ChatMessage(String role, String content) {}

public final class SupportPromptBuilder {

    private final String instructions;
    private final String productCatalog;

    public SupportPromptBuilder(String instructions, String productCatalog) {
        this.instructions = instructions;
        this.productCatalog = productCatalog;
    }

    public List<ChatMessage> build(String customerQuestion, LocalDate today) {
        return List.of(
                new ChatMessage("system", instructions),                          // static
                new ChatMessage("system", "Product Catalog:\n" + productCatalog), // static
                new ChatMessage("user", "Today: " + today + "\nQuestion: " + customerQuestion)); // dynamic
    }
}

The instructions and product catalog don’t change between customers, so they appear at the beginning. The current date and customer question change per request, so they appear in the last message.

A common mistake is writing "You are an assistant. It is now " + LocalDateTime.now() at the very start of the system prompt. The timestamp changes with each request, so the cache never hits. Some similar mistakes are harder to spot:

• Serializing tool definitions from a HashMap, causing key order to be unstable.
• Embedding a user name or request ID into the system prompt.
• Sorting RAG documents randomly between calls.

You can write a test to keep the prefix stable as code changes:

import static org.assertj.core.api.Assertions.assertThat;

import java.time.LocalDate;
import org.junit.jupiter.api.Test;

class SupportPromptBuilderTest {

    private final SupportPromptBuilder builder =
            new SupportPromptBuilder("You are a store support agent.", "T-shirts, Jeans, Shoes");

    @Test
    void prefixIsIdenticalAcrossDifferentRequests() {
        var first = builder.build("Is there a size M?", LocalDate.of(2024, 1, 1));
        var second = builder.build("How much is shipping?", LocalDate.of(2024, 1, 2));

        assertThat(first.subList(0, 2)).isEqualTo(second.subList(0, 2));
        assertThat(first.get(2)).isNotEqualTo(second.get(2));
    }
}

The test compares the first two messages of two different requests. If someone later injects dynamic data into the static part, the test will fail immediately.

3.3. Marking Cache Boundaries Explicitly

With providers that require it, the request usually has a field indicating “cache up to here”. For example, the JSON body might look like this. The field name and placement depend on each provider’s API:

{
  "model": "your-model",
  "system": [
    { "type": "text", "text": "You are a support agent..." },
    { "type": "text", "text": "Product Catalog: ...", "cache_control": { "type": "ephemeral" } }
  ],
  "messages": [ { "role": "user", "content": "Is there a size M?" } ]
}

The marker goes at the end of the last static content block. Everything before the marker is a candidate for caching.

3.4. Measuring Cache Hit Ratio

Responses usually return the number of tokens read from cache. You should log this number, because without it you won’t know if caching is working:

public static double cacheHitRatio(TokenUsage u) {
    return u.inputTokens() == 0 ? 0 : (double) u.cachedInputTokens() / u.inputTokens();
}

Push this ratio to Micrometer or log it as a metric. When it suddenly drops to 0 after a deployment, it’s almost certain someone changed the prefix.

4. Caching Responses on the Application Side

Prompt caching only reduces the cost of input. If the exact same question is asked repeatedly, you can skip the LLM call altogether. Add these dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<dependency>
    <groupId>com.github.ben-manes.caffeine</groupId>
    <artifactId>caffeine</artifactId>
</dependency>

spring.cache.cache-names=faq-answers
spring.cache.caffeine.spec=maximumSize=10000,expireAfterWrite=6h

public interface LlmClient {
    String complete(String model, String prompt);
}

@Service
public class FaqAnswerService {

    private final LlmClient llmClient;

    public FaqAnswerService(LlmClient llmClient) {
        this.llmClient = llmClient;
    }

    @Cacheable(cacheNames = "faq-answers", key = "#root.target.normalize(#question)")
    public String answer(String question) {
        return llmClient.complete("small-model", "Answer this FAQ question briefly: " + question);
    }

    public String normalize(String question) {
        return question.strip().toLowerCase(Locale.ROOT).replaceAll("\\s+", " ");
    }
}

You need to add @EnableCaching to a @Configuration class. The cache key is the normalized question: extra whitespace is removed and everything is lowercased. This way, “Shipping fee?” and “ shipping FEE? “ share the same result.

Note: @Cacheable works through a proxy. If another method in the same class calls answer() directly, the cache will be bypassed. Don’t cache responses with personalized data, such as a specific customer’s order status, using a key that doesn’t include the user’s identifier.

5. Routing to Smaller Models

Many tasks don’t need the most powerful model: classifying tickets, extracting fields from emails, short summaries, format validation. Smaller models are often significantly cheaper and faster. This is why “model routing” is getting attention from many teams.

A practical approach is cascade: call the small model first, check the output, and only escalate to the large model if the output is invalid.

import java.util.Set;

public class TicketClassifier {

    private static final Set<String> LABELS = Set.of("BILLING", "SHIPPING", "TECHNICAL", "OTHER");

    private final LlmClient llm;
    private final String smallModel;
    private final String largeModel;

    public TicketClassifier(LlmClient llm, String smallModel, String largeModel) {
        this.llm = llm;
        this.smallModel = smallModel;
        this.largeModel = largeModel;
    }

    public String classify(String ticket) {
        String prompt = "Classify the ticket into one of these labels " + LABELS + ". Return only the label.\nTicket: " + ticket;

        String label = llm.complete(smallModel, prompt).strip();
        if (LABELS.contains(label)) {
            return label;
        }
        return llm.complete(largeModel, prompt).strip();
    }
}

The small model processes first. If it returns a label that’s in the allowed set, we use that result immediately. If not—for example, if the model rambles—we call the large model instead. The validation step must be deterministic: parse JSON, match enum, validate schema. Don’t ask the small model if it’s “confident.”

Test it with a fake LlmClient:

import static org.assertj.core.api.Assertions.assertThat;

import java.util.ArrayList;
import java.util.List;
import org.junit.jupiter.api.Test;

class TicketClassifierTest {

    @Test
    void escalatesToLargeModelWhenSmallModelReturnsInvalidLabel() {
        List<String> calledModels = new ArrayList<>();
        LlmClient fake = (model, prompt) -> {
            calledModels.add(model);
            return model.equals("small-model") ? "Maybe billing?" : "BILLING";
        };

        var classifier = new TicketClassifier(fake, "small-model", "large-model");

        assertThat(classifier.classify("I was charged twice")).isEqualTo("BILLING");
        assertThat(calledModels).containsExactly("small-model", "large-model");
    }

    @Test
    void staysOnSmallModelWhenOutputIsValid() {
        List<String> calledModels = new ArrayList<>();
        LlmClient fake = (model, prompt) -> {
            calledModels.add(model);
            return "SHIPPING";
        };

        new TicketClassifier(fake, "small-model", "large-model").classify("Package hasn't arrived");

        assertThat(calledModels).containsExactly("small-model");
    }
}

You should put model names in configuration instead of hard-coding them. This way you can change models without rebuilding:

llm:
  models:
    small: small-model
    large: large-model

You should also monitor the escalation rate. If most requests still need to go to the large model, cascading is actually costing you more because each request calls twice.

6. Batch Processing

6.1. Combining Multiple Items in One Request

When you need to analyze many short items, such as 50 product reviews, you can combine them into a single prompt. The instruction section is sent once instead of 50 times:

public record Review(String id, String text) {}

public static String buildSentimentPrompt(List<Review> reviews) {
    var sb = new StringBuilder("""
            For each review, return one line of JSON: {"id": "...", "sentiment": "POSITIVE|NEGATIVE|NEUTRAL"}
            """);
    reviews.forEach(r -> sb.append(r.id()).append(": ").append(r.text()).append('\n'));
    return sb.toString();
}

You should keep each group at a reasonable size. A group that’s too large makes the model more likely to miss items. Also, a parsing error will ruin the entire group. Always verify that every input id appears in the output, and retry individually any missing items.

6.2. Asynchronous Batch API

Many providers offer a Batch API. You send a file with many requests, and the provider processes it over a longer time period in exchange for lower pricing than synchronous calls. The common format is JSONL, each line a request with a custom_id to correlate results. The exact structure depends on the provider:

import com.fasterxml.jackson.databind.ObjectMapper;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;

public static String toJsonl(List<Review> reviews, String model, ObjectMapper mapper) {
    return reviews.stream()
            .map(r -> Map.of(
                    "custom_id", "review-" + r.id(),
                    "body", Map.of(
                            "model", model,
                            "messages", List.of(Map.of("role", "user",
                                    "content", "Classify sentiment: " + r.text())))))
            .map(line -> {
                try {
                    return mapper.writeValueAsString(line);
                } catch (Exception e) {
                    throw new IllegalStateException(e);
                }
            })
            .collect(Collectors.joining("\n"));
}

Each review becomes an independent JSON line. The custom_id lets you know which result belongs to which review, since the order of results is not guaranteed. The output looks like this:

{"custom_id":"review-1","body":{"model":"small-model","messages":[{"role":"user","content":"Classify sentiment: Fast delivery"}]}}
{"custom_id":"review-2","body":{"model":"small-model","messages":[{"role":"user","content":"Classify sentiment: Shirt was torn"}]}}

The order of keys in JSON may differ because Map.of doesn’t maintain order. This doesn’t affect the Batch API. The typical workflow has four steps: a scheduled job creates the file at night, uploads it and creates a batch, stores the batch ID in the database, then another periodic job checks the status and downloads the results.

7. When NOT to Use These Techniques

• Prompt caching provides little benefit when prompts are short, when the prefix changes per user, or when traffic is sparse enough that the cache expires before the next request. With providers that charge for cache writes, marking content for cache that’s never read again increases costs.
• Response caching doesn’t work when answers depend on frequently changing data like inventory or prices, when personalization is needed, or when users expect varied responses.
• Small models aren’t suitable for tasks requiring multi-step reasoning, complex code writing or review, or high-stakes decisions (legal, medical, financial) when you don’t have a deterministic way to verify the output. Don’t switch models just because it “feels” right. Run it on a test dataset first.
• Batch processing can’t be used for every flow requiring immediate responses like chat or autocomplete. Combining items also doesn’t work when each item needs its own long context, or when one item’s failure can’t be allowed to affect others.

Note: Measure before optimizing. Log TokenUsage, which model was used, and which feature made the call for each request. Usually only a few endpoints consume most of your budget, and that’s where you should start.

8. Conclusion

LLM costs mainly come from repeated tokens and from using more powerful models than necessary. Each technique in this article targets a source of waste:

• Prompt caching: put static content first, keep the prefix stable, and measure cache hit ratio.
• Response caching: skip the LLM call for repeated questions.
• Cascade small → large model: verify output deterministically and escalate only when needed.
• Batch processing: combine items to share the instruction section, and use the Batch API for work that doesn’t need immediate results.

These techniques can be combined. For example, a nightly classification job could use a small model, send via Batch API, with a prompt that has a stable prefix. The key is measuring the cost per request. With data, you’ll know which changes actually work.