Observability for LLM Applications: Tracking Prompts, Costs, and Quality
1. Overview
When you deploy a feature using LLM (Large Language Model) to production, the first questions are usually: “How much did we spend this month?”, “Why was the answer wrong yesterday?”, “Is the new prompt better than the old one?”. Standard HTTP metrics don’t answer these questions.
This article explains the concept of observability for LLM applications and why this topic is gaining attention. Then, you’ll build a decorator layer around the LLM client in Spring Boot 3 to record traces, tokens, costs, and quality signals. The final section discusses common mistakes and when not to overdo it.
2. Why LLM needs separate observability
A normal REST API is deterministic. Same input gives same output. You just need to monitor latency, error rate, and throughput. LLM calls differ in several ways:
- Non-deterministic: same prompt can produce different results. The most common error is not HTTP 500 but HTTP 200 with a wrong answer.
- Costs calculated per token: the bill depends on prompt length and response length, not request count. A bloated prompt template can multiply costs several times without anyone noticing.
- Large latency swings: response time is proportional to the number of tokens generated. The average value is almost meaningless. You need to see the distribution (p95, p99).
- Prompt is code: changing a sentence in the system prompt is like a “deploy”. You need to know which request ran with which prompt version.
That’s why observability for LLM usually revolves around three pillars:
- Tracing prompt/response: which request, which model, which prompt version, how long it took, why it ended.
- Cost: input/output tokens, conversion to money, breakdown by feature.
- Quality: user feedback, automated checks, offline evaluation.
The OpenTelemetry community is also building a naming convention for GenAI attributes (prefix gen_ai.*). This convention is still evolving. In this article, you’ll borrow its naming for easier future integration.
3. Project setup
Add Actuator, Prometheus registry, and OpenTelemetry bridge for tracing. Versions are managed by Spring Boot BOM so you don’t need to declare them:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>Configure the endpoint and where to send traces:
management:
endpoints:
web:
exposure:
include: health,prometheus
tracing:
sampling:
probability: 0.1
otlp:
tracing:
endpoint: http://localhost:4318/v1/tracesHere probability: 0.1 means only 10% of requests get sent as traces. Metrics are still counted 100%. This distinction is important when calculating costs in section 5.
4. Wrap LLM client with Observation
You define a small abstraction independent of the vendor:
public record LlmRequest(String model, String promptVersion, String feature, String prompt) {}
public record LlmResponse(String content, int inputTokens, int outputTokens, String finishReason) {}
public interface LlmClient {
LlmResponse complete(LlmRequest request);
}LlmResponse carries token counts. Most vendor APIs return a usage field in the response, so you just need to map to this record. Use the vendor’s numbers instead of estimating tokens yourself.
Next is the decorator adding observability:
public class ObservedLlmClient implements LlmClient {
private final LlmClient delegate;
private final ObservationRegistry observations;
private final MeterRegistry meters;
private final CostCalculator costs;
public ObservedLlmClient(LlmClient delegate, ObservationRegistry observations,
MeterRegistry meters, CostCalculator costs) {
this.delegate = delegate;
this.observations = observations;
this.meters = meters;
this.costs = costs;
}
@Override
public LlmResponse complete(LlmRequest req) {
Observation obs = Observation.createNotStarted("llm.request", observations)
.lowCardinalityKeyValue("gen_ai.request.model", req.model())
.lowCardinalityKeyValue("prompt.version", req.promptVersion())
.lowCardinalityKeyValue("feature", req.feature());
return obs.observe(() -> {
LlmResponse res = delegate.complete(req);
obs.lowCardinalityKeyValue("gen_ai.response.finish_reason", res.finishReason());
obs.highCardinalityKeyValue("gen_ai.usage.input_tokens", String.valueOf(res.inputTokens()));
obs.highCardinalityKeyValue("gen_ai.usage.output_tokens", String.valueOf(res.outputTokens()));
Tags tags = Tags.of("model", req.model(), "feature", req.feature());
meters.counter("llm.tokens", tags.and("type", "input")).increment(res.inputTokens());
meters.counter("llm.tokens", tags.and("type", "output")).increment(res.outputTokens());
meters.counter("llm.cost.usd", tags).increment(costs.cost(req.model(), res).doubleValue());
return res;
});
}
}Key points:
Observation.createNotStarted("llm.request", ...)creates an observation. WhenDefaultMeterObservationHandleris present (Spring Boot configures it automatically), this observation produces a timer namedllm.request. With tracing bridge, it also produces a span.lowCardinalityKeyValueattaches tags to both metrics and spans. Use only for values with finite sets like model name, prompt version, feature name.highCardinalityKeyValueattaches only to spans. Token counts per request fit here. If you put them in metrics, each different value creates a new time series.observe(...)calls start/stop automatically. If there’s an exception, it records the error in the observation.- The two counters
llm.tokensandllm.cost.usdare counted for every request, regardless of tracing sampling.
Register the bean:
@Configuration
@EnableConfigurationProperties(LlmPricingProperties.class)
class LlmConfig {
@Bean
LlmClient llmClient(ObservationRegistry observations, MeterRegistry meters,
LlmPricingProperties pricing) {
LlmClient raw = new VendorLlmClient(); // your real API client
return new ObservedLlmClient(raw, observations, meters,
new CostCalculator(pricing.pricing()));
}
}On /actuator/prometheus you’ll see series like:
llm_request_seconds_count{feature="support-chat",gen_ai_request_model="model-a",prompt_version="v3",...} 42
llm_tokens_total{feature="support-chat",model="model-a",type="input"} 50400
llm_cost_usd_total{feature="support-chat",model="model-a"} 0.0441Note: If you use Spring AI, the framework already has observation for chat model. Understanding the mechanism above helps you read those metrics correctly and supplement what’s missing, like
featureorprompt.versiontags.
5. Calculate cost from tokens
Pricing changes by vendor and over time, so don’t hard-code it. Put it in configuration:
@ConfigurationProperties(prefix = "llm")
public record LlmPricingProperties(Map<String, ModelPrice> pricing) {
public record ModelPrice(BigDecimal inputPerMillion, BigDecimal outputPerMillion) {}
}llm:
pricing:
model-a:
input-per-million: 0.50 # example price, replace with your actual pricing
output-per-million: 1.50Calculation class:
public class CostCalculator {
private static final BigDecimal MILLION = BigDecimal.valueOf(1_000_000);
private final Map<String, LlmPricingProperties.ModelPrice> pricing;
public CostCalculator(Map<String, LlmPricingProperties.ModelPrice> pricing) {
this.pricing = pricing;
}
public BigDecimal cost(String model, LlmResponse res) {
var price = pricing.get(model);
if (price == null) {
return BigDecimal.ZERO; // should log warning: model doesn't have price
}
return price.inputPerMillion().multiply(BigDecimal.valueOf(res.inputTokens()))
.add(price.outputPerMillion().multiply(BigDecimal.valueOf(res.outputTokens())))
.divide(MILLION, 8, RoundingMode.HALF_UP);
}
}With 1,200 input tokens and 300 output tokens at the example price, the result is:
(1200 × 0.50 + 300 × 1.50) / 1_000_000 = 0.00105This counter gives you estimated cost near real-time, broken down by feature and model. From this you can set alerts, e.g., “cost for support-chat in 1 hour exceeded threshold X”. The official invoice still comes from the vendor. Reconcile regularly to find discrepancies, like when cached tokens or batch discounts apply that the simple formula doesn’t account for.
6. Log prompts safely
When debugging a wrong answer, you usually want to see the full prompt. But prompts often contain user data like emails, phone numbers, or order contents. So you need to do two things: redact sensitive information and sample.
public class PromptLogger {
private static final Logger log = LoggerFactory.getLogger(PromptLogger.class);
private static final Pattern EMAIL = Pattern.compile("[\\w.+-]+@[\\w-]+\\.[\\w.]+");
private static final Pattern PHONE = Pattern.compile("\\b\\d{10}\\b");
private final double sampleRate;
public PromptLogger(double sampleRate) {
this.sampleRate = sampleRate;
}
public void log(LlmRequest req, LlmResponse res) {
if (ThreadLocalRandom.current().nextDouble() >= sampleRate) {
return;
}
log.info("llm.exchange feature={} promptVersion={} prompt=\"{}\" response=\"{}\"",
req.feature(), req.promptVersion(), redact(req.prompt()), redact(res.content()));
}
static String redact(String text) {
String noEmail = EMAIL.matcher(text).replaceAll("[EMAIL]");
return PHONE.matcher(noEmail).replaceAll("[PHONE]");
}
}With input "Contact us at john@shop.com or 1234567890", the redact function returns:
Contact us at [EMAIL] or [PHONE]When using tracing, Micrometer automatically puts traceId in MDC. This way the log line links to the corresponding span. Regex is only minimal protection. For truly sensitive data, consider not logging prompts at all, or log to a separate store with access control and deletion expiry.
7. Measure quality
Quality is the hardest pillar because there’s no “status code” for a correct answer. In practice, people combine three signal sources.
7.1. Cheap automated checks
Some bad signs can be detected immediately in code: truncated response (finishReason is length), empty output, or output that won’t parse when you requested JSON.
public class QualityChecks {
private final MeterRegistry meters;
public QualityChecks(MeterRegistry meters) {
this.meters = meters;
}
public void inspect(LlmRequest req, LlmResponse res) {
if ("length".equals(res.finishReason())) {
count(req, "truncated");
}
if (res.content() == null || res.content().isBlank()) {
count(req, "empty");
}
}
private void count(LlmRequest req, String issue) {
meters.counter("llm.quality.issue",
"feature", req.feature(), "prompt.version", req.promptVersion(), "issue", issue)
.increment();
}
}The prompt.version tag lets you compare error rates between v2 and v3 right on the dashboard after switching prompts.
7.2. User feedback
A thumbs up/down button is a cheap and valuable signal. Send it with traceId to trace back to the exact request:
@RestController
@RequestMapping("/api/feedback")
class FeedbackController {
private final MeterRegistry meters;
FeedbackController(MeterRegistry meters) {
this.meters = meters;
}
record Feedback(String traceId, String feature, String promptVersion, boolean helpful) {}
@PostMapping
ResponseEntity<Void> submit(@RequestBody Feedback fb) {
meters.counter("llm.feedback", "feature", fb.feature(),
"prompt.version", fb.promptVersion(),
"helpful", String.valueOf(fb.helpful())).increment();
// save fb.traceId() to DB for later lookup
return ResponseEntity.accepted().build();
}
}traceId is not added to metric tags because its values are infinite. It’s only stored in the database.
7.3. Offline evaluation
For large changes like switching models or rewriting system prompt, you should have a set of sample questions with expected answers. Run this set in CI or on schedule. Scoring can use exact match, rules, or another LLM as judge. Results should be recorded by prompt.version for comparison. This is outside the scope of this article’s code, but it complements runtime metrics above.
8. Write tests for observability
You can test the decorator without calling the real API. Just use SimpleMeterRegistry and a fake client:
class ObservedLlmClientTest {
@Test
void recordsLatencyTokensAndCost() {
SimpleMeterRegistry meters = new SimpleMeterRegistry();
ObservationRegistry observations = ObservationRegistry.create();
observations.observationConfig()
.observationHandler(new DefaultMeterObservationHandler(meters));
var pricing = Map.of("model-a", new LlmPricingProperties.ModelPrice(
new BigDecimal("0.50"), new BigDecimal("1.50")));
LlmClient fake = req -> new LlmResponse("Hello", 1200, 300, "stop");
var client = new ObservedLlmClient(fake, observations, meters, new CostCalculator(pricing));
client.complete(new LlmRequest("model-a", "v3", "support-chat", "Hi there"));
assertThat(meters.get("llm.request").timer().count()).isEqualTo(1);
assertThat(meters.get("llm.tokens").tag("type", "input").counter().count())
.isEqualTo(1200);
assertThat(meters.get("llm.tokens").tag("type", "output").counter().count())
.isEqualTo(300);
assertThat(meters.get("llm.cost.usd").counter().count())
.isCloseTo(0.00105, within(1e-9));
}
}This test confirms three things: timer llm.request is recorded exactly once, tokens are split by type, and cost matches the calculation in section 5. If later someone renames a metric or breaks the pricing formula, the test fails and your dashboard won’t silently lose data.
9. Common mistakes and when not to overdo it
Some common mistakes:
- Infinite cardinality in metric tags:
userId,traceId, prompt content, or specific token count. They explode the number of time series and can crash the metrics system. These values belong in spans or logs. - Calculate cost from sampled traces: if only 10% of requests have traces, totaling costs from traces will be wrong. Use counters like in section 4.
- Store full prompts forever: privacy and storage cost risks. Redact sensitive data, sample, and set expiry.
- Don’t attach prompt version: when quality drops, you won’t know if it’s from changing prompts, changing models, or changing input data.
- Don’t just look at average latency: with LLM, look at p95/p99 and separately track time-to-first-token if you use streaming.
Note: Not every project needs all of the above. For an internal prototype, a few dozen requests per day, small budget, logging usage from the response and checking your vendor’s invoice is enough. Tracing, prompt storage, and evaluation pipelines add operational overhead. Start with token/cost counters and
prompt.versiontags. Expand the rest when traffic, costs, or quality requirements really demand it.
Also, if your organization already uses a dedicated LLM observability platform, don’t rewrite everything yourself. The decorator in this article still helps standardize tags, but storage and visualization should use the existing tool.
10. Conclusion
Observability for LLM applications extends the three familiar pillars (metrics, traces, logs) to answer three new questions: which prompt is running, how much did it cost, and is the answer good.
In this article, you wrapped the LLM client with Micrometer Observation to get timer and span with gen_ai.* tags. You counted tokens and converted to cost via pricing config, logged prompts with sensitive data redacted, and gathered quality signals from automated checks and user feedback. A sensible order of adoption is: costs and prompt version first, tracing and quality after, then expand as real needs require.
AI code review: how to use effectively and safely
Understand what AI code review is, integrate it into review processes and CI pipelines, hide secrets before sending diffs, prevent prompt injection, and know when not to use it.
Local-first software: offline apps and sync with CRDT
Learn about local-first software, why CRDTs let apps work offline and merge data automatically, with Java 21 and Spring Boot 3 examples.
Postgres Does Everything: Queue, Search, Vector in One Database
Use PostgreSQL as a job queue, full-text search, and vector search with Spring Boot. Includes code examples, tests, and guidance on when to move to specialized systems.
Are Vector Databases Still Necessary With Longer Context Windows?
Longer context windows let you fit more documents in prompts, but they don't fully replace retrieval. This analysis shows when you need a vector database and when you don't.
Documentation beats memory: building repos for humans and AI
Coding agents forget context between sessions. This guide shows how to embed conventions in your repo using AGENTS.md, verification scripts, ArchUnit, and tests that both humans and AI agents can read.