EN
Tiếng ViệtEnglish한국어Deutsch日本語PortuguêsEspañolFrançais简体中文
Home / Architecture & Design / Kafka

Kafka Monitoring: consumer lag and key metrics to track

Cris The Dev · 10/12/2026 · 9 min read · 418

1. Overview

A Kafka system may appear “healthy”: brokers are running, producers are sending messages. But consumers might be falling behind millions of messages without anyone knowing. The metric that reflects this situation is consumer lag.

In this article, you’ll understand how consumer lag is calculated and how to measure it three ways: CLI, AdminClient in Java, and Micrometer in Spring Boot. Then we’ll go through the most important broker, producer, and consumer metrics and how to set up alerts for them.

2. What is consumer lag?

Each partition in Kafka is an append-only log. There are two offsets you need to care about:

  • Log-end offset (LEO): the offset of the next message to be written to the partition.
  • Committed offset: the offset that the consumer group has committed, i.e., the position the group will resume reading from if it restarts.

The lag for a partition is calculated as follows:

lag = log-end offset - committed offset

The lag of an entire consumer group is the sum of lag across all partitions the group reads.

Lag of zero means the consumer has caught up. Slight lag fluctuations are normal, since messages arrive continuously. Lag that grows steadily over time is the red flag. In that case, the consumer is processing slower than the producer’s write rate.

Note: With topics using transactions or compaction, offsets may have “gaps” (control records, deleted messages). Since lag is calculated from offsets, it can be somewhat larger than the actual number of messages still to be read. This is not an error.

3. Checking lag with CLI

The quickest way is to use the script that comes with Kafka:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
  --describe --group order-service

Output looks like:

GROUP          TOPIC   PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID       HOST
order-service  orders  0          15230           15232           2     consumer-1-a1b2   /10.0.0.5
order-service  orders  1          14980           18410           3430  consumer-1-c3d4   /10.0.0.6
order-service  orders  2          15102           15102           0     consumer-1-e5f6   /10.0.0.7

Partition 1 is lagging 3430 messages, while the other two are nearly caught up. Such uneven lag typically comes from two causes. One is a “hot” key piling into one partition. The other is that the consumer on host 10.0.0.6 has an issue.

The CLI is very useful when investigating issues. However, you can’t run this command every minute. The next section will automate lag measurement.

4. Computing lag with AdminClient

Add the kafka-clients dependency (Spring Boot manages the version):

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
</dependency>

First, extract the calculation into a pure function for easier testing:

public final class LagCalculator {

    private LagCalculator() {}

    public static Map<TopicPartition, Long> compute(Map<TopicPartition, Long> committed,
                                                    Map<TopicPartition, Long> endOffsets) {
        Map<TopicPartition, Long> lag = new HashMap<>();
        committed.forEach((tp, committedOffset) -> {
            Long end = endOffsets.get(tp);
            if (end != null) {
                lag.put(tp, Math.max(0L, end - committedOffset));
            }
        });
        return lag;
    }
}

This function only computes lag for partitions that have both committed offset and end offset. The Math.max(0L, …) has its reason. The committed offset and end offset are fetched with two separate requests, so sometimes the committed offset is “newer” than the just-fetched end offset. A negative result is meaningless in this case, so we clamp it to zero.

Next is the class that calls AdminClient:

public class ConsumerLagChecker {

    private final Admin admin;

    public ConsumerLagChecker(Admin admin) {
        this.admin = admin;
    }

    public Map<TopicPartition, Long> lagOf(String groupId) throws Exception {
        Map<TopicPartition, Long> committed = admin.listConsumerGroupOffsets(groupId)
                .partitionsToOffsetAndMetadata()
                .get(10, TimeUnit.SECONDS)
                .entrySet().stream()
                .filter(e -> e.getValue() != null)
                .collect(Collectors.toMap(Map.Entry::getKey, e -> e.getValue().offset()));

        Map<TopicPartition, OffsetSpec> request = committed.keySet().stream()
                .collect(Collectors.toMap(tp -> tp, tp -> OffsetSpec.latest()));

        Map<TopicPartition, Long> endOffsets = admin.listOffsets(request)
                .all()
                .get(10, TimeUnit.SECONDS)
                .entrySet().stream()
                .collect(Collectors.toMap(Map.Entry::getKey, e -> e.getValue().offset()));

        return LagCalculator.compute(committed, endOffsets);
    }
}

Here’s what happens step by step:

  • listConsumerGroupOffsets returns the committed offsets of the group. The value can be null if a partition has never been committed, so we filter those out.
  • listOffsets with OffsetSpec.latest() fetches the log-end offset of each partition.
  • Each call has a 10-second timeout. This prevents the thread from hanging indefinitely if the cluster runs into trouble.

Usage:

try (Admin admin = Admin.create(Map.of(
        AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092"))) {
    new ConsumerLagChecker(admin).lagOf("order-service")
            .forEach((tp, lag) -> System.out.println(tp + " -> " + lag));
}
orders-0 -> 2
orders-1 -> 3430
orders-2 -> 0

4.1. Testing LagCalculator

Use JUnit 5 and AssertJ (included in spring-boot-starter-test):

class LagCalculatorTest {

    private final TopicPartition p0 = new TopicPartition("orders", 0);
    private final TopicPartition p1 = new TopicPartition("orders", 1);

    @Test
    void shouldComputeLagPerPartition() {
        Map<TopicPartition, Long> lag = LagCalculator.compute(
                Map.of(p0, 100L, p1, 50L),
                Map.of(p0, 120L, p1, 50L));

        assertThat(lag).containsEntry(p0, 20L).containsEntry(p1, 0L);
    }

    @Test
    void shouldClampNegativeLagToZero() {
        Map<TopicPartition, Long> lag = LagCalculator.compute(
                Map.of(p0, 130L),
                Map.of(p0, 120L));

        assertThat(lag).containsEntry(p0, 0L);
    }

    @Test
    void shouldSkipPartitionWithoutEndOffset() {
        Map<TopicPartition, Long> lag = LagCalculator.compute(
                Map.of(p0, 10L), Map.of());

        assertThat(lag).isEmpty();
    }
}

5. Exporting lag to Micrometer in Spring Boot

Add Actuator and the Prometheus registry:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
management.endpoints.web.exposure.include=health,prometheus

5.1. Built-in client-side metrics

When you have MeterRegistry, Spring Boot automatically attaches MicrometerConsumerListener to the ConsumerFactory. All internal metrics from the Kafka consumer will appear at /actuator/prometheus, including:

kafka_consumer_fetch_manager_records_lag_max{client_id="consumer-order-service-1",...} 3430.0
kafka_consumer_fetch_manager_records_lag{topic="orders",partition="1",...} 3430.0

records-lag-max is the highest lag this particular consumer observes. The Kafka client computes it from the fetch position, not from committed offset.

5.2. External gauge to measure lag independently

Client-side metrics have one major weakness. When the consumer dies, the metric disappears. Right when lag is growing fastest, you lose your data. That’s why you should measure lag independently using AdminClient from a separate process:

@Component
public class ConsumerLagMetrics {

    private final ConsumerLagChecker checker;
    private final MeterRegistry registry;
    private final Map<TopicPartition, AtomicLong> gauges = new ConcurrentHashMap<>();

    public ConsumerLagMetrics(KafkaAdmin kafkaAdmin, MeterRegistry registry) {
        this.checker = new ConsumerLagChecker(Admin.create(kafkaAdmin.getConfigurationProperties()));
        this.registry = registry;
    }

    @Scheduled(fixedDelay = 30_000)
    public void refresh() throws Exception {
        checker.lagOf("order-service").forEach((tp, lag) ->
                gauges.computeIfAbsent(tp, this::register).set(lag));
    }

    private AtomicLong register(TopicPartition tp) {
        AtomicLong holder = new AtomicLong();
        Gauge.builder("kafka.consumer.group.lag", holder, AtomicLong::get)
                .tag("group", "order-service")
                .tag("topic", tp.topic())
                .tag("partition", String.valueOf(tp.partition()))
                .register(registry);
        return holder;
    }
}

Remember to enable scheduling with @EnableScheduling on a @Configuration class. Every 30 seconds, the kafka_consumer_group_lag gauge gets updated. This value is based on committed offset, so it remains accurate even if all consumers in the group have stopped.

In production, many teams use specialized tools like kafka-exporter or Burrow instead of the code above. Their principle remains the same: compare committed offset against log-end offset.

Note: Don’t put this gauge in the consumer service itself if the service has multiple instances. Each instance will export the same lag value, and your dashboard’s sum will be wrong. Run it in a separate monitoring service, or on just one instance.

6. Metrics to track

Lag tells you that there’s an issue. The metrics below help you find where the issue is.

6.1. Broker

Metric (JMX) Meaning Desired value
UnderReplicatedPartitions Partitions with out-of-sync replicas 0
OfflinePartitionsCount Partitions with no leader 0
ActiveControllerCount Number of active controllers (cluster-wide) exactly 1
IsrShrinksPerSec / IsrExpandsPerSec ISR shrinks and grows near 0, not fluctuating constantly
RequestHandlerAvgIdlePercent Request handler thread idle percentage > 0.3
NetworkProcessorAvgIdlePercent Network processor thread idle percentage > 0.3

OfflinePartitionsCount > 0 is a critical incident: those partitions cannot be read or written. Sustained UnderReplicatedPartitions usually signals that brokers are overloaded, disks are slow, or there’s a network issue.

6.2. Producer

  • record-error-rate: rate of send failures after exhausting retries. Any non-zero value deserves investigation.
  • request-latency-avg: request latency to brokers. If it climbs, you’ll see throughput drop.
  • record-retry-rate: high retry rate means brokers or the network are unstable.
  • buffer-available-bytes: low buffer space causes send() to block.

6.3. Consumer

  • records-lag-max: mentioned above.
  • records-consumed-rate: consumption rate. Compare it to the producer’s write rate for the topic.
  • rebalance-rate-per-hour and last-rebalance-seconds-ago: constant rebalancing freezes the consumer. This is a common culprit for lag growth.
  • commit-latency-avg: slow commits reduce throughput, especially with synchronous commits.
  • last-poll-seconds-ago: if it exceeds max.poll.interval.ms, the consumer gets kicked from the group.

7. Setting up reasonable alerts

The most common mistake is setting absolute thresholds like “alert if lag > 1000”. For a topic with 50,000 messages/second, 1000 is meaningless. For a topic with a few messages/minute, 1000 could already be hours behind.

There are two better approaches. The first is to alert when lag grows steadily:

groups:
  - name: kafka-consumer
    rules:
      - alert: KafkaConsumerLagGrowing
        expr: sum by (group, topic) (deriv(kafka_consumer_group_lag[10m])) > 0
          and sum by (group, topic) (kafka_consumer_group_lag) > 500
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Lag for {{ $labels.group }} on {{ $labels.topic }} is growing continuously for 15 minutes"

This rule only fires when both conditions are true: lag has an upward trend (deriv > 0) and exceeds a minimum threshold. The condition must persist for 15 minutes. This way, you avoid false alarms from brief spikes after deployment.

The second approach is to convert lag to time. Divide lag by consumption rate to estimate how many seconds behind the consumer is. This number ties directly to your SLA. For example, “orders must be processed within 2 minutes” is much clearer than “lag under 5000”.

Also, you should have a separate alert for when metrics go missing, for example using absent(kafka_consumer_group_lag). No data doesn’t mean everything is fine.

8. Common pitfalls

  • Relying only on records-lag-max. When the consumer dies, this metric vanishes. Always have an external source measuring lag.
  • Reading lag for groups with no active consumers. The CLI still shows committed offset, but the CONSUMER-ID column is empty. Lag growing here means no one is processing.
  • Adding more consumers than partitions. Excess consumers sit idle and lag doesn’t shrink. To scale up, you need to add partitions.
  • Ignoring rebalancing. Processing a batch too long causes exceeding max.poll.interval.ms. The consumer gets ejected, the group rebalances, then it repeats. Lag grows in a sawtooth pattern. Either lower max.poll.records or push heavy work to another thread.
  • Querying AdminClient too often. Fetching offsets every second for hundreds of groups creates unnecessary load on brokers. A cycle of 15–60 seconds is sufficient for most systems.

9. Conclusion

Consumer lag is the most direct metric showing whether your Kafka system is keeping up with data. Lag is calculated as log-end offset minus committed offset. You can view lag via kafka-consumer-groups.sh, compute it with AdminClient, or export it to Prometheus through Micrometer.

For effective monitoring, combine client-side metrics with an independent measurement source. Alongside lag, track broker metrics like UnderReplicatedPartitions and OfflinePartitionsCount, as well as rebalance frequency on the consumer side. Finally, alert based on trends and latency time rather than a fixed lag number. This way you’ll reduce false alarms without missing real incidents.

kafkamonitoringspring bootmicrometerprometheus
App by the authorGlowmi · Lantern NightA cozy Mid-Autumn match-3 puzzleiOS · Android

More on this topic