Consumer groups and rebalancing in Kafka: why consumers stop

1. Overview

You have a service reading messages from Kafka that ran smoothly for many days. One day consumer lag spikes. Logs are filled with Revoking previously assigned partitions and CommitFailedException. The culprit is usually rebalance of the consumer group.

This article explains how consumer groups distribute partitions, when rebalance occurs, and why it causes consumers to “freeze up”. After reading, you’ll know how to configure Spring Kafka to reduce rebalance frequency and minimize downtime with each occurrence.

2. How consumer groups work

A consumer group is a set of consumers sharing the same group.id. Kafka assigns topic partitions to members of the group. Each partition is assigned to only one consumer at any given time.

Suppose topic orders has 6 partitions. You run 3 instances of order-service:

order-service-1  -> orders-0, orders-1
order-service-2  -> orders-2, orders-3
order-service-3  -> orders-4, orders-5

If you run 7 instances, one instance will be idle since there are no more partitions to assign.

First, add the dependency:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

A simple listener in Spring Boot 3.x:

@Component
public class OrderListener {

    private static final Logger log = LoggerFactory.getLogger(OrderListener.class);

    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onOrder(ConsumerRecord<String, String> record) {
        log.info("Received order {} from partition {}", record.key(), record.partition());
    }
}

The @KafkaListener annotation creates a listener container. Inside the container is a thread running a poll() loop. Every instance using groupId = "order-service" joins the same group. One broker acts as group coordinator, managing the member list and partition assignment.

3. What is rebalance and when does it happen

Rebalance is the process where the group coordinator re-distributes partitions among members. Situations that trigger rebalance:

1. New consumer joins: scaling up, deploying new instances.
2. Consumer leaves group intentionally: shutdown, rolling restart.
3. Consumer considered dead for not sending heartbeat within session.timeout.ms. Default is 45 seconds.
4. Consumer not calling poll() within max.poll.interval.ms. Default is 5 minutes.
5. Topic partition count changes, or pattern-based subscription matches new topics.

The two timeout mechanisms in items 3 and 4 are different and often confused:

Config	Default	Checks
heartbeat.interval.ms	3000	Heartbeat send interval (from background thread)
session.timeout.ms	45000	Process still alive
max.poll.interval.ms	300000	Processing thread stuck

Heartbeat is sent from a separate background thread. Therefore the process still “lives” even when the processing thread is stuck. Conversely, max.poll.interval.ms correctly detects when your message processing code is too slow.

4. Why consumers stop

4.1. Eager rebalance: stop the entire group

With the traditional protocol and eager assignors (RangeAssignor, RoundRobinAssignor, StickyAssignor), each rebalance proceeds as follows:

1. All consumers revoke all partitions they hold.
2. All consumers re-submit JoinGroup requests and wait for each other.
3. The group leader calculates a new assignment plan, then the coordinator sends the result back.
4. Consumers receive new partitions and resume processing.

During steps 1–3, no one in the group processes messages. Starting just one new instance stops the entire group. This is the “stop-the-world” phenomenon.

4.2. Processing too slow: exceeding max.poll.interval.ms

The most common cause in practice is slow processing code. The following example calls an external service taking ~1 second per message:

@KafkaListener(topics = "orders", groupId = "order-service")
public void onOrder(ConsumerRecord<String, String> record) throws InterruptedException {
    paymentService.charge(record.value()); // assume ~1 second
}

By default max.poll.records = 500. One poll() call can return 500 records, meaning ~500 seconds of processing. This exceeds the 300 seconds of max.poll.interval.ms. When this happens, the consumer leaves the group, and you’ll see logs like this:

[Consumer clientId=consumer-order-service-1, groupId=order-service] consumer poll timeout has expired.
This means the time between subsequent calls to poll() was longer than the configured
max.poll.interval.ms, which typically implies that the poll loop is spending too much time
processing messages. You can address this either by increasing max.poll.interval.ms or by
reducing the maximum size of batches returned in poll() with max.poll.records.

Then offset commit fails:

org.apache.kafka.clients.consumer.CommitFailedException: Offset commit cannot be completed
since the consumer is not part of an active group for auto partition assignment; it is likely
that the consumer was kicked out of the group.

This triggers a chain of issues:

• This consumer’s partitions are assigned to others.
• Since offsets weren’t committed, the new partition holder reprocesses those messages.
• The old consumer rejoins, causing another rebalance.
• If the next batch is still slow, the cycle repeats endlessly. Lag grows indefinitely.

5. How to fix it

5.1. Adjust batch size and poll time

Key rule: max.poll.records × processing time per record must be well below max.poll.interval.ms.

spring:
  kafka:
    consumer:
      group-id: order-service
      max-poll-records: 50
      properties:
        max.poll.interval.ms: 120000

With 50 records and 1 second per record, a batch takes ~50 seconds. This figure is safely within the 120 second limit.

Note: Don’t increase max.poll.interval.ms to hours as a “quick fix”. This value is also the maximum time for Kafka to detect a truly stuck consumer. Also, a rebalance must wait for the slowest consumer to call poll() again, so a large value makes rebalance take longer.

If each message inherently takes long to process, such as calling slow APIs, set timeouts for external calls. Another approach is to push messages to an internal queue instead of holding the poll thread too long.

5.2. Cooperative rebalancing

CooperativeStickyAssignor uses incremental rebalance. Consumers only revoke partitions that actually need to be moved; other partitions keep processing normally.

spring:
  kafka:
    consumer:
      properties:
        partition.assignment.strategy: org.apache.kafka.clients.consumer.CooperativeStickyAssignor

When adding order-service-4 to a group of 3 instances as in section 2, only some partitions move to the new instance:

order-service-1  -> orders-0, orders-1        (unchanged, no pause)
order-service-2  -> orders-2                  (lost orders-3)
order-service-3  -> orders-4                  (lost orders-5)
order-service-4  -> orders-3, orders-5

Note: Don’t switch directly from eager to cooperative on a running group. You need to deploy twice, rolling. First time, configure both: CooperativeStickyAssignor,RangeAssignor. Next time, remove RangeAssignor. If members don’t have a common assignor, the group reports an error.

5.3. Static membership

Each rolling restart, one instance leaves and rejoins the group. This results in two rebalances per pod. Static membership solves this. You assign each instance a fixed identifier via group.instance.id:

spring:
  kafka:
    consumer:
      properties:
        group.instance.id: ${HOSTNAME}
        session.timeout.ms: 60000

Consumers with group.instance.id won’t send LeaveGroup when shutting down. If it returns within session.timeout.ms with the same id, the coordinator returns its old partitions without rebalancing.

Note: HOSTNAME must be stable across restarts, e.g., pod name from a Kubernetes StatefulSet. With a regular Deployment, pod names change each time, so static membership won’t work. Two instances running simultaneously with the same id will kick each other out (FencedInstanceIdException). When concurrency > 1, Spring Kafka automatically appends -0, -1… to the id for each sub-consumer.

The trade-off: when an instance truly dies, its partitions won’t be processed by anyone until session.timeout.ms expires.

5.4. New consumer group protocol

Since Kafka 4.0, the new rebalance protocol (KIP-848) is production-ready. You enable it with group.protocol=consumer. Partition assignment moves entirely to the broker and always happens incrementally. Consumers no longer wait for each other at a common JoinGroup barrier.

With this protocol, client-side partition.assignment.strategy is no longer used. Instead, the broker chooses the assignor (client-side config is group.remote.assignor). If your cluster and client are recent enough, this is worth considering. Check the kafka-clients version that Spring Boot pulls before enabling it.

6. Handling partition revocation

No matter how you optimize, rebalance will still happen. What matters is committing processed offsets before losing the partition, to reduce duplicate processing.

Spring Kafka automatically commits pending offsets before partitions are revoked. If you use KafkaConsumer directly, you must do this yourself with ConsumerRebalanceListener:

public class CommitOnRevokeListener implements ConsumerRebalanceListener {

    private final Consumer<String, String> consumer;
    private final Map<TopicPartition, OffsetAndMetadata> pending = new ConcurrentHashMap<>();

    public CommitOnRevokeListener(Consumer<String, String> consumer) {
        this.consumer = consumer;
    }

    public void markProcessed(ConsumerRecord<String, String> record) {
        pending.put(new TopicPartition(record.topic(), record.partition()),
                new OffsetAndMetadata(record.offset() + 1));
    }

    @Override
    public void onPartitionsRevoked(Collection<TopicPartition> partitions) {
        Map<TopicPartition, OffsetAndMetadata> toCommit = new HashMap<>();
        for (TopicPartition tp : partitions) {
            OffsetAndMetadata offset = pending.remove(tp);
            if (offset != null) {
                toCommit.put(tp, offset);
            }
        }
        if (!toCommit.isEmpty()) {
            consumer.commitSync(toCommit);
        }
    }

    @Override
    public void onPartitionsAssigned(Collection<TopicPartition> partitions) {
        // Can reload local state for new partitions here
    }
}

Explanation of each part:

• markProcessed() records the next offset to read, i.e. offset + 1, after a record is processed.
• onPartitionsRevoked() runs on the poll thread itself, right before partitions are taken away. It only commits offsets for revoked partitions. This is important with cooperative rebalancing, since only some partitions are revoked.
• commitSync() blocks until the broker confirms, ensuring offsets are saved before another consumer gets the partition.

You register the listener when subscribing: consumer.subscribe(List.of("orders"), listener).

Test this behavior with MockConsumer from kafka-clients:

class CommitOnRevokeListenerTest {

    @Test
    void shouldCommitOnlyRevokedPartitions() {
        MockConsumer<String, String> consumer = new MockConsumer<>(OffsetResetStrategy.EARLIEST);
        TopicPartition p0 = new TopicPartition("orders", 0);
        TopicPartition p1 = new TopicPartition("orders", 1);
        consumer.assign(List.of(p0, p1));

        CommitOnRevokeListener listener = new CommitOnRevokeListener(consumer);
        listener.markProcessed(new ConsumerRecord<>("orders", 0, 41L, "order-1", "{}"));
        listener.markProcessed(new ConsumerRecord<>("orders", 1, 7L, "order-2", "{}"));

        listener.onPartitionsRevoked(List.of(p0));

        Map<TopicPartition, OffsetAndMetadata> committed = consumer.committed(Set.of(p0, p1));
        assertThat(committed.get(p0).offset()).isEqualTo(42L);
        assertThat(committed.get(p1)).isNull();
    }
}

The test simulates only orders-0 being revoked. The expected outcome is offset 42 committed for orders-0. Meanwhile orders-1 is still held, so no offset is committed for it yet.

Note: Code in onPartitionsRevoked() must be fast. This callback runs during rebalance. If it’s slow, the entire group has to wait.

7. Common mistakes

• Asynchronous processing but still committing with poll. You push records to an ExecutorService, then the container commits offsets right after poll(). If rebalance happens, in-flight messages might be lost. Only commit offsets for fully processed records.
• Confusing session.timeout.ms with max.poll.interval.ms. Increasing session timeout doesn’t help if the cause is slow processing.
• Setting concurrency larger than partition count. Extra consumers are idle. They only make the group larger and rebalance slower.
• Long GC pauses or CPU throttling on container. Heartbeat thread can’t keep up, consumer is kicked out even though code isn’t slow. Check JVM metrics and CPU limits.
• Not designing for idempotent processing. Rebalance can always cause some messages to be reprocessed. Don’t assume “exactly once” if you’re not using transactions or haven’t implemented deduplication, such as checking if an orderId was already processed.

8. Conclusion

Rebalance is a necessary mechanism for consumer groups to auto-scale and tolerate failures. The price is that consumers must pause. The most common reason consumers keep “stopping” is the poll() loop exceeding max.poll.interval.ms. Then the consumer is kicked out, commit fails, and rebalance repeats endlessly.

To minimize, you can:

• Keep each batch small with max.poll.records.
• Switch to CooperativeStickyAssignor or the new consumer group protocol.
• Use static membership for deployments with stable identifiers.
• Commit offsets at the right time in onPartitionsRevoked().

Finally, always design message processing to be idempotent. Rebalance can’t be eliminated entirely; you can only make it happen less often and cause less harm.