Kafka Retry and Dead Letter Topic with Spring Kafka

1. Overview

A Kafka consumer will inevitably encounter message processing errors. Some are temporary errors, for example a payment gateway timeout. Some are permanent errors, for example malformed data. Without proper handling, you’ll end up in one of two situations: losing messages, or the consumer getting stuck on a message that never succeeds.

This guide shows you how to configure retry in Spring Kafka in two ways: blocking with DefaultErrorHandler and non-blocking with @RetryableTopic. Messages that still fail after exhausting retries are pushed to a dead letter topic (DLT). After reading, you’ll know when to choose which approach and be able to write tests to verify the behavior.

2. Setup

Add the Spring Kafka dependency. Spring Boot 3.x manages the version automatically, so you don’t need to declare it:

<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka-test</artifactId>
    <scope>test</scope>
</dependency>

Minimal configuration in application.properties:

spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=order-service
spring.kafka.consumer.auto-offset-reset=earliest

Boot uses StringSerializer and StringDeserializer by default. To keep the example concise, the message value is an order ID as a String.

3. A Consumer That Can Fail

This is the payment processing service. A malformed order ID is a permanent error. An error calling the payment gateway is a temporary error:

public class PaymentGatewayException extends RuntimeException {
    public PaymentGatewayException(String message) {
        super(message);
    }
}

@Service
public class PaymentService {

    public void charge(String orderId) {
        if (orderId == null || !orderId.startsWith("ORD-")) {
            throw new IllegalArgumentException("Invalid order ID: " + orderId);
        }
        // Call payment gateway; may throw PaymentGatewayException on timeout
    }
}

The listener just calls the service:

@Component
public class OrderListener {

    private static final Logger log = LoggerFactory.getLogger(OrderListener.class);
    private final PaymentService paymentService;

    public OrderListener(PaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @KafkaListener(topics = "orders")
    public void onOrder(String orderId) {
        log.info("Processing order {}", orderId);
        paymentService.charge(orderId);
    }
}

If you don’t configure anything, Spring Kafka uses DefaultErrorHandler with FixedBackOff(0, 9). Error messages are retried 9 times in a row, for a total of 10 attempts. After that, the handler only logs and skips it. The message is considered lost. That’s why you need to reconfigure it.

4. Blocking Retry with DefaultErrorHandler

4.1. Configuring Backoff and DLT

When the listener throws an exception, DefaultErrorHandler seeks back to the offset of the failed message so the next poll will read it again. After exhausting retries, the handler calls a ConsumerRecordRecoverer. Here, the recoverer is DeadLetterPublishingRecoverer:

@Configuration
public class KafkaErrorConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
        var recoverer = new DeadLetterPublishingRecoverer(template);

        var backOff = new ExponentialBackOffWithMaxRetries(3);
        backOff.setInitialInterval(1_000L);
        backOff.setMultiplier(2.0);
        backOff.setMaxInterval(10_000L);

        var handler = new DefaultErrorHandler(recoverer, backOff);
        handler.addNotRetryableExceptions(IllegalArgumentException.class);
        return handler;
    }
}

Breaking it down:

• DeadLetterPublishingRecoverer(template) sends the failed message to topic <topic-name>.DLT, on the same partition as the original message. For topic orders, the DLT is orders.DLT.
• ExponentialBackOffWithMaxRetries(3) retries up to 3 times. The intervals are 1s, 2s, 4s respectively and don’t exceed 10s.
• addNotRetryableExceptions declares exceptions that don’t need retrying. IllegalArgumentException goes straight to DLT because retrying won’t fix malformed data.

Spring Boot automatically binds the CommonErrorHandler bean to the default container factory. You don’t need to create ConcurrentKafkaListenerContainerFactory yourself.

When sending ORD-1 and the payment gateway keeps timing out, the log will look like this:

10:00:00.010 INFO  Processing order ORD-1
10:00:01.020 INFO  Processing order ORD-1
10:00:03.030 INFO  Processing order ORD-1
10:00:07.040 INFO  Processing order ORD-1
10:00:07.060 ERROR Backoff ... exhausted for orders-0@42

There are 4 total attempts: 1 initial and 3 retries. After that, the message sits in orders.DLT.

4.2. Headers in DLT

Messages in the DLT retain their original key and value. The recoverer adds headers so you can investigate the cause:

kafka_dlt-original-topic      = orders
kafka_dlt-original-partition  = 0
kafka_dlt-original-offset     = 42
kafka_dlt-exception-fqcn      = org.springframework.kafka.listener.ListenerExecutionFailedException
kafka_dlt-exception-cause-fqcn = com.example.PaymentGatewayException
kafka_dlt-exception-message   = Listener method ... threw exception
kafka_dlt-exception-stacktrace = ...

Your original exception is at kafka_dlt-exception-cause-fqcn. To read headers in code, use constants in KafkaHeaders, for example KafkaHeaders.DLT_ORIGINAL_TOPIC.

Note: Because the recoverer sends to the same partition by default, orders.DLT must have at least as many partitions as orders. Otherwise, publishing will fail. You can create the DLT using a NewTopic bean with the same number of partitions. Another way is to pass a destination resolver that returns new TopicPartition(r.topic() + ".DLT", -1) to let Kafka choose the partition.

4.3. When Blocking Retry Makes Sense

Blocking retry preserves the order of messages in the partition. The next message is only processed once the previous one succeeds or goes to the DLT. This is the right choice when order matters, for example events OrderCreated → OrderPaid → OrderShipped for the same order.

The price you pay is that the entire partition is blocked while waiting. From Spring Kafka 2.9 onwards, the handler waits for the backoff without exceeding max.poll.interval.ms thanks to seek and pause mechanisms. Still, if you set the total backoff to several minutes, the lag will spike.

5. Non-Blocking Retry with @RetryableTopic

5.1. Configuring via Annotation

With non-blocking retry, failed messages are sent to a separate retry topic. This way, the consumer continues processing the next message immediately. Another consumer reads the retry topic after the specified backoff period:

@Component
public class OrderRetryListener {

    private static final Logger log = LoggerFactory.getLogger(OrderRetryListener.class);
    private final PaymentService paymentService;

    public OrderRetryListener(PaymentService paymentService) {
        this.paymentService = paymentService;
    }

    @RetryableTopic(
            attempts = "4",
            backoff = @Backoff(delay = 1_000, multiplier = 2.0, maxDelay = 10_000),
            exclude = IllegalArgumentException.class)
    @KafkaListener(topics = "payments")
    public void onPayment(String orderId,
                          @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
        log.info("Processing {} from {}", orderId, topic);
        paymentService.charge(orderId);
    }

    @DltHandler
    public void onDlt(String orderId,
                      @Header(KafkaHeaders.EXCEPTION_MESSAGE) String error) {
        log.error("Order {} went to DLT: {}", orderId, error);
    }
}

Key points:

• attempts = "4" means 4 total attempts: 1 on the original topic and 3 on retry topics. The value is a String to support placeholders like "${app.retry.attempts}".
• @Backoff comes from package org.springframework.retry.annotation. Spring Retry comes bundled with spring-kafka.
• exclude lists exceptions that go straight to the DLT, equivalent to addNotRetryableExceptions.
• @DltHandler is placed in the same class. This is where you log, send alerts, or save to a table for manual processing.

5.2. Topics Created

With incremental backoff, by default each retry gets its own topic, suffixed by the delay interval:

payments
payments-retry-1000
payments-retry-2000
payments-retry-4000
payments-dlt

Spring Kafka automatically creates these topics on startup (autoCreateTopics = "true" is the default). Production environments usually disable auto-create on the broker. In that case, you need to create the topics in advance with the correct names and partition count. If you want to use an index instead of the delay, set topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE.

Note the suffix: @RetryableTopic uses -dlt, while DeadLetterPublishingRecoverer uses .DLT. Remember this when setting up monitoring.

5.3. Trade-Offs of Non-Blocking

Non-blocking retry breaks the order. Suppose ORD-1 fails and goes to the retry topic, while ORD-2 succeeds immediately. Then ORD-2 is processed before ORD-1. If your business logic depends on event order, don’t use this approach.

Additionally, the number of topics increases and so does the number of consumers. In return, a single problematic message won’t block the entire partition.

6. Testing the DLT Flow

The test below uses @EmbeddedKafka. The test sends a malformed order ID and verifies that the message appears in orders.DLT without being retried:

@SpringBootTest(properties = "spring.kafka.bootstrap-servers=${spring.embedded.kafka.brokers}")
@EmbeddedKafka(partitions = 1, topics = {"orders", "orders.DLT"})
class OrderDltTest {

    @Autowired
    private KafkaTemplate<String, String> kafkaTemplate;

    @Autowired
    private EmbeddedKafkaBroker broker;

    @Test
    void givenInvalidOrderId_whenConsumed_thenSentToDltWithoutRetry() {
        kafkaTemplate.send("orders", "order-key", "INVALID-99");

        Map<String, Object> props = KafkaTestUtils.consumerProps("dlt-test", "false", broker);
        try (Consumer<String, String> consumer = new DefaultKafkaConsumerFactory<>(
                props, new StringDeserializer(), new StringDeserializer()).createConsumer()) {

            broker.consumeFromAnEmbeddedTopic(consumer, "orders.DLT");
            ConsumerRecord<String, String> record =
                    KafkaTestUtils.getSingleRecord(consumer, "orders.DLT", Duration.ofSeconds(10));

            assertThat(record.value()).isEqualTo("INVALID-99");
            assertThat(record.key()).isEqualTo("order-key");

            String cause = new String(record.headers()
                    .lastHeader(KafkaHeaders.DLT_EXCEPTION_CAUSE_FQCN).value());
            assertThat(cause).isEqualTo(IllegalArgumentException.class.getName());

            String originalTopic = new String(record.headers()
                    .lastHeader(KafkaHeaders.DLT_ORIGINAL_TOPIC).value());
            assertThat(originalTopic).isEqualTo("orders");
        }
    }
}

The test works as follows:

• ${spring.embedded.kafka.brokers} points the application to the embedded broker.
• KafkaTestUtils.consumerProps creates consumer configuration with auto.offset.reset=earliest, so the test won’t miss messages published before the consumer subscribes.
• Because IllegalArgumentException is non-retryable, the message reaches the DLT almost immediately. The test doesn’t need to wait for the entire backoff sequence.
• The assertions check three things: key and value are preserved, headers correctly record the original exception, and headers correctly record the source topic.

If you want to test the retry flow, expose the backoff value in properties. In the test, set the delay to a few dozen milliseconds to make the test run quickly.

7. Common Mistakes

• Not classifying exceptions. Retrying NullPointerException or validation errors just wastes time. Be clear about which exceptions are retried and which go straight to the DLT. Deserialization errors like DeserializationException are already non-retryable by default. But for the handler to receive this error, you need to wrap the deserializer with ErrorHandlingDeserializer.
• KafkaTemplate serializer mismatch. The recoverer republishes the deserialized value. If the consumer reads JSON into an object but the template uses StringSerializer, sending to the DLT will fail. Use a template with an appropriate serializer. For deserialization errors, the value is byte[], so you need ByteArraySerializer.
• Listener is not idempotent. Retry means the same message can be processed multiple times. If charge already deducted money and then times out, retry will deduct again. Prevent duplicates using an idempotency key, for example saving the processed orderId.
• Nobody is reading the DLT. The DLT is not a trash bin. You need to alert when new messages arrive in the DLT and have a replay process after fixing the issue.
• Total backoff too long with blocking retry. The partition is blocked the entire time. If the total wait time exceeds tens of seconds, consider switching to non-blocking.

Note: Don’t use @RetryableTopic for flows that require strict ordering by key. Conversely, don’t use blocking retry with long backoff for high-throughput topics. A single failed message can cause lag to spike tens of thousands.

8. Conclusion

This guide has presented two ways to handle failed messages in Spring Kafka. DefaultErrorHandler combined with DeadLetterPublishingRecoverer for blocking retry, preserves order but blocks the partition. @RetryableTopic with @DltHandler for non-blocking retry, doesn’t block the consumer but sacrifices order.

Regardless of which approach you choose, you should do four things: classify retryable and non-retryable exceptions, ensure the listener is idempotent, create the DLT with appropriate partitions, and monitor the DLT regularly. A test with @EmbeddedKafka like in section 6 helps ensure your configuration works correctly before going to production.