Kafka Producer: Getting acks, retry, and idempotence Right

1. Overview

When a producer sends a message to Kafka, there are three questions you must answer: when is sending considered successful, how should you retry on error, and does retry cause message duplication. This article explains acks, retries, and enable.idempotence — three configurations that determine the reliability and correctness of a producer. After reading, you’ll know how to configure a “production-ready” producer and avoid common mistakes that cause data loss or duplication.

2. Required Dependencies

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka-clients</artifactId>
    <version>3.7.0</version>
</dependency>

The examples in this article use the pure Kafka client (KafkaProducer), not Spring Kafka, so you can clearly see how each configuration affects behavior.

3. acks: When a Producer Considers Sending Successful

acks determines how many broker acknowledgments the producer must wait for before considering a message sent.

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.ACKS_CONFIG, "all");

• acks=0: the producer doesn’t wait for any acknowledgments. Fastest, but messages are lost if a broker goes down immediately after receiving — you won’t even know they’re lost.
• acks=1: only waits for the leader partition to write to its local log and return an ACK. If the leader crashes before followers can sync, the message is lost.
• acks=all (or -1): waits for all replicas in the ISR (in-sync replica set) to acknowledge. Safest, but higher latency.

Note: acks=all is truly safe only when paired with min.insync.replicas >= 2 on the broker. If min.insync.replicas=1, acks=all only requires 1 replica to acknowledge — no different from acks=1 in terms of safety.

3.1. Accompanying Broker Configuration

# server.properties or topic config
min.insync.replicas=2

With replication factor = 3 and min.insync.replicas=2, the system can tolerate 1 broker going down while still writing, and ensures data is on at least 2 copies before acknowledging.

4. retries: Handling Temporary Errors

A producer may encounter temporary errors like NotLeaderForPartitionException or brief network disconnections. retries allows the producer to automatically resend without manual error handling code.

props.put(ProducerConfig.RETRIES_CONFIG, Integer.MAX_VALUE);
props.put(ProducerConfig.RETRY_BACKOFF_MS_CONFIG, 100);
props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);

• retries: maximum number of retry attempts. From Kafka 2.1 onward, set this very high (or Integer.MAX_VALUE) and use delivery.timeout.ms to control total wait time instead of counting attempts.
• retry.backoff.ms: wait time between retry attempts, preventing broker spam.
• delivery.timeout.ms: maximum total time from calling send() until getting a result (success or final error). Must be larger than request.timeout.ms + linger.ms.

Problem: without care, retries can cause message duplication or out-of-order delivery.

4.1. Why Retries Cause Message Duplication

1. Producer sends batch containing message A
2. Broker writes A successfully, sends ACK back
3. ACK is lost on the network (network timeout)
4. Producer doesn't receive ACK -> considers it a failure -> retries
5. Broker receives request again -> writes A a second time
=> Topic now has 2 copies of message A

This is a classic scenario: the broker successfully wrote the message but the producer doesn’t know it, so it resends. The idempotence section later addresses this problem correctly.

4.2. Why Retries Cause Out-of-Order Delivery

If max.in.flight.requests.per.connection > 1, the producer can send multiple batches in parallel. Batch 2 may be written by the broker before batch 1 if batch 1 errors and is being retried.

max.in.flight.requests.per.connection=5

With this configuration but without idempotence, message order in the partition is not guaranteed when retries occur.

5. Idempotence: Solving Duplication from Retries

enable.idempotence=true helps the producer automatically avoid duplicate writes when retrying, thanks to Kafka attaching a Producer ID (PID) and sequence number to each batch.

props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);

When idempotence is enabled, Kafka automatically sets the following values (you don’t need to configure them yourself, but should be aware):

acks = all
retries = Integer.MAX_VALUE (or very high)
max.in.flight.requests.per.connection <= 5

5.1. How It Works

Each producer session is assigned a unique PID by the broker. Each message sent within a partition has an incrementing sequence number. The broker stores the last sequence number written for each PID + partition combination.

Producer sends batch seq=5 for partition 0
Broker: "last sequence I stored is 4, 5 is next -> write it, store seq=5"
Producer resends (due to retry) batch seq=5
Broker: "seq=5 I already wrote -> skip it, return success ACK anyway"

As a result, retries don’t create duplicate records — the broker automatically detects and discards duplicates based on the sequence number, without requiring manual handling by the producer or consumer.

Note: idempotence only guarantees no duplicates within a single producer session and within a single partition. If the producer restarts (a new PID is assigned), deduplication by the old PID no longer applies. To achieve exactly-once across multiple partitions/topics, you need transactions (transactional.id) — beyond the scope of simple idempotence.

5.2. Example: Production-Ready Producer Configuration

public ProducerFactory() {
    Properties props = new Properties();
    props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
    props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

    // Enable idempotence -> automatically sets acks=all, retries high, in-flight <= 5
    props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, true);
    props.put(ProducerConfig.DELIVERY_TIMEOUT_MS_CONFIG, 120_000);

    this.producer = new KafkaProducer<>(props);
}

public void sendOrderEvent(String orderId, String payload) {
    ProducerRecord<String, String> record =
            new ProducerRecord<>("order-events", orderId, payload);

    producer.send(record, (metadata, exception) -> {
        if (exception != null) {
            log.error("Failed to send order {}", orderId, exception);
        } else {
            log.info("Order {} written to partition {} offset {}",
                    orderId, metadata.partition(), metadata.offset());
        }
    });
}

• ENABLE_IDEMPOTENCE_CONFIG = true is the most important line — it automatically pulls in acks=all.
• The callback in send() shows the actual result: if exception != null, it means retries exhausted delivery.timeout.ms and still failed — this is when you need to handle the business logic (log, alert, save to error table).

6. Verifying Behavior with Tests

To illustrate deduplication, let’s create a simple MessageDeduplicator that mimics the sequence number mechanism at the application level (useful when you can’t enable idempotence, such as in legacy systems):

public class MessageDeduplicator {

    private final Map<String, Long> lastSeenSequence = new ConcurrentHashMap<>();

    public boolean shouldWrite(String partitionKey, long sequence) {
        Long last = lastSeenSequence.get(partitionKey);
        if (last != null && sequence <= last) {
            return false; // duplicate, skip
        }
        lastSeenSequence.put(partitionKey, sequence);
        return true;
    }
}

class MessageDeduplicatorTest {

    @Test
    void shouldSkipDuplicateSequence() {
        MessageDeduplicator deduplicator = new MessageDeduplicator();

        assertThat(deduplicator.shouldWrite("order-1", 5)).isTrue();
        assertThat(deduplicator.shouldWrite("order-1", 5)).isFalse(); // retry same seq
        assertThat(deduplicator.shouldWrite("order-1", 6)).isTrue();
    }

    @Test
    void shouldAllowDifferentPartitionKeysIndependently() {
        MessageDeduplicator deduplicator = new MessageDeduplicator();

        assertThat(deduplicator.shouldWrite("order-1", 1)).isTrue();
        assertThat(deduplicator.shouldWrite("order-2", 1)).isTrue();
    }
}

The first test asserts that the same sequence sent again is skipped — true to the spirit of producer idempotence. The second test asserts that deduplication applies only within a key/partition scope and doesn’t affect other keys.

7. Configuration Summary by Use Case

Use Case	acks	enable.idempotence	Risk
Logs, non-critical metrics	0	false	Message loss on error, but highest throughput
Balance between speed and safety	1	false	Message loss if leader crashes before followers sync
Critical business data (orders, payments)	all	true	Higher latency, but no loss or duplication from retries

8. Common Mistakes

• Setting acks=all but forgetting min.insync.replicas: you think it’s safe but the broker only needs 1 ISR to ACK — no different from acks=1.
• Writing your own retry logic alongside Kafka client’s default retries: easily causes duplicate sends because both retry layers are active.
• Enabling idempotence but setting max.in.flight.requests.per.connection > 5: Kafka will reject this configuration immediately during producer initialization (ConfigException), as it exceeds the limit supported by idempotence.
• Confusing idempotence with system-wide exactly-once: idempotence only protects producer-to-broker. If a consumer processes a message then makes a non-idempotent side-effect call (sending email, deducting funds), you can still have duplicates on the consumer side.

Note: don’t enable enable.idempotence=true if your system is running Kafka versions older than 0.11 — this feature is not yet supported by the broker and the producer will fail on initialization.

9. Conclusion

acks, retries, and enable.idempotence are three inseparable pieces when you need a reliable producer. acks=all ensures data is replicated sufficiently before considering it successful; retries help overcome temporary errors; and idempotence ensures those retries don’t create duplicate records. For critical business data, use all three together by default — the latency cost is far less than the cost of dealing with incorrect or duplicate data later.