Kafka Basics: Topics, Partitions, and Offsets
1. Overview
Kafka is a distributed message broker system used to transmit data between services with high throughput. Before writing your first producer/consumer, you need to grasp three core concepts: topic, partition, offset. This article explains each concept with concrete Spring Boot examples. After reading, you’ll understand how data is organized and read in Kafka, and avoid common mistakes when starting out.
2. What is a Topic
A topic is like a “table” or “channel” that holds messages. Producers send messages to a topic, and consumers read messages from a topic. A single topic can have multiple producers and consumers at the same time.
For example, in an e-commerce system, you might have topics like:
orders.created
orders.cancelled
payments.processedEach topic holds messages related to one type of event. Kafka doesn’t enforce a schema on messages — you decide the format yourself (JSON, Avro, Protobuf…).
Create a topic via CLI for better visualization:
kafka-topics.sh --create \
--topic orders.created \
--bootstrap-server localhost:9092 \
--partitions 3 \
--replication-factor 1This command creates the topic orders.created with 3 partitions. This is the next concept we’ll explore.
3. Partitions — the Unit of Parallelization
3.1. Why We Need Partitions
A topic is divided into multiple partitions. Each partition is an independent log file where messages are appended to the end and never modified or deleted in place. Partitions allow Kafka to:
- Distribute data across multiple brokers.
- Allow multiple consumers to read in parallel (each consumer reads from one partition).
Topic: orders.created (3 partitions)
Partition 0: [msg0][msg1][msg4][msg7]...
Partition 1: [msg2][msg5][msg8]...
Partition 2: [msg3][msg6][msg9]...3.2. How Key Determines the Partition
When sending a message, Kafka uses the key to calculate which partition receives it (via hashing), ensuring that messages with the same key always go to the same partition — preserving order for related messages.
// Assuming we send an event with orderId as the key
ProducerRecord<String, String> record =
new ProducerRecord<>("orders.created", "order-123", payloadJson);
kafkaTemplate.send(record);This creates a ProducerRecord with key "order-123". All events for this order will always fall into the same partition, so consumers read them in the exact order they were sent.
Note: If you don’t provide a key (key =
null), Kafka distributes messages in round-robin fashion (or using the sticky partitioner from Kafka 2.4+), which doesn’t guarantee order among messages.
3.3. Order is Only Guaranteed Within a Partition
This is the most confusing point: Kafka guarantees order only within a partition, not across the entire topic. If you need strict ordering for a group of messages (for example, all events for an orderId), use the same key for that group.
4. Offset — the Read Position Within a Partition
Each message in a partition has a monotonically increasing number called an offset, starting from 0.
Partition 0:
offset: 0 1 2 3 4
message: m0 m1 m2 m3 m4Offset is a number unique within the scope of one partition, not the entire topic. Consumers use offset to track “how far they’ve read”.
4.1. Consumer Groups and Offset Commits
Kafka tracks read offsets through consumer groups. Each consumer group maintains its own offset for each partition, stored in an internal topic called __consumer_offsets.
# application.properties
spring.kafka.consumer.group-id=order-service
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=falsegroup-id: identifies the consumer group. Multiple instances with the same group-id will share the partitions of the topic.auto-offset-reset=earliest: if no offset has been saved yet (new consumer group), read from the beginning of the topic.enable-auto-commit=false: disables auto-commit so you control when to commit the offset (prevents message loss if the consumer crashes mid-processing).
4.2. Multiple Consumers in the Same Group
Kafka assigns partitions to consumers in the same group following this rule: a partition can only be read by one consumer in the group at any given time.
Topic orders.created has 3 partitions
Consumer group "order-service" has 2 instances:
Consumer A -> partition 0, partition 1
Consumer B -> partition 2If you run 4 consumers for a topic with 3 partitions, the 4th consumer will be idle and won’t get any partition. Therefore, the maximum useful number of consumers equals the number of partitions.
5. Setting Up Kafka in Spring Boot
5.1. Dependencies
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>5.2. Producer Sending Messages
@Service
public class OrderEventProducer {
private final KafkaTemplate<String, String> kafkaTemplate;
public OrderEventProducer(KafkaTemplate<String, String> kafkaTemplate) {
this.kafkaTemplate = kafkaTemplate;
}
public void publishOrderCreated(String orderId, String payloadJson) {
kafkaTemplate.send("orders.created", orderId, payloadJson);
}
}kafkaTemplate.send(topic, key, value) sends a message to the orders.created topic, using orderId as the key. Spring Boot automatically creates the KafkaTemplate bean based on the spring.kafka.bootstrap-servers configuration.
5.3. Consumer Reading Messages and Manual Offset Commits
@Component
public class OrderEventConsumer {
@KafkaListener(topics = "orders.created", groupId = "order-service")
public void onOrderCreated(
ConsumerRecord<String, String> record,
Acknowledgment ack) {
System.out.printf("partition=%d offset=%d key=%s value=%s%n",
record.partition(), record.offset(), record.key(), record.value());
// Process business logic here
ack.acknowledge(); // commit offset after successful processing
}
}record.partition()andrecord.offset()tell you which partition the message is in and at what position.ack.acknowledge()commits the offset only after processing completes — if the service crashes mid-processing, the message will be read again on the next restart (thanks toenable-auto-commit=falsecombined withAckMode.MANUAL).
You need to add this configuration for Acknowledgment to work:
spring.kafka.listener.ack-mode=manualOutput when testing (sending 3 messages with different keys):
partition=1 offset=0 key=order-123 value={"status":"CREATED"}
partition=0 offset=0 key=order-456 value={"status":"CREATED"}
partition=2 offset=0 key=order-789 value={"status":"CREATED"}Each key falls into a different partition due to different hash values — exactly as explained in section 3.2.
6. Testing with JUnit 5
Use spring-kafka-test to run an embedded Kafka in unit tests without needing a real broker.
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>@SpringBootTest
@EmbeddedKafka(partitions = 1, topics = "orders.created")
class OrderEventProducerTest {
@Autowired
private OrderEventProducer producer;
@Autowired
private EmbeddedKafkaBroker embeddedKafka;
@Test
void shouldPublishMessageWithGivenKey() throws Exception {
Consumer<String, String> consumer = createTestConsumer();
embeddedKafka.consumeFromAnEmbeddedTopic(consumer, "orders.created");
producer.publishOrderCreated("order-123", "{\"status\":\"CREATED\"}");
ConsumerRecord<String, String> record =
KafkaTestUtils.getSingleRecord(consumer, "orders.created");
assertThat(record.key()).isEqualTo("order-123");
assertThat(record.value()).contains("CREATED");
assertThat(record.offset()).isEqualTo(0L);
}
private Consumer<String, String> createTestConsumer() {
Map<String, Object> props = KafkaTestUtils.consumerProps(
"test-group", "true", embeddedKafka);
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
DefaultKafkaConsumerFactory<String, String> factory =
new DefaultKafkaConsumerFactory<>(props);
return factory.createConsumer();
}
}@EmbeddedKafka starts an in-memory Kafka broker just for testing. KafkaTestUtils.getSingleRecord retrieves the first message from the topic, and we assert that the key, value, and offset match our expectations — offset 0 because it’s the first message of the only partition.
7. Common Mistakes
- Confusing the number of partitions with the number of consumers: creating too many consumers compared to partitions means some consumers will sit idle. The number of partitions determines the maximum degree of parallelism.
- Not setting a key when order matters: if you need events from the same entity (order, user…) to be processed in the correct order, you must use a key — don’t leave it as
null. - Increasing partitions after going to production: increasing the number of partitions for an existing topic will change how keys are hashed → messages with the same key might fall into a different partition, breaking order guarantees for old data. Determine the right number of partitions from the start.
- Default auto-commit (
enable-auto-commit=true): offsets are committed on a fixed interval (default 5s), not tied to whether message processing is complete or not — prone to message loss if the consumer crashes mid-processing.
Note: If your application doesn’t need strict ordering and throughput isn’t an issue, you can skip optimizing key/partition — Kafka works fine with default settings. Don’t optimize prematurely without real requirements.
8. Conclusion
A topic holds messages by subject, a partition is the unit for parallelization and distribution, and an offset is the read position within each partition. Understanding these three concepts helps you design correctly: choose appropriate keys to maintain order, select a partition count that matches your consumer count, and control offset commits to prevent data loss on errors. This is the foundation you must master before diving into advanced topics like rebalancing, exactly-once semantics, or Kafka Streams.
Glowmi · Lantern NightA cozy Mid-Autumn match-3 puzzleiOS · Android
Consumer Groups and Rebalancing in Kafka with Spring Boot
Understand how Kafka divides partitions among consumer groups, when rebalancing occurs, and how to reduce rebalancing impact in Spring Boot applications.
Change Data Capture with Debezium and Kafka
Use Debezium to read PostgreSQL's WAL, push every data change to Kafka and consume them with Spring Boot, without modifying the data writing code.
Event Sourcing with Kafka: Save Events Instead of State
Use Kafka as an event log for Order: record immutable events, rebuild state through replay, build read models using projections. Plus the limitations to know before applying.
Kafka Monitoring: consumer lag and key metrics to track
What is consumer lag, how to measure it with CLI, AdminClient and Micrometer, which broker/producer/consumer metrics to track and how to set up reasonable alerts.
Kafka producer: acks, idempotence, and batching
Configure acks, enable.idempotence, batch.size and linger.ms for Kafka producer to prevent message loss, avoid duplicates while achieving high throughput.