Schema Registry and Avro/Protobuf in Kafka with Spring Boot

1. Overview

When multiple services read from and write to the same Kafka topic, the biggest risk is message format. If just one producer changes a field name in the JSON, consumers downstream can break instantly. Confluent Schema Registry solves this problem. It stores schemas centrally, attaches each message with a schema ID, and blocks changes that break compatibility.

This article shows you how to set up Schema Registry, send and receive messages using Avro and Protobuf in Spring Boot 3.x, configure compatibility, and write tests without needing a real Kafka.

2. How Schema Registry Works

Schema Registry is a separate REST service that stands alongside the Kafka cluster. Inside, it stores schemas in the _schemas topic. Each schema belongs to a subject. By default (TopicNameStrategy), the subject for the value in topic orders is named orders-value.

When a producer sends a message, the serializer performs these steps:

1. Register or look up the schema on the Registry to get the schema ID. The result is cached.
2. Write the payload according to Confluent’s wire format:

| byte 0: magic byte (0) | byte 1-4: schema ID (int) | byte 5..: Avro/Protobuf binary data |

The consumer reads the first 5 bytes, extracts the schema ID, loads the schema once and caches it, then decodes the payload. Messages don’t need to carry the entire schema, so they’re much smaller than JSON.

3. Setting Up the Local Environment

Use Docker Compose with Kafka running in KRaft mode and Schema Registry:

services:
  kafka:
    image: confluentinc/cp-kafka:7.7.1
    ports: ["9092:9092"]
    environment:
      KAFKA_NODE_ID: 1
      KAFKA_PROCESS_ROLES: broker,controller
      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:29093
      KAFKA_LISTENERS: PLAINTEXT://kafka:29092,CONTROLLER://kafka:29093,EXTERNAL://0.0.0.0:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,EXTERNAL://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,CONTROLLER:PLAINTEXT,EXTERNAL:PLAINTEXT
      KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      CLUSTER_ID: MkU3OEVBNTcwNTJENDM2Qk

  schema-registry:
    image: confluentinc/cp-schema-registry:7.7.1
    depends_on: [kafka]
    ports: ["8081:8081"]
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKA_STORE_BOOTSTRAP_SERVERS: kafka:29092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

After running docker compose up -d, check the Registry:

curl -s http://localhost:8081/subjects

[]

The result is an empty list because no subjects have been registered yet.

4. Avro with Spring Boot

4.1. Dependencies and Plugins

Confluent’s serializers are not available on Maven Central. You need to add Confluent’s repository:

<repositories>
    <repository>
        <id>confluent</id>
        <url>https://packages.confluent.io/maven/</url>
    </repository>
</repositories>

<dependencies>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>io.confluent</groupId>
        <artifactId>kafka-avro-serializer</artifactId>
        <version>7.7.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.avro</groupId>
        <artifactId>avro</artifactId>
        <version>1.12.0</version>
    </dependency>
</dependencies>

The avro-maven-plugin generates Java classes from .avsc files:

<plugin>
    <groupId>org.apache.avro</groupId>
    <artifactId>avro-maven-plugin</artifactId>
    <version>1.12.0</version>
    <executions>
        <execution>
            <phase>generate-sources</phase>
            <goals><goal>schema</goal></goals>
            <configuration>
                <sourceDirectory>${project.basedir}/src/main/avro</sourceDirectory>
                <outputDirectory>${project.build.directory}/generated-sources/avro</outputDirectory>
                <stringType>String</stringType>
            </configuration>
        </execution>
    </executions>
</plugin>

The stringType=String option makes string fields generate as java.lang.String instead of CharSequence. This makes comparisons in code and tests simpler.

4.2. Defining the Schema

Create the file src/main/avro/Order.avsc:

{
  "type": "record",
  "name": "Order",
  "namespace": "dev.nonla.order.avro",
  "fields": [
    { "name": "id", "type": "string" },
    { "name": "customerId", "type": "string" },
    { "name": "amount", "type": "double" }
  ]
}

Run mvn generate-sources. The plugin generates the class dev.nonla.order.avro.Order with a builder and schema embedded in the class.

4.3. Configuring Producer and Consumer

spring:
  kafka:
    bootstrap-servers: localhost:9092
    properties:
      schema.registry.url: http://localhost:8081
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: io.confluent.kafka.serializers.KafkaAvroSerializer
    consumer:
      group-id: shipping-service
      auto-offset-reset: earliest
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: io.confluent.kafka.serializers.KafkaAvroDeserializer
      properties:
        specific.avro.reader: true

The schema.registry.url property placed under spring.kafka.properties applies to both producer and consumer. The specific.avro.reader: true property is critical. Without it, the consumer receives GenericRecord instead of the generated Order class.

4.4. Sending and Receiving Messages

@Service
public class OrderPublisher {

    private final KafkaTemplate<String, Order> kafkaTemplate;

    public OrderPublisher(KafkaTemplate<String, Order> kafkaTemplate) {
        this.kafkaTemplate = kafkaTemplate;
    }

    public void publish(String orderId, String customerId, double amount) {
        Order order = Order.newBuilder()
                .setId(orderId)
                .setCustomerId(customerId)
                .setAmount(amount)
                .build();
        kafkaTemplate.send("orders", order.getId(), order);
    }
}

@Component
public class ShippingListener {

    private static final Logger log = LoggerFactory.getLogger(ShippingListener.class);

    @KafkaListener(topics = "orders")
    public void onOrder(Order order) {
        log.info("Preparing to ship order {} to customer {}, amount {}",
                order.getId(), order.getCustomerId(), order.getAmount());
    }
}

On the first send, KafkaAvroSerializer automatically registers the schema to the orders-value subject. You can verify it:

curl -s http://localhost:8081/subjects/orders-value/versions/latest

{"subject":"orders-value","version":1,"id":1,"schema":"{\"type\":\"record\",\"name\":\"Order\",...}"}

At this point, messages on the topic only carry schema ID 1, not the entire schema.

5. Protobuf with Spring Boot

The steps with Protobuf are nearly identical to Avro, differing only in the serializer and code generation method. Add the following dependency:

<dependency>
    <groupId>io.confluent</groupId>
    <artifactId>kafka-protobuf-serializer</artifactId>
    <version>7.7.1</version>
</dependency>

Define src/main/proto/order_event.proto:

syntax = "proto3";

package dev.nonla.order;

option java_package = "dev.nonla.order.proto";
option java_multiple_files = true;

message OrderEvent {
  string id = 1;
  string customer_id = 2;
  double amount = 3;
}

You generate Java classes using a protoc plugin for Maven, such as io.github.ascopes:protobuf-maven-plugin. Then configure the serializer:

spring:
  kafka:
    producer:
      value-serializer: io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer
    consumer:
      value-deserializer: io.confluent.kafka.serializers.protobuf.KafkaProtobufDeserializer
      properties:
        specific.protobuf.value.type: dev.nonla.order.proto.OrderEvent

The specific.protobuf.value.type property tells the deserializer the target class. Without it, you receive DynamicMessage.

Which format should you choose?

• Avro suits the Kafka ecosystem and data pipelines like Kafka Connect or ksqlDB. Schemas are in JSON, and evolution rules are based on field names and default values.
• Protobuf is suitable if your organization already uses gRPC. You can reuse existing .proto files. Evolution rules are based on field numbers.

6. Schema Evolution and Compatibility

6.1. Compatibility Levels

Each subject has a compatibility level. The default level is BACKWARD.

Level	Meaning	Safe Deployment Order
BACKWARD	New schema can read old data	Consumer first, then producer
FORWARD	Old schema can read new data	Producer first, then consumer
FULL	Both directions work	Any order
*_TRANSITIVE	Checked against all old versions, not just the previous one	As above

You set the level for each subject via REST API:

curl -s -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "FULL_TRANSITIVE"}' \
  http://localhost:8081/config/orders-value

6.2. A Valid Change

Suppose you want to add a currency field to Order. With Avro, the new field must have a default value for the new schema to achieve FULL compatibility:

{ "name": "currency", "type": "string", "default": "VND" }

Before merging code, you should check the new schema with Registry:

curl -s -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"schema": "{\"type\":\"record\",\"name\":\"Order\",\"namespace\":\"dev.nonla.order.avro\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"customerId\",\"type\":\"string\"},{\"name\":\"amount\",\"type\":\"double\"},{\"name\":\"currency\",\"type\":\"string\",\"default\":\"VND\"}]}"}' \
  http://localhost:8081/compatibility/subjects/orders-value/versions/latest

{"is_compatible":true}

If you omit "default": "VND", the result returns false. Then the producer will receive a 409 Conflict error when trying to register the schema.

With Protobuf, the corresponding rule is: never change or reuse field numbers. When removing a field, mark it reserved to prevent reuse of that number:

message OrderEvent {
  reserved 2;
  reserved "customer_id";
  string id = 1;
  double amount = 3;
  string currency = 4;
}

7. Writing Tests with Mock Schema Registry

Confluent’s serializer supports URLs like mock://<scope>. With this URL, the serializer uses an in-memory mock Registry. You can test serialization without Docker:

import static org.assertj.core.api.Assertions.assertThat;

class OrderAvroSerdeTest {

    private static final Map<String, Object> CONFIG = Map.of(
            "schema.registry.url", "mock://order-serde-test",
            "specific.avro.reader", true);

    @Test
    void shouldSerializeWithWireFormatAndDeserializeBack() {
        try (var serializer = new KafkaAvroSerializer();
             var deserializer = new KafkaAvroDeserializer()) {
            serializer.configure(CONFIG, false);
            deserializer.configure(CONFIG, false);

            Order order = Order.newBuilder()
                    .setId("ORD-1001")
                    .setCustomerId("CUS-42")
                    .setAmount(250_000.0)
                    .build();

            byte[] bytes = serializer.serialize("orders", order);

            assertThat(bytes[0]).isZero();                       // magic byte
            assertThat(ByteBuffer.wrap(bytes, 1, 4).getInt()).isPositive(); // schema ID

            Object result = deserializer.deserialize("orders", bytes);
            assertThat(result).isInstanceOf(Order.class).isEqualTo(order);
        }
    }
}

This test verifies three things:

• The first byte is magic byte 0.
• The next four bytes form a valid schema ID.
• The deserializer returns the correct Order class, not GenericRecord. This confirms that specific.avro.reader is in effect.

For full integration tests, you can use Testcontainers to run Kafka with Schema Registry. However, tests with mock:// are fast and sufficient to catch most configuration errors.

Note: Serializers sharing the same mock:// scope will share a single mock Registry. Each test class should use a separate scope to avoid interference.

8. Common Errors and When Not to Use

Poison pill. A message with an incorrect wire format causes KafkaAvroDeserializer to throw SerializationException before reaching the listener. The consumer can be stuck on that offset forever. You should wrap the deserializer with ErrorHandlingDeserializer:

spring:
  kafka:
    consumer:
      value-deserializer: org.springframework.kafka.support.serializer.ErrorHandlingDeserializer
      properties:
        spring.deserializer.value.delegate.class: io.confluent.kafka.serializers.KafkaAvroDeserializer

Auto-registering schemas in production. By default, auto.register.schemas=true, meaning any producer can register new schemas. In production, set auto.register.schemas: false and use.latest.version: true. Schemas should only be registered via CI/CD pipeline after compatibility checks.

Receiving GenericRecord or DynamicMessage unintentionally. The usual cause is forgetting specific.avro.reader or specific.protobuf.value.type. The result is ClassCastException in the listener.

Changing namespace or record name in Avro. With Avro, this is a completely different schema. The specific reader won’t find the corresponding class. Treat namespace as an immutable part of the contract.

Setting compatibility to NONE for convenience. This defeats the entire purpose of Registry. Only use NONE for experimental topics.

Note: Schema Registry is a runtime dependency. Because the serializer caches schemas, a temporary Registry outage rarely affects already-known schemas. However, a new producer starting up or an uncached schema will fail immediately. Run Registry in multi-instance mode on production.

When not to use Schema Registry:

• Small systems where a single team owns both producer and consumer, with few changes. Here, JSON with Spring’s JsonSerializer is much simpler.
• Messages need to be read directly by clients that don’t understand Confluent’s wire format, like a third-party tool that only reads plain JSON.
• You can’t operate an additional infrastructure service and don’t have access to a managed Registry.

9. Conclusion

Schema Registry transforms Kafka message formats into a versioned contract with automatic validation. You’ve learned how to:

• Set up Registry with Docker.
• Send and receive messages using KafkaAvroSerializer and KafkaProtobufSerializer in Spring Boot 3.
• Understand the wire format with magic byte and schema ID.
• Set compatibility levels and evolve schemas safely: new fields with defaults in Avro, reserved in Protobuf.
• Write fast tests using mock:// Registry.

When applying to production, disable auto schema registration, add compatibility checks to your CI pipeline, and always wrap deserializers with ErrorHandlingDeserializer. This way, a schema change that breaks compatibility is caught at review time, not when it crashes consumers at midnight.