Designing a multi-channel notification system: push, email, SMS with retry and priority
1. Overview
Nearly every product needs to send notifications: OTP via SMS, order confirmation email, push for new messages. Sending one notification is easy. The hard part is sending millions of notifications without losing any, without duplicates, and making sure OTP doesn’t queue behind marketing emails.
In this article, we’ll design an end-to-end multi-channel notification system. The scope includes requirements, data model, API, processing flow, and solutions to key challenges: preventing duplicate sends, maintaining consistency between DB and Kafka, and handling traffic spikes. Code examples use Java 21, Spring Boot 3.x, PostgreSQL, Kafka, and Redis.
2. Problem Requirements
2.1. Functional Requirements
- Other services (Order, Auth, Marketing…) call an API to send notifications to a user via one or more channels: PUSH, EMAIL, SMS.
- Each notification has a priority level:
HIGH(OTP, security alerts),NORMAL(transactions),LOW(marketing). - The system automatically retries on temporary provider errors. After max retries, the notification is marked
FAILEDfor audit. - The system respects user preferences, e.g., if a user disabled marketing emails.
2.2. Scale and Constraints
- Around 50 million notifications per day. Average ~600/second, spikes during marketing campaigns can reach 20,000/second.
HIGHpriority notifications must be sent within seconds, even during peak load.- SMS/email providers rate-limit, e.g., 500 requests/second per API key.
- Infrastructure semantics is at-least-once, but we must prevent duplicates at a reasonable level. A user receiving two OTP codes is a bad experience. Receiving two SMS bills you twice.
3. High-Level Architecture
Client service ──POST──▶ Notification API ──(1 transaction)──▶ PostgreSQL
│ notification, delivery (PENDING)
▼
Outbox Relay (poll, SKIP LOCKED)
│ publish by priority
┌───────────────────────────────┼───────────────────────────┐
▼ ▼ ▼
notifications.high notifications.normal notifications.low
│ │ │
└────────────▶ Delivery Workers ◀───────────────────────────┘
│ dedup (Redis) + rate limit (Redis)
▼
FCM/APNs · Email provider · SMS provider
│
success → SENT │ transient error → PENDING + next_attempt_atKey idea: the delivery table is the source of truth. Kafka is just the transport layer. Retry doesn’t need a separate retry topic. Workers only need to set a record back to PENDING with a future next_attempt_at, and the Outbox Relay will automatically repush it when the time comes.
4. Data Model
CREATE TABLE notification (
id UUID PRIMARY KEY,
idempotency_key VARCHAR(100) NOT NULL UNIQUE,
user_id BIGINT NOT NULL,
template VARCHAR(50) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE delivery (
id UUID PRIMARY KEY,
notification_id UUID NOT NULL REFERENCES notification(id),
channel VARCHAR(10) NOT NULL, -- PUSH | EMAIL | SMS
priority SMALLINT NOT NULL, -- 0 HIGH, 1 NORMAL, 2 LOW
status VARCHAR(10) NOT NULL, -- PENDING | QUEUED | SENT | FAILED
attempts INT NOT NULL DEFAULT 0,
next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(),
last_error TEXT,
UNIQUE (notification_id, channel)
);
CREATE INDEX idx_delivery_due
ON delivery (priority, next_attempt_at)
WHERE status = 'PENDING';A few key points:
idempotency_key UNIQUEprevents the client from creating duplicate notifications if the API is called again.- Each channel is a separate
deliveryrow. An email failure doesn’t force push to retry. priorityis duplicated indeliveryso the partial indexidx_delivery_dueserves the exact query the relay needs. The index only containsPENDINGrows, so it stays small even with billions ofSENTrows.
5. Notification Intake API
First, declare the required dependencies:
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>Next, the API. Clients must send an Idempotency-Key header. Usually that’s a business ID, e.g., order-123-confirmed.
public enum Priority { HIGH, NORMAL, LOW }
public enum Channel { PUSH, EMAIL, SMS }
public record SendRequest(long userId, String template, Map<String, Object> payload,
Priority priority, Set<Channel> channels) {}
@RestController
@RequestMapping("/api/notifications")
class NotificationController {
private final NotificationService service;
NotificationController(NotificationService service) { this.service = service; }
@PostMapping
ResponseEntity<Map<String, UUID>> send(@RequestHeader("Idempotency-Key") String key,
@RequestBody SendRequest req) {
UUID id = service.accept(key, req);
return ResponseEntity.accepted().body(Map.of("id", id));
}
}The API returns 202 Accepted because sending happens asynchronously. The main processing is in the service:
@Service
class NotificationService {
private final JdbcTemplate jdbc;
private final ObjectMapper mapper;
NotificationService(JdbcTemplate jdbc, ObjectMapper mapper) {
this.jdbc = jdbc;
this.mapper = mapper;
}
@Transactional
public UUID accept(String key, SendRequest req) {
List<UUID> inserted = jdbc.queryForList("""
INSERT INTO notification (id, idempotency_key, user_id, template, payload)
VALUES (?, ?, ?, ?, ?::jsonb)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id
""", UUID.class, UUID.randomUUID(), key, req.userId(), req.template(), toJson(req.payload()));
if (inserted.isEmpty()) { // retried request → return old notification
return jdbc.queryForObject(
"SELECT id FROM notification WHERE idempotency_key = ?", UUID.class, key);
}
UUID id = inserted.getFirst();
for (Channel ch : req.channels()) {
jdbc.update("""
INSERT INTO delivery (id, notification_id, channel, priority, status)
VALUES (?, ?, ?, ?, 'PENDING')
""", UUID.randomUUID(), id, ch.name(), req.priority().ordinal());
}
return id;
}
private String toJson(Map<String, Object> payload) {
try {
return mapper.writeValueAsString(payload);
} catch (JsonProcessingException e) {
throw new IllegalArgumentException("invalid payload", e);
}
}
}ON CONFLICT ... DO NOTHING RETURNING id handles race conditions directly in the database. When two requests with the same key arrive concurrently, only one insert succeeds. The other request gets an empty list and reads back the old ID. You don’t need application-level locking.
Note that the service doesn’t send to Kafka within this transaction. That’s exactly the Transactional Outbox pattern, explained next.
6. Outbox Relay and Priority Queuing
Suppose you write to DB then send Kafka immediately after. If the app crashes between the two steps, the notification sits in DB and is never sent. Do it backwards—send Kafka first then write DB—and workers might process a record that doesn’t exist yet. Outbox solves this: DB is the only place written in the transaction, and a relay reads DB and publishes after.
@Component
class OutboxRelay {
private static final String[] TOPICS =
{"notifications.high", "notifications.normal", "notifications.low"};
private final JdbcTemplate jdbc;
private final KafkaTemplate<String, String> kafka;
OutboxRelay(JdbcTemplate jdbc, KafkaTemplate<String, String> kafka) {
this.jdbc = jdbc;
this.kafka = kafka;
}
@Scheduled(fixedDelay = 200)
@Transactional
public void relay() {
List<Map<String, Object>> due = jdbc.queryForList("""
SELECT id, priority FROM delivery
WHERE status = 'PENDING' AND next_attempt_at <= now()
ORDER BY priority, next_attempt_at
LIMIT 500
FOR UPDATE SKIP LOCKED
""");
for (Map<String, Object> row : due) {
UUID id = (UUID) row.get("id");
int priority = ((Number) row.get("priority")).intValue();
kafka.send(TOPICS[priority], id.toString(), id.toString()).join();
jdbc.update("UPDATE delivery SET status = 'QUEUED' WHERE id = ?", id);
}
}
}This code does three things:
ORDER BY priority, next_attempt_atfetchesHIGHrows first. Even with 5 million marketing emails queued, a new OTP is selected on the next poll cycle.FOR UPDATE SKIP LOCKEDallows multiple relay instances to run in parallel. Each instance locks a different batch and skips rows locked by others. No instance waits for another..join()waits for broker acknowledgment before changing status. If relay crashes after sending but before commit, the row staysPENDINGand is resent. This is at-least-once. Workers must deduplicate (see section 7.2).
Priority is enforced again on the consumer side via separate topics and separate concurrency:
@Component
class DeliveryListeners {
private final DeliveryWorker worker;
DeliveryListeners(DeliveryWorker worker) { this.worker = worker; }
@KafkaListener(topics = "notifications.high", groupId = "delivery", concurrency = "12")
void high(String deliveryId) { worker.process(UUID.fromString(deliveryId)); }
@KafkaListener(topics = "notifications.normal", groupId = "delivery", concurrency = "6")
void normal(String deliveryId) { worker.process(UUID.fromString(deliveryId)); }
@KafkaListener(topics = "notifications.low", groupId = "delivery", concurrency = "2")
void low(String deliveryId) { worker.process(UUID.fromString(deliveryId)); }
}Kafka has no built-in priority queue within a topic. Separate topics are the practical solution. When a marketing campaign backs up notifications.low with millions of messages, the consumer for notifications.high runs on separate threads and is unaffected. The concurrency number must be less than or equal to the number of partitions for that topic.
7. Worker: Send and Retry
7.1. Retry Policy
Not all errors should be retried. Timeout or HTTP 429/503 is transient. A wrong phone number or a revoked push token is permanent—retry only costs money.
public class RetryPolicy {
private final Duration base;
private final Duration max;
private final int maxAttempts;
public RetryPolicy(Duration base, Duration max, int maxAttempts) {
this.base = base;
this.max = max;
this.maxAttempts = maxAttempts;
}
/** attempt: number of attempts so far (starts at 1). Empty = stop retrying. */
public Optional<Duration> nextDelay(int attempt) {
if (attempt >= maxAttempts) return Optional.empty();
long millis = base.toMillis() * (1L << Math.min(attempt - 1, 20));
return Optional.of(Duration.ofMillis(Math.min(millis, max.toMillis())));
}
}With base = 2s, max = 5m, maxAttempts = 6, the retry delays are:
1st fail → wait 2s
2nd fail → wait 4s
3rd fail → wait 8s
4th fail → wait 16s
5th fail → wait 32s
6th fail → FAILEDIn production, add random jitter to the delay. Otherwise, thousands of notifications failing at once when a provider goes down will retry at exactly the same moment—a phenomenon called thundering herd.
You can test this policy with a simple unit test:
class RetryPolicyTest {
private final RetryPolicy policy =
new RetryPolicy(Duration.ofSeconds(2), Duration.ofSeconds(10), 6);
@Test
void backoffGrowsExponentiallyAndIsCappedAtMax() {
assertThat(policy.nextDelay(1)).contains(Duration.ofSeconds(2));
assertThat(policy.nextDelay(3)).contains(Duration.ofSeconds(8));
assertThat(policy.nextDelay(4)).contains(Duration.ofSeconds(10)); // capped by max
}
@Test
void stopRetryingAfterMaxAttempts() {
assertThat(policy.nextDelay(6)).isEmpty();
}
}7.2. Worker: Deduplication, Rate Limiting, and Status Update
public class TransientSendException extends RuntimeException {
public TransientSendException(String msg) { super(msg); }
}
public interface ChannelSender {
Channel channel();
void send(UUID deliveryId); // throws TransientSendException on transient errors
}
@Service
class DeliveryWorker {
private final JdbcTemplate jdbc;
private final StringRedisTemplate redis;
private final Map<Channel, ChannelSender> senders;
private final RetryPolicy retry = new RetryPolicy(Duration.ofSeconds(2), Duration.ofMinutes(5), 6);
DeliveryWorker(JdbcTemplate jdbc, StringRedisTemplate redis, List<ChannelSender> list) {
this.jdbc = jdbc;
this.redis = redis;
this.senders = list.stream().collect(Collectors.toMap(ChannelSender::channel, s -> s));
}
public void process(UUID id) {
Map<String, Object> d = jdbc.queryForMap(
"SELECT channel, attempts, status FROM delivery WHERE id = ?", id);
if ("SENT".equals(d.get("status"))) return; // duplicate from Kafka
Boolean first = redis.opsForValue()
.setIfAbsent("notif:sending:" + id, "1", Duration.ofMinutes(10));
if (!Boolean.TRUE.equals(first)) return; // another worker is sending
Channel channel = Channel.valueOf((String) d.get("channel"));
int attempt = ((Number) d.get("attempts")).intValue() + 1;
try {
if (!acquireRate(channel)) {
throw new TransientSendException("rate limited"); // back off, don't count as provider error
}
senders.get(channel).send(id);
jdbc.update("UPDATE delivery SET status='SENT', attempts=? WHERE id=?", attempt, id);
} catch (TransientSendException e) {
redis.delete("notif:sending:" + id);
retry.nextDelay(attempt).ifPresentOrElse(
delay -> jdbc.update("""
UPDATE delivery SET status='PENDING', attempts=?, last_error=?,
next_attempt_at = now() + make_interval(secs => ?)
WHERE id=?""", attempt, e.getMessage(), delay.toSeconds(), id),
() -> markFailed(id, attempt, e.getMessage()));
} catch (RuntimeException e) { // permanent error
markFailed(id, attempt, e.getMessage());
}
}
private void markFailed(UUID id, int attempt, String error) {
jdbc.update("UPDATE delivery SET status='FAILED', attempts=?, last_error=? WHERE id=?",
attempt, error, id);
}
private boolean acquireRate(Channel channel) {
String key = "rate:" + channel + ":" + Instant.now().getEpochSecond();
Long count = redis.opsForValue().increment(key);
if (count != null && count == 1) redis.expire(key, Duration.ofSeconds(2));
return count != null && count <= 500;
}
}The worker has two layers of deduplication:
- Check
status = 'SENT'to drop messages republished after successful send. - Redis
SET NXlock with TTL prevents two consumers from receiving the same message, e.g., during Kafka rebalance. The 10-minute TTL ensures the lock self-releases if a worker dies.
This is still not exactly-once. If a worker calls the provider successfully then crashes before UPDATE ... 'SENT', the notification can be resent after the lock expires. For tighter semantics, pass deliveryId as an idempotency key to the provider, if supported. Many email providers and FCM support this.
acquireRate is a fixed-window rate limiter on Redis. All worker instances share a single counter per second per channel, so total throughput never exceeds the provider’s quota of 500 requests/second. When exceeded, the notification is put back as PENDING instead of calling the provider and getting HTTP 429.
8. Key Challenges and Solutions
8.1. Traffic Spikes
When Marketing sends 10 million notifications at once, the system absorbs load at each tier:
- API only inserts to DB and returns, so it handles high load easily. For large campaigns, offer a separate batch API that takes a list of users or a segment. A background job will generate
deliveryrows in 1,000-row batches. - Kafka absorbs the overflow.
lowtopic backlog is normal, as long ashightopic flows. - Rate limiter protects the provider.
LOWnotifications sending slightly slower is no concern.
Add one more safeguard: in acquireRate, reserve quota for HIGH, e.g., LOW can only use 60% of the limit. This way, OTP is never rate-limited by marketing emails.
8.2. DB-Kafka Consistency
Outbox combined with at-least-once and idempotent consumers is the standard triple. You don’t need distributed transactions (XA). The cost is an additional poll cycle delay—here, 200ms. For lower latency, replace polling with CDC (reading PostgreSQL WAL), but your infrastructure gets more complex.
8.3. Data Explosion
The delivery table grows by 60–80 million rows per day. Partition by date (PARTITION BY RANGE (created_at)) and drop partitions older than 30–90 days. The partial index in section 4 keeps the relay query fast, but doesn’t replace cleanup.
8.4. User Preferences and Quiet Hours
Check preferences when creating delivery rows in NotificationService. Skip channels the user disabled. For quiet hours, e.g., no marketing push after 22:00, just set next_attempt_at to 08:00 next morning. The relay handles the rest—no extra scheduler needed.
Note: Don’t call a provider inside a long
@Transactionalblock. One slow 30-second HTTP call will hold a DB connection and row locks, draining the pool fast. InDeliveryWorker, eachUPDATEis a short, independent transaction.
Note: Don’t retry forever and don’t retry permanent errors. One wrong phone number, retry 6 times, times 1 million users = a huge SMS bill. Categorize provider error codes carefully.
8.5. When NOT to Use This Design
If you send only a few thousand emails per day, PostgreSQL plus SKIP LOCKED is enough. Workers poll the delivery table directly—drop Kafka entirely. Kafka only pays off when you need consumer separation by priority at large scale, or when multiple systems consume the same event stream. Conversely, if you need strict ordering of notifications for the same user, this design isn’t enough. Use the user ID as the Kafka key and skip async retry for those notifications.
9. Conclusion
We’ve designed an end-to-end multi-channel notification system with these key decisions:
- The
deliverytable is the source of truth. Retry is just settingPENDINGwith a futurenext_attempt_at. - Transactional Outbox with
FOR UPDATE SKIP LOCKEDkeeps DB and Kafka consistent without distributed transactions, and lets multiple relays run in parallel. - Priority is enforced at two levels: poll order in SQL and separate topics with separate concurrency in Kafka.
- Multi-layer deduplication:
Idempotency-Keyat API, status check and Redis lock at worker, idempotency key passed to provider. - Shared Redis rate limiter protects provider quota during traffic spikes.
From here, you can expand with multi-language templates, webhooks for delivery status, or dashboards tracking failure rates by channel.
Designing a Google Drive-like File Storage and Sharing System
Case study on file storage system design: requirements, data model, chunked upload API, sharing, sync, conflict handling, and burst traffic.
Designing a Real-Time Order Tracking and Shipper Location System
Case study on designing a real-time order tracking and shipper location system using Spring Boot, Kafka, Redis GEO, and SSE. Article analyzes concurrency, consistency, and load spikes.
Designing a Voucher System: Usage Limits Under High Load
Case study on discount code system design: preventing over-issuance during flash sales using atomic UPDATE, Redis Lua, Kafka, and time-limited reservations.
Designing an online auction system: concurrent bids and closure
Case study of auction system design: data model, API, concurrent bid handling with conditional updates, Redis Lua, anti-sniping, and safe session closure.
Designing 1-1 and Group Chat: Storing Messages and Read Status
Case study for 1-1 and group chat design. Using seq-based data model, idempotent send API, watermark-based read state, outbox + Kafka, and handling traffic spikes.