Designing a Facebook-like News Feed: Fan-out on Write or Read

1. Overview

A news feed is the homepage where you see the latest posts from people you follow. The core question when designing it is: should you assemble the feed at write time (fan-out on write) or at read time (fan-out on read)?

This article walks through a complete case study. We’ll cover requirements and scale estimation, then build a data model and API. Next, we’ll compare the two fan-out strategies and choose a hybrid model. Finally, we’ll tackle hard problems like celebrities, consistency, and traffic spikes. By the end, you’ll be able to build a working feed using Spring Boot, PostgreSQL, Redis, and Kafka.

2. Problem Requirements

2.1. Functional Requirements

• Users post content (text, optionally with images).
• Users follow/unfollow others.
• Users view their feed: posts from people they follow, newest first, with pagination (infinite scroll).

2.2. Scale and Constraints

Let’s assume these numbers:

DAU (daily active users)        : 300 million
Feed opens per person per day   : 10 times     -> 3 billion reads/day ≈ 35K QPS, peak ~100K QPS
New posts per person per day    : 0.5          -> 150 million posts/day ≈ 1.7K QPS, peak ~5K QPS
Average followers per user      : 300
Largest account                 : > 50 million followers

From this, we derive several constraints:

• Reads far exceed writes (about 20x). Reading the feed must be fast, p99 under 200 ms.
• Accept eventual consistency: new posts appearing in a follower’s feed after a few seconds is acceptable.
• Follower distribution is highly skewed. Most users have a few hundred followers; a tiny fraction have tens of millions.

3. Data Model

The source of truth lives in PostgreSQL:

CREATE TABLE users (
    id              BIGINT PRIMARY KEY,
    name            TEXT   NOT NULL,
    follower_count  BIGINT NOT NULL DEFAULT 0
);

CREATE TABLE posts (
    id          BIGINT PRIMARY KEY,           -- Snowflake ID
    author_id   BIGINT      NOT NULL,
    content     TEXT        NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    deleted     BOOLEAN     NOT NULL DEFAULT false
);
CREATE INDEX idx_posts_author_time ON posts (author_id, created_at DESC);

CREATE TABLE follows (
    follower_id BIGINT NOT NULL,
    followee_id BIGINT NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (follower_id, followee_id)
);
CREATE INDEX idx_follows_followee ON follows (followee_id, follower_id);

• Index idx_posts_author_time serves queries like “get the N most recent posts from author X”.
• The primary key of follows answers “who do I follow”. Index idx_follows_followee answers “who follows me”—this is exactly what fan-out needs.

Pre-assembled feeds live in Redis. Each user has a sorted set:

Key    : feed:{userId}
Member : postId (as a string)
Score  : createdAt in epoch milliseconds

Important: Don’t use Snowflake IDs as the score. Redis scores are double and can only precisely represent integers up to 2^53. A 64-bit Snowflake ID will be rounded, and two different posts might end up with the same score. Use epoch milliseconds as the score and keep the ID in the member field.

Feed stores only IDs, not post content. Content is “hydrated” from the post cache when reading. This way, if you edit or delete a post, you only update it in one place.

4. API

@RestController
@RequestMapping("/api")
public class FeedController {

    private final PostService postService;
    private final FeedService feedService;

    public FeedController(PostService postService, FeedService feedService) {
        this.postService = postService;
        this.feedService = feedService;
    }

    @PostMapping("/posts")
    public ResponseEntity<PostDto> create(@RequestHeader("X-User-Id") long userId,
                                          @RequestBody CreatePostRequest req) {
        return ResponseEntity.status(HttpStatus.CREATED).body(postService.create(userId, req.content()));
    }

    @GetMapping("/feed")
    public FeedPage feed(@RequestHeader("X-User-Id") long userId,
                         @RequestParam(required = false) String cursor,
                         @RequestParam(defaultValue = "20") int limit) {
        return feedService.getFeed(userId, Cursor.parse(cursor), Math.min(limit, 50));
    }
}

public record CreatePostRequest(String content) {}
public record FeedPage(List<PostDto> items, String nextCursor) {}

Example response:

GET /api/feed?limit=2
{
  "items": [ { "id": 7253..., "authorId": 42, "content": "Hello" }, { ... } ],
  "nextCursor": "1760245800123:7253..."
}

We use cursor-based pagination (createdAtMillis:postId), not offset-based. The feed changes constantly. With offset, when new posts appear at the top, subsequent pages may repeat items you’ve already seen.

5. Fan-out on Write (Push)

Idea: when an author posts, the system immediately writes the post ID into each follower’s feed. When reading, just one Redis command (ZREVRANGEBYSCORE) is needed.

Add dependencies:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

PostService saves the post to the database, then emits a PostCreated event to Kafka. A separate consumer does the fan-out:

public record PostCreatedEvent(long postId, long authorId, long createdAtMillis, long followerCount) {}

@Component
public class FanoutConsumer {

    static final int FEED_MAX = 500;
    static final long CELEBRITY_THRESHOLD = 10_000;
    private static final int BATCH = 1_000;

    private final FollowRepository follows;
    private final StringRedisTemplate redis;

    public FanoutConsumer(FollowRepository follows, StringRedisTemplate redis) {
        this.follows = follows;
        this.redis = redis;
    }

    @KafkaListener(topics = "post-created", groupId = "feed-fanout")
    public void onPostCreated(PostCreatedEvent e) {
        if (e.followerCount() >= CELEBRITY_THRESHOLD) {
            return; // celebrity: let the read path pull it
        }
        long lastId = 0;
        List<Long> batch;
        do {
            batch = follows.findActiveFollowerIds(e.authorId(), lastId, BATCH);
            pushToFeeds(batch, e);
            if (!batch.isEmpty()) lastId = batch.getLast();
        } while (batch.size() == BATCH);
    }

    private void pushToFeeds(List<Long> followerIds, PostCreatedEvent e) {
        redis.executePipelined((RedisCallback<Object>) conn -> {
            StringRedisConnection c = (StringRedisConnection) conn;
            for (Long followerId : followerIds) {
                String key = "feed:" + followerId;
                c.zAdd(key, e.createdAtMillis(), String.valueOf(e.postId()));
                c.zRemRange(key, 0, -(FEED_MAX + 1)); // keep only the 500 newest posts
            }
            return null;
        });
    }
}

Explanation:

• findActiveFollowerIds iterates followers using keyset pagination (follower_id > lastId), fetching 1,000 at a time. It only fetches users active in the past 30 days.
• executePipelined batches 2,000 commands into a single round-trip instead of 2,000 separate round-trips.
• zRemRange(key, 0, -501) trims the tail, keeping at most 500 items in the feed. This prevents memory from growing indefinitely.
• ZADD with the same member and score is idempotent. If Kafka redelivers the event (at-least-once guarantee), the feed won’t have duplicates.

The corresponding SQL:

SELECT f.follower_id
FROM follows f
JOIN user_activity a ON a.user_id = f.follower_id
WHERE f.followee_id = :authorId
  AND f.follower_id > :lastId
  AND a.last_seen_at > now() - interval '30 days'
ORDER BY f.follower_id
LIMIT :batch;

Advantages: Reads are extremely fast, O(log N) on a single Redis key. Disadvantages: Writes are amplified. 150 million posts × 300 followers ≈ 45 billion ZADD commands per day. A single celebrity with 50 million followers posting one post generates 50 million write commands.

6. Fan-out on Read (Pull)

Opposite idea: no pre-assembly at all. When reading, the system fetches the list of people you follow, then queries their recent posts:

SELECT p.id, p.created_at
FROM posts p
WHERE p.author_id IN (SELECT followee_id FROM follows WHERE follower_id = :userId)
  AND p.created_at < :before
  AND p.deleted = false
ORDER BY p.created_at DESC
LIMIT 20;

Advantages: Cheap to write (just one INSERT), new posts appear immediately, no memory overhead for feeds. Disadvantages: Each read must touch hundreds of authors. At 100K QPS, this is hundreds of thousands of queries. When posts are sharded by author_id, one read must scatter-gather across multiple shards. Keeping p99 under 200 ms becomes very difficult.

Summary:

                      Fan-out on Write        Fan-out on Read
Write Cost            High (× followers)      Low
Read Cost             Low (1 Redis key)       High (× followees)
Display Latency       A few seconds           Immediate
Celebrities           Terrible                OK
Heavy Followers       OK                      Slow

7. Hybrid Model

Facebook, Twitter, and most large systems use a hybrid approach:

• Typical authors (< 10K followers): push into each follower’s feed.
• Celebrities (≥ 10K followers): don’t push. When reading, the system pulls posts from celebrities the user follows, then merges with the pre-pushed feed.

Each user only follows dozens of celebrities, so the pull portion is small. Posts from celebrities are read by millions, so caching is highly effective.

The core part is the merge logic. Extract it into pure Java for easy testing:

public record FeedEntry(long postId, long createdAtMillis) {}

public record Cursor(long createdAtMillis, long postId) {
    static final Cursor START = new Cursor(Long.MAX_VALUE, Long.MAX_VALUE);

    public static Cursor parse(String raw) {
        if (raw == null || raw.isBlank()) return START;
        String[] p = raw.split(":");
        return new Cursor(Long.parseLong(p[0]), Long.parseLong(p[1]));
    }

    boolean isAfter(FeedEntry e) { // is e older than this cursor?
        return e.createdAtMillis() < createdAtMillis
            || (e.createdAtMillis() == createdAtMillis && e.postId() < postId);
    }
}

public final class FeedMerger {

    private static final Comparator<FeedEntry> NEWEST_FIRST =
        Comparator.comparingLong(FeedEntry::createdAtMillis)
                  .thenComparingLong(FeedEntry::postId)
                  .reversed();

    public static List<FeedEntry> merge(List<FeedEntry> pushed, List<FeedEntry> pulled,
                                        Cursor cursor, int limit) {
        return Stream.concat(pushed.stream(), pulled.stream())
            .filter(cursor::isAfter)
            .collect(Collectors.toMap(FeedEntry::postId, e -> e, (a, b) -> a))
            .values().stream()
            .sorted(NEWEST_FIRST)
            .limit(limit)
            .toList();
    }
}

• filter(cursor::isAfter) keeps only posts older than the cursor. When two posts have the same millisecond, postId breaks ties.
• toMap(..., (a, b) -> a) deduplicates. A post may appear in both sources, for example when an author just crossed the celebrity threshold.
• Then the list is sorted newest-first and truncated to the limit.

FeedService assembles the pieces:

public FeedPage getFeed(long userId, Cursor cursor, int limit) {
    double max = cursor == Cursor.START ? Double.POSITIVE_INFINITY : cursor.createdAtMillis();

    List<FeedEntry> pushed = redis.opsForZSet()
        .reverseRangeByScoreWithScores("feed:" + userId, 0, max, 0, limit + 1)
        .stream()
        .map(t -> new FeedEntry(Long.parseLong(t.getValue()), t.getScore().longValue()))
        .toList();

    List<Long> celebs = followRepository.findCelebrityFolloweeIds(userId);
    List<FeedEntry> pulled = celebrityTimelineCache.recentPosts(celebs, cursor, limit);

    List<FeedEntry> page = FeedMerger.merge(pushed, pulled, cursor, limit);
    List<PostDto> items = postCache.hydrate(page); // multi-get, skip deleted posts
    String next = page.isEmpty() ? null
        : page.getLast().createdAtMillis() + ":" + page.getLast().postId();
    return new FeedPage(items, next);
}

The Redis query uses max inclusive of the boundary, because there may be multiple posts with the same millisecond as the cursor. FeedMerger will filter precisely again.

8. Testing the Merge Logic

class FeedMergerTest {

    @Test
    void mergesSortsDedupesAndRespectsCursor() {
        List<FeedEntry> pushed = List.of(
            new FeedEntry(5, 5_000), new FeedEntry(3, 3_000), new FeedEntry(1, 1_000));
        List<FeedEntry> pulled = List.of(
            new FeedEntry(4, 4_000), new FeedEntry(3, 3_000), new FeedEntry(2, 2_000));

        List<FeedEntry> firstPage = FeedMerger.merge(pushed, pulled, Cursor.START, 3);

        assertThat(firstPage).extracting(FeedEntry::postId).containsExactly(5L, 4L, 3L);

        Cursor next = new Cursor(3_000, 3);
        List<FeedEntry> secondPage = FeedMerger.merge(pushed, pulled, next, 3);

        assertThat(secondPage).extracting(FeedEntry::postId).containsExactly(2L, 1L);
    }
}

Post 3 is in both sources but appears only once. The second page starts right after the cursor, with no repeats or gaps.

9. Challenges and Solutions

9.1. Celebrity Hot Keys

Everyone reading celeb:{id}:recent concentrates load onto a single Redis node. Three approaches:

• Add a local cache (Caffeine) on each instance with TTL 2–5 seconds. 100K QPS then becomes just a few hundred requests/second to Redis.
• Enable Redis read replicas for these keys.
• When a celebrity posts, actively write to the timeline cache instead of waiting for a cache miss. This avoids the thundering herd.

9.2. Consistency When Deleting Posts, Unfollowing, Following

• Deleting a post: No need to scan millions of feeds. When hydrating, postCache skips posts with deleted = true. A background job gradually cleans up IDs later.
• Unfollowing: Filter when reading using the user’s current followee set. Simultaneously emit an event to asynchronously delete that person’s posts from feed:{userId}.
• Following someone new: Run backfill, pulling about 20 recent posts from the newly followed user, then ZADD them into the feed.

Because the feed contains only IDs, all discrepancies lie in the “reference list”. Content is always correct according to the source of truth.

9.3. Lost Events Between DB and Kafka

If you INSERT a post then call kafkaTemplate.send(), the send may fail after the transaction has committed. The post then exists but is never fanned out.

Note: Use a transactional outbox. Write the post and a row to the outbox table in the same transaction. A relay (or CDC like Debezium) reads the outbox and pushes to Kafka. The consumer side must be idempotent, and ZADD already satisfies this.

9.4. Burst Traffic

Major events like soccer matches or New Year’s can cause post volume to increase 10x. Mitigation strategies:

• Kafka acts as a buffer. The consumer processes at its own pace; if lag increases, the feed delays by a few more seconds. This is acceptable by our constraints.
• Partition by authorId to preserve post order from the same author. Scale consumers by increasing the number of partitions.
• Skip inactive users. They don’t get pushed. When they return and hit a cache miss, feed is rebuilt using fan-out on read, then written to Redis.
• Lower the celebrity threshold dynamically. When lag exceeds a threshold, reduce CELEBRITY_THRESHOLD to move more authors into pull mode.

9.5. Redis Memory Estimation

~200 million active users × 500 entries × ~60 bytes ≈ 6 TB

This requires a Redis Cluster. You can lower FEED_MAX to 200–300, since very few people scroll beyond a few hundred posts. Deeper scrolling falls back to pull.

10. When Not to Use This Model

• Small applications (tens of thousands of users): fan-out on read with a single indexed SQL query is sufficient. Adding Kafka and Redis only increases complexity.
• Algorithm-ranked feeds (ML-based ranking): the pushed feed serves only as “candidates”. Ranking runs separately at read time, so time-based score is no longer the final order.
• Strong consistency requirement (post must appear immediately for all followers): asynchronous fan-out can’t guarantee this. You must pull, or insert the author’s own post into the response immediately after posting (read-your-writes).

11. Conclusion

Fan-out on write optimizes reads but amplifies writes. Fan-out on read does the opposite. With skewed follower distribution like social networks, the practical answer is a hybrid model: push for typical authors, pull for celebrities, then merge at read time.

Key points to remember when implementing:

• Feed stores only IDs; score is epoch milliseconds, not Snowflake ID.
• Use cursor (createdAt, postId) for stable pagination.
• Consumer must be idempotent; combine with outbox to avoid losing events.
• Handle post deletion and unfollowing by filtering when reading.
• Use Kafka with dynamic celebrity threshold to absorb traffic spikes.