System Design for High-Traffic URL Shortener

1. Overview

URL shortening is a familiar system design problem. The core logic is simply mapping a short code to a long URL. The challenge lies in scale: tens of thousands of redirects per second, sudden viral links, and the requirement to never generate duplicate codes.

This article covers the entire process: requirements, load estimation, data model, API, code generation approach, caching, and challenging points. After reading, you’ll have a design framework with Spring Boot 3 code to build yourself or use in interviews.

2. Requirements

2.1. Functional Requirements

• Create short links from long URLs. Users can set custom aliases, e.g., /sale-2026.
• Accessing a short link redirects to the original URL.
• Links can have an expiration time (expires_at) and can be disabled by the owner.
• Track click counts per link. Statistics can lag by a few seconds.

2.2. Scale

Assume 100 million new links per month and a 100:1 read-to-write ratio.

Write:   100M / (30 * 86400s)  ≈ 40 links/s   (peak ×10 ≈ 400/s)
Read:    40 × 100              ≈ 4,000 rps  (peak ×10 ≈ 40,000 rps)
Storage 5 years: 6 billion links × ~500 bytes ≈ 3 TB
Code space 7-character base62: 62^7 ≈ 3.5 trillion  (plenty for 6 billion)

Two conclusions from these numbers. First, the system is read-heavy, so caching is central to the design. Second, write throughput is low, so a single PostgreSQL primary suffices for now.

2.3. Constraints

• Redirects have p99 latency < 20 ms.
• Short codes must be unique and unpredictable in order, so others cannot enumerate private links.
• The system continues to redirect even if analytics or Kafka fails.

3. Data Model and API

Dependencies to use (versions managed by Spring Boot):

<dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-jdbc</artifactId>
    </dependency>
    <dependency>
        <groupId>org.postgresql</groupId>
        <artifactId>postgresql</artifactId>
        <scope>runtime</scope>
    </dependency>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-data-redis</artifactId>
    </dependency>
    <dependency>
        <groupId>org.springframework.kafka</groupId>
        <artifactId>spring-kafka</artifactId>
    </dependency>
    <dependency>
        <groupId>com.github.ben-manes.caffeine</groupId>
        <artifactId>caffeine</artifactId>
    </dependency>
</dependencies>

PostgreSQL Schema:

CREATE TABLE links (
    code        VARCHAR(16) PRIMARY KEY,
    long_url    TEXT        NOT NULL,
    owner_id    BIGINT,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    expires_at  TIMESTAMPTZ,
    disabled    BOOLEAN     NOT NULL DEFAULT false
);
CREATE INDEX idx_links_owner ON links (owner_id, created_at DESC);

CREATE SEQUENCE link_id_seq START 1 INCREMENT BY 1000;

Explanation of key decisions:

• code is the primary key. The hottest query is WHERE code = ?, so it goes straight to the primary key index.
• idx_links_owner serves the “My Links” page.
• link_id_seq increments by 1000 each time. Each call to nextval gives the application a whole block of IDs (see section 4.2).

Two main APIs:

POST /api/links          {"url": "https://shop.vn/...", "alias": null}
  -> 201 {"code": "0bX3kQz", "shortUrl": "https://nonla.ly/0bX3kQz"}
  -> 409 if alias already exists

GET  /{code}
  -> 302 Location: https://shop.vn/...
  -> 404 if nonexistent / expired / disabled

Note: Use 302, not 301. Browsers cache 301 permanently. Then you lose click metrics and cannot disable already-spread links.

4. Generating Short Codes

4.1. Choosing a Strategy

Method	Pros	Cons
Hash URL (MD5 then truncate to 7 chars)	No state needed	May collide, must check and retry
Random 7 chars	Simple	More links mean more collisions, must check DB each time
Counter + base62	Never collides	Reveals sequence, needs a central ID source

This article chooses the third approach and mitigates its two drawbacks. To hide the sequence, we scramble IDs with a bijection. To reduce dependence on a central ID source, we allocate IDs in blocks.

import java.math.BigInteger;

public final class ShortCode {
    private static final String ALPHABET =
            "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ";
    private static final int LENGTH = 7;
    private static final BigInteger SPACE = BigInteger.valueOf(62).pow(LENGTH);
    // Odd number, not divisible by 31 => coprime with 62^7
    private static final BigInteger MULTIPLIER = BigInteger.valueOf(1_580_030_173L);

    private ShortCode() {}

    public static String of(long id) {
        if (id < 0 || id >= SPACE.longValue()) {
            throw new IllegalArgumentException("id outside code space: " + id);
        }
        long mixed = BigInteger.valueOf(id).multiply(MULTIPLIER).mod(SPACE).longValue();
        char[] out = new char[LENGTH];
        for (int i = LENGTH - 1; i >= 0; i--) {
            out[i] = ALPHABET.charAt((int) (mixed % 62));
            mixed /= 62;
        }
        return new String(out);
    }
}

How it works:

• The modular multiplication id × MULTIPLIER mod 62^7 is a bijection because MULTIPLIER is coprime with 62^7. Each ID produces exactly one code, and two different IDs never produce the same code.
• The loop converts to base62 and left-pads with zeros, so the code is always exactly 7 characters.
• IDs 1 and 2 produce different codes starting from the second character, so outsiders find it hard to guess the next link.
• BigInteger avoids overflow when multiplying. At 400 writes/s, this cost is negligible.
• This only makes codes hard to guess, not encrypted. Anyone knowing MULTIPLIER can reverse it. Truly private links need additional authentication.

4.2. Allocating IDs in Blocks

Calling the DB for each ID makes the primary a bottleneck. Instead, each instance takes a block of 1000 IDs and allocates them gradually in memory:

import java.util.concurrent.locks.ReentrantLock;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Component;

@Component
public class IdBlockAllocator {
    private static final int BLOCK_SIZE = 1000; // must match INCREMENT BY
    private final JdbcTemplate jdbc;
    private final ReentrantLock lock = new ReentrantLock();
    private long next;
    private long end;

    public IdBlockAllocator(JdbcTemplate jdbc) {
        this.jdbc = jdbc;
    }

    public long nextId() {
        lock.lock();
        try {
            if (next >= end) {
                Long start = jdbc.queryForObject("SELECT nextval('link_id_seq')", Long.class);
                next = start;
                end = start + BLOCK_SIZE;
            }
            return next++;
        } finally {
            lock.unlock();
        }
    }
}

Key points:

• nextval is atomic. Two instances never get the same block, so no distributed lock is needed.
• At 400 writes/s, each instance touches the DB only a few times per second.
• Code uses ReentrantLock instead of synchronized. On Java 21, virtual threads get pinned to a carrier thread while waiting on I/O inside a synchronized block.
• Instance restarts waste the remaining IDs in a block. The code space has 3.5 trillion values, so this is acceptable.

5. Creating a Link Flow

import java.net.URI;
import java.time.Duration;
import org.springframework.dao.DuplicateKeyException;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;

@Service
public class LinkWriteService {
    static final String KEY = "link:";
    private final JdbcTemplate jdbc;
    private final StringRedisTemplate redis;
    private final IdBlockAllocator allocator;

    public LinkWriteService(JdbcTemplate jdbc, StringRedisTemplate redis, IdBlockAllocator allocator) {
        this.jdbc = jdbc;
        this.redis = redis;
        this.allocator = allocator;
    }

    public String create(String longUrl, String alias, Long ownerId) {
        String scheme = URI.create(longUrl).getScheme();
        if (!"http".equals(scheme) && !"https".equals(scheme)) {
            throw new IllegalArgumentException("Only http/https accepted");
        }
        String code = (alias != null) ? alias : ShortCode.of(allocator.nextId());
        try {
            jdbc.update("INSERT INTO links(code, long_url, owner_id) VALUES (?, ?, ?)",
                    code, longUrl, ownerId);
        } catch (DuplicateKeyException e) {
            throw new AliasTakenException(code);
        }
        redis.opsForValue().set(KEY + code, longUrl, Duration.ofHours(24));
        return code;
    }
}

class AliasTakenException extends RuntimeException {
    AliasTakenException(String code) { super("Alias already taken: " + code); }
}

The flow has four steps:

1. Check the scheme to block javascript: or file:. Skip this and your service becomes a tool for spreading malicious links.
2. Get the code: use custom alias, or generate from ID.
3. INSERT. The primary key constraint is the final defense against duplicate codes.
4. Write through to Redis so the first redirect doesn’t hit the DB.

In production, custom aliases should use different characters or length from auto-generated codes, e.g., require a - or be at least 8 chars. Otherwise, a user alias might pre-claim a code the generator would produce later, causing DuplicateKeyException when it shouldn’t happen.

6. Redirect Flow and Multi-Layer Caching

At 40,000 rps during peak, most requests must be served without touching the DB. We use three tiers: Caffeine in JVM, shared Redis, then PostgreSQL.

import com.github.benmanes.caffeine.cache.Cache;
import com.github.benmanes.caffeine.cache.Caffeine;
import java.time.Duration;
import java.util.List;
import java.util.Optional;
import org.springframework.data.redis.core.StringRedisTemplate;
import org.springframework.jdbc.core.JdbcTemplate;
import org.springframework.stereotype.Service;

@Service
public class LinkReadService {
    private static final String NOT_FOUND = ""; // sentinel for negative cache
    private final Cache<String, String> local = Caffeine.newBuilder()
            .maximumSize(100_000)
            .expireAfterWrite(Duration.ofSeconds(30))
            .build();
    private final JdbcTemplate jdbc;
    private final StringRedisTemplate redis;

    public LinkReadService(JdbcTemplate jdbc, StringRedisTemplate redis) {
        this.jdbc = jdbc;
        this.redis = redis;
    }

    public Optional<String> resolve(String code) {
        String url = local.get(code, this::loadFromRedisOrDb);
        return url.isEmpty() ? Optional.empty() : Optional.of(url);
    }

    private String loadFromRedisOrDb(String code) {
        String key = LinkWriteService.KEY + code;
        String cached = redis.opsForValue().get(key);
        if (cached != null) return cached;

        List<String> rows = jdbc.queryForList("""
                SELECT long_url FROM links
                WHERE code = ? AND NOT disabled
                  AND (expires_at IS NULL OR expires_at > now())""", String.class, code);
        String url = rows.isEmpty() ? NOT_FOUND : rows.getFirst();
        redis.opsForValue().set(key, url, url.isEmpty() ? Duration.ofMinutes(5) : Duration.ofHours(24));
        return url;
    }
}

Key points:

• Caffeine’s local.get(code, loader) coalesces concurrent loads for the same key. If 5,000 requests ask for an uncached code simultaneously, only one thread goes to Redis/DB. Others wait for that result. This is how to prevent cache stampede within an instance.
• Negative caching: nonexistent codes are also cached as an empty string for 5 minutes. This prevents bots from enumeration attacks on the DB.
• Caffeine doesn’t cache null, so we use sentinel "" instead.

The controller returns 302 and pushes click events to Kafka:

import java.net.URI;
import java.util.Map;
import org.springframework.http.HttpStatus;
import org.springframework.http.ResponseEntity;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.web.bind.annotation.*;

@RestController
public class LinkController {
    private final LinkWriteService writer;
    private final LinkReadService reader;
    private final KafkaTemplate<String, String> kafka;

    public LinkController(LinkWriteService writer, LinkReadService reader,
                          KafkaTemplate<String, String> kafka) {
        this.writer = writer;
        this.reader = reader;
        this.kafka = kafka;
    }

    public record CreateLinkRequest(String url, String alias) {}

    @PostMapping("/api/links")
    public ResponseEntity<Map<String, String>> create(@RequestBody CreateLinkRequest req) {
        String code = writer.create(req.url(), req.alias(), null);
        return ResponseEntity.status(HttpStatus.CREATED)
                .body(Map.of("code", code, "shortUrl", "https://nonla.ly/" + code));
    }

    @GetMapping("/{code:[0-9a-zA-Z-]{1,16}}")
    public ResponseEntity<Void> redirect(@PathVariable String code) {
        return reader.resolve(code)
                .map(url -> {
                    kafka.send("link-clicks", code, String.valueOf(System.currentTimeMillis()));
                    return ResponseEntity.status(HttpStatus.FOUND).location(URI.create(url)).<Void>build();
                })
                .orElse(ResponseEntity.notFound().build());
    }

    @ExceptionHandler(AliasTakenException.class)
    ResponseEntity<String> conflict(AliasTakenException e) {
        return ResponseEntity.status(HttpStatus.CONFLICT).body(e.getMessage());
    }
}

kafka.send is asynchronous, so redirects don’t wait for the broker. A separate consumer batches events by code every few seconds, then writes them to the analytics table. Configure the producer for low latency:

spring.kafka.producer.acks=1
spring.kafka.producer.properties.linger.ms=20
spring.kafka.producer.properties.max.block.ms=50

max.block.ms=50 prevents the redirect thread from hanging long if the broker fails, matching the requirement “redirects work even if analytics fails”. We accept losing a few click events in that scenario.

7. Hard Problems and Solutions

7.1. Race Conditions

• Two people setting the same alias: Don’t SELECT then INSERT. Both requests might see ‘doesn’t exist’ and insert anyway. Let the DB’s unique constraint decide and catch DuplicateKeyException as in section 5.
• Multiple instances generating codes: Sequences allocate separate blocks per instance, so no contention.

7.2. Consistency

• Read immediately after create: Reading from a replica, replication lag might make the new link return 404. Writing to Redis immediately (write-through) hides this lag.
• Disabling a link: Update the DB then delete the Redis key. Caffeine still holds the old value up to 30 seconds per instance. For immediate blocking, e.g., scam links, publish an invalidation event via Redis Pub/Sub or Kafka so instances delete their local cache.
• Expiration: Cache TTL should be min(24h, expires_at - now). Otherwise, an expired link redirects for up to 24 more hours.

7.3. Traffic Spikes

• Hot key: A viral link concentrates all traffic on one Redis shard. Local Caffeine absorbs most of it. Each instance queries Redis about once per 30 seconds per key.
• Cold start: New instances have empty Caffeine, but load coalescing from section 6 protects Redis from thundering herd.
• Link creation spam: Rate-limit by API key or IP, e.g., use token bucket on Redis, before calling LinkWriteService.

8. Testing Code Generation Behavior

The most important properties of the system are: codes never collide, have fixed length, and don’t reveal order:

import static java.util.stream.Collectors.toSet;
import static org.assertj.core.api.Assertions.assertThat;
import static org.assertj.core.api.Assertions.assertThatThrownBy;

import java.util.Set;
import java.util.stream.LongStream;
import org.junit.jupiter.api.Test;

class ShortCodeTest {

    @Test
    void codesHaveCorrectLengthAndAlphabet() {
        assertThat(ShortCode.of(1)).hasSize(7).matches("[0-9a-zA-Z]{7}");
    }

    @Test
    void noCollisionsIn100kSequentialIds() {
        Set<String> codes = LongStream.range(0, 100_000).mapToObj(ShortCode::of).collect(toSet());
        assertThat(codes).hasSize(100_000);
    }

    @Test
    void sequentialIdsDoNotProduceSequentialCodes() {
        assertThat(ShortCode.of(1).substring(0, 5)).isNotEqualTo(ShortCode.of(2).substring(0, 5));
    }

    @Test
    void rejectIdOutsideCodespace() {
        assertThatThrownBy(() -> ShortCode.of(-1)).isInstanceOf(IllegalArgumentException.class);
    }
}

The second test checks a sample and doesn’t prove bijectivity. Bijectivity comes from math (gcd = 1). Tests only protect you from someone accidentally changing MULTIPLIER to an even number.

9. Common Mistakes and When Not to Overcomplicate

Note: Changing MULTIPLIER or ALPHABET after launch makes new IDs collide with old codes. Treat them as permanent constants.

• Writing each click directly to the links table (UPDATE ... SET clicks = clicks + 1) creates hot row locks on viral links. Batch through Kafka instead.
• Unbounded Caffeine leads to OutOfMemoryError when enumeration attacks hit large numbers of random codes.
• For an internal tool with only hundreds of links per day, a PostgreSQL table and a controller are enough. Redis, Kafka, and ID block allocation only pay off when load actually demands it.

10. Conclusion

High-traffic URL shortener design revolves around three decisions. Generate codes using block-allocated counters with a bijection, for uniqueness without predictability. Use multi-layer caching with negative caching and load coalescing to handle hot keys and enumeration. Push non-critical work like click analytics off the redirect path via Kafka.

Start with load estimation. The numbers in section 2 tell you when to add each layer, rather than building everything from day one.