EN
FrançaisPortuguêsEnglishEspañol한국어日本語DeutschTiếng Việt简体中文
Home / Architecture & Design / Real-world case studies

Designing a Google Drive-like File Storage and Sharing System

Cris The Dev · 10/13/2026 · 11 min read · 75

1. Overview

This article designs a file storage and sharing system similar to Google Drive. The content follows this sequence: finalize requirements, build the data model, define APIs, describe upload and sync flows. Then the article focuses on three challenging points: write conflicts, consistency, and burst traffic.

After reading, you’ll have a design suitable for a system design interview. You’ll also have Spring Boot code snippets illustrating the core components.

2. Requirements

2.1. Functional Requirements

  • Upload and download files, maximum size 10 GB.
  • Organize files in a directory tree, support renaming and moving.
  • Share files or folders with others with VIEWER or EDITOR permissions. Support public links as well.
  • Save version history and allow recovery of old versions.
  • Sync changes across multiple devices of the same user.

2.2. Scale and Constraints

Assumptions for calculation:

Users: 50 million, DAU: 10 million
Average upload: 2 files/person/day, ~ 500 KB/file
=> ~ 20 million files/day, ~ 230 files/second (peak x5 ≈ 1,200/second)
=> ~ 10 TB new data/day, ~ 3.6 PB/year (not counting replicas)
Read:write ratio ≈ 5:1

From these numbers, three constraints emerge:

  • Data must never be lost. Durability must reach “11 nines”. Therefore, file content must live on object storage like S3, not on app server disks.
  • Metadata needs strong consistency. A file cannot have two “latest versions”. This uses a relational database.
  • App servers must not become a bandwidth bottleneck. File bytes should go straight from client to storage.

3. Overall Architecture

Client ──► API Gateway (auth, rate limit) ──► Metadata Service ──► PostgreSQL (sharded by owner_id)
   │                                              │
   │  presigned URL                               ├──► Redis (cache permissions, rate limit)
   ▼                                              └──► Outbox ──► Kafka ──► Notification / Search / Thumbnail
Object Storage (S3) ◄── CDN (download shared files widely)

The main idea is to separate metadata from content. Files are split into 4 MB blocks. Each block is identified by the SHA-256 hash of its own content (content-addressed). This approach delivers three benefits:

  • Free deduplication. Two identical blocks are stored once.
  • Resume uploads. If the network drops, the client only re-sends missing blocks.
  • Delta sync. Editing a small section in a 1 GB file only uploads a few blocks.

4. Data Model

CREATE TABLE files (
    id            UUID PRIMARY KEY,
    owner_id      BIGINT       NOT NULL,
    parent_id     UUID         REFERENCES files(id),
    name          VARCHAR(255) NOT NULL,
    is_folder     BOOLEAN      NOT NULL,
    head_version  INT          NOT NULL DEFAULT 0,
    size_bytes    BIGINT       NOT NULL DEFAULT 0,
    deleted_at    TIMESTAMPTZ,
    UNIQUE (parent_id, name)
);

CREATE TABLE file_versions (
    file_id     UUID   NOT NULL REFERENCES files(id),
    version     INT    NOT NULL,
    block_list  TEXT[] NOT NULL,          -- list of hashes in order
    created_by  BIGINT NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (file_id, version)
);

CREATE TABLE blocks (
    hash       CHAR(64) PRIMARY KEY,      -- SHA-256 hex
    size_bytes INT      NOT NULL,
    ref_count  BIGINT   NOT NULL DEFAULT 0
);

CREATE TABLE shares (
    file_id    UUID   NOT NULL REFERENCES files(id),
    grantee_id BIGINT NOT NULL,
    role       VARCHAR(10) NOT NULL,      -- VIEWER | EDITOR
    PRIMARY KEY (file_id, grantee_id)
);

CREATE TABLE change_journal (
    seq      BIGSERIAL PRIMARY KEY,
    user_id  BIGINT NOT NULL,
    file_id  UUID   NOT NULL,
    action   VARCHAR(20) NOT NULL
);

Role of each part in the schema:

  • head_version is the optimistic locking column for each file (see section 6.1).
  • file_versions.block_list makes each version a list of hashes. This way, recovering an old version is just creating a new version pointing to the old block_list, without copying any bytes.
  • blocks.ref_count helps the garbage collector know which blocks are no longer in use to delete from S3.
  • change_journal is a change log per user. Devices sync by reading from a seq (cursor) onward.
  • UNIQUE (parent_id, name) prevents two files with the same name in a folder at the DB level.

Note: Checking permissions along the directory tree, like “if you’re granted access to parent folder, you can see child files”, is a recursive query and expensive. Cache permission results in Redis with key perm:{userId}:{fileId} and short TTL. When sharing changes, delete the cache.

5. APIs and Upload Flow

Upload proceeds in three steps:

POST /api/files/{fileId}/uploads      {baseVersion, blocks:[hash...]} -> {missing:[{hash,url}]}
PUT  <presigned-url>                  (client sends block directly to S3)
POST /api/files/{fileId}/commit       {baseVersion, blocks:[hash...]} -> {version}
GET  /api/changes?cursor=1234         -> {changes:[...], nextCursor}

Maven dependencies to use:

<dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
    <version>2.28.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
</dependency>

5.1. Identifying Missing Blocks

The dedup logic should be written in pure Java for easy testing:

public class ChunkPlanner {

    public List<String> missingBlocks(List<String> requested, Set<String> existing) {
        return requested.stream()
                .distinct()
                .filter(hash -> !existing.contains(hash))
                .toList();
    }
}

Client sends a list of hashes in order. Server queries the blocks table to get the existing set. Only hashes that don’t exist yet need to be uploaded. distinct() handles the case where a file contains multiple identical blocks, for example, a region of all zero bytes.

5.2. Granting Presigned URLs

@RestController
@RequestMapping("/api/files")
public class UploadController {

    private final S3Presigner presigner;
    private final BlockRepository blockRepository;
    private final ChunkPlanner planner = new ChunkPlanner();

    public UploadController(S3Presigner presigner, BlockRepository blockRepository) {
        this.presigner = presigner;
        this.blockRepository = blockRepository;
    }

    @PostMapping("/{fileId}/uploads")
    public UploadPlan initiate(@PathVariable UUID fileId, @RequestBody UploadRequest req) {
        Set<String> existing = blockRepository.findExisting(req.blocks());
        List<BlockUrl> urls = planner.missingBlocks(req.blocks(), existing).stream()
                .map(hash -> new BlockUrl(hash, presign(hash)))
                .toList();
        return new UploadPlan(urls);
    }

    private String presign(String hash) {
        return presigner.presignPutObject(r -> r
                .signatureDuration(Duration.ofMinutes(15))
                .putObjectRequest(o -> o.bucket("drive-blocks").key("blocks/" + hash)))
            .url().toString();
    }
}

record UploadRequest(int baseVersion, List<String> blocks) {}
record BlockUrl(String hash, String url) {}
record UploadPlan(List<BlockUrl> missing) {}

For each missing block, the server signs a PUT URL valid for 15 minutes. Client sends bytes directly to S3, so the app server only processes a few KB of JSON instead of megabytes of data. Example response:

{"missing":[{"hash":"9f86d0...","url":"https://drive-blocks.s3.amazonaws.com/blocks/9f86d0...?X-Amz-Signature=..."}]}

Note: Since the key is a hash, a client could send content that doesn’t match the declared hash. At commit time, the server must verify again, for example using the x-amz-checksum-sha256 checksum that S3 supports. If you skip this step, a user could “poison” shared blocks used by others.

5.3. Test for Dedup Logic

class ChunkPlannerTest {

    private final ChunkPlanner planner = new ChunkPlanner();

    @Test
    void shouldReturnOnlyBlocksNotYetStored() {
        List<String> requested = List.of("a1", "b2", "a1", "c3");
        Set<String> existing = Set.of("b2");

        assertThat(planner.missingBlocks(requested, existing))
                .containsExactly("a1", "c3");
    }

    @Test
    void shouldReturnEmptyWhenEverythingIsDeduplicated() {
        assertThat(planner.missingBlocks(List.of("a1"), Set.of("a1"))).isEmpty();
    }
}

The second test simulates the case where two people upload the same installed software. The second person doesn’t need to send any bytes, and the upload is “instant”.

6. Difficult Points and Solutions

6.1. Conflict: Two Devices Editing the Same File

Laptop and phone both edit from version 7. If you apply “last-write-wins”, one side’s changes will be lost. The solution is to use optimistic locking: client sends baseVersion, and commit only succeeds if head_version still equals that value.

@Service
public class CommitService {

    private final JdbcTemplate jdbc;

    public CommitService(JdbcTemplate jdbc) {
        this.jdbc = jdbc;
    }

    @Transactional
    public int commit(UUID fileId, long userId, int baseVersion, List<String> blocks, long size) {
        int updated = jdbc.update("""
                UPDATE files SET head_version = head_version + 1, size_bytes = ?
                WHERE id = ? AND head_version = ? AND deleted_at IS NULL
                """, size, fileId, baseVersion);
        if (updated == 0) {
            throw new VersionConflictException(fileId, baseVersion);
        }
        int newVersion = baseVersion + 1;
        jdbc.update("INSERT INTO file_versions(file_id, version, block_list, created_by) VALUES (?,?,?,?)",
                fileId, newVersion, blocks.toArray(String[]::new), userId);
        jdbc.update("UPDATE blocks SET ref_count = ref_count + 1 WHERE hash = ANY(?)",
                (Object) blocks.toArray(String[]::new));
        jdbc.update("INSERT INTO change_journal(user_id, file_id, action) VALUES (?,?,'MODIFIED')",
                userId, fileId);
        return newVersion;
    }
}

The UPDATE ... WHERE head_version = ? statement both checks and increments the version in an atomic operation. The later commit receives updated == 0. Then the API returns 409 Conflict. Client doesn’t overwrite but saves its version as a new file, for example report (conflict from Lan's Laptop).docx. Dropbox and Google Drive handle binary files this way too.

You shouldn’t use pessimistic locking (SELECT ... FOR UPDATE held throughout the upload). Upload can take many minutes, and holding the lock that long will congest the DB. Real-time collaborative editing like Google Docs is a different problem, requiring OT or CRDT, and is outside the scope of this article.

6.2. Consistency Between DB, Storage, and Other Services

There are two questions to answer.

First: what if a block is already on S3 but commit fails? It’s fine. Metadata only points to blocks after commit succeeds. Orphaned blocks with ref_count = 0 will be deleted by a GC job after 24 hours. The order is always: write bytes first, write metadata after, so metadata never points to non-existent data.

Second: how do you notify other devices, search indexes, and thumbnail services? If you call kafkaTemplate.send() directly in the transaction, you face the dual-write problem: DB rolls back but the message already sent, or vice versa. The solution is the transactional outbox pattern. Write events to the outbox table in the same transaction as commit, then a relay pushes events to Kafka:

@Component
public class OutboxRelay {

    private final JdbcTemplate jdbc;
    private final KafkaTemplate<String, String> kafka;

    public OutboxRelay(JdbcTemplate jdbc, KafkaTemplate<String, String> kafka) {
        this.jdbc = jdbc;
        this.kafka = kafka;
    }

    @Scheduled(fixedDelay = 500)
    @Transactional
    public void publish() {
        List<Map<String, Object>> rows = jdbc.queryForList("""
                SELECT id, owner_id, payload FROM outbox
                ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED
                """);
        for (Map<String, Object> row : rows) {
            kafka.send("file-events", row.get("owner_id").toString(), (String) row.get("payload")).join();
            jdbc.update("DELETE FROM outbox WHERE id = ?", row.get("id"));
        }
    }
}

Key points to note:

  • FOR UPDATE SKIP LOCKED allows multiple relay instances to run without sending the same batch twice.
  • Kafka key is owner_id, so all events from one user go to the same partition and maintain order.
  • The relay can still send duplicates, for example if it crashes after send but before DELETE. Therefore the consumer must be idempotent: store processed (file_id, version) and skip duplicates.
  • Devices don’t need to receive content over Kafka. Notification service just pushes a “there are changes” signal over WebSocket. The device then calls GET /api/changes?cursor=... to read from change_journal. If a device is offline for three days, it just reads from its old cursor onward, losing no events.

6.3. Burst Traffic

There are three common scenarios.

A public shared file goes viral, for example an exam is shared with hundreds of thousands of people. Downloads go through CDN via signed URLs. Blocks are immutable since the key is the hash, so they can be cached forever (Cache-Control: immutable). The result is the origin bears almost no load.

First thing in the morning, millions of devices sync simultaneously. Clients use exponential backoff with jitter. Server rate-limits per user with Redis:

@Component
public class RateLimiter {

    private final StringRedisTemplate redis;

    public RateLimiter(StringRedisTemplate redis) {
        this.redis = redis;
    }

    public boolean tryAcquire(long userId, int limitPerMinute) {
        String key = "rl:%d:%d".formatted(userId, Instant.now().getEpochSecond() / 60);
        Long count = redis.opsForValue().increment(key);
        if (count != null && count == 1) {
            redis.expire(key, Duration.ofSeconds(70));
        }
        return count != null && count <= limitPerMinute;
    }
}

Each user has a counter per minute. INCR is atomic on Redis, so multiple app servers sharing the counter still count correctly. When the threshold is exceeded, the API returns 429 Too Many Requests with Retry-After header. Fixed window allows double burst at the boundary of two minutes. For more precision, use sliding window or token bucket implemented with Lua script.

A single account with millions of files. Shard PostgreSQL by owner_id so a user’s entire directory tree lives on the same shard. This way, renaming or moving a folder is just a local transaction. Moving just changes the parent_id of one row, doesn’t touch child files.

Note: A folder shared between accounts will live on the owner’s shard. The person granted access crosses shards via Metadata Service, not by joining directly between DBs.

6.4. Things You Shouldn’t Do

  • Store files in BYTEA/BLOB columns. Backups grow huge, replication slows, and the DB becomes a bottleneck.
  • Let files flow through the app server. Server bandwidth and memory scale linearly with file size.
  • Delete blocks immediately when deleting a file. Blocks might be used by other files or versions. Always use GC based on ref_count and soft delete with deleted_at to support recycle bins.
  • Use fixed-size chunks for all file types when good delta sync matters. Inserting a byte at the start will shift every block after. Content-defined chunking (rolling hash) solves it but is more complex. Only apply when real data shows clear benefit.

7. Conclusion

Designing a Google Drive-like system revolves around one central decision: separate metadata (relational database, strong consistency) from content (object storage, immutable blocks identified by hash). From there, dedup, resume upload, version history, and CDN caching come almost for free.

The difficult points have fairly standard solutions:

  • Write conflicts: optimistic locking with head_version and creating conflict copies instead of overwriting.
  • Consistency: write bytes first, commit metadata after, combined with transactional outbox and idempotent consumer.
  • Burst traffic: presigned URLs, CDN, rate limit with Redis, and shard by owner.

When using this article as a framework for an interview or real project, start from the scale estimation in section 2. Each number there determines an architectural choice downstream.

system-designspring-bootkafkarediscase-study
App by the authorHoanTien.proCashback on Shopee, TikTok Shop and Lazada (Vietnam)iOS · Android · Web

More on this topic