Hot keys and big keys in Redis: detection and handling

1. Overview

Redis processes commands on a single main thread. Because of this, even a single oversized key (big key) or a key that gets hammered with traffic (hot key) can slow down your entire instance. In this article, you’ll learn how to detect both types of keys using Redis’s built-in tools and Java code. Then, we’ll handle them in Spring Boot 3.x applications using key sharding, safe deletion, local cache, and key replication techniques.

2. What are hot keys and big keys?

A big key is one that consumes a lot of memory or contains too many elements. There’s no hard standard, but many teams use these as guidelines:

• String larger than around 10 KB.
• Hash, List, Set, or ZSet with more than around 5,000 elements, or total storage exceeding a few MB.

A hot key is one that receives an abnormally large proportion of requests compared to other keys. Examples include keys holding flash sale data, homepage config, or shared settings.

These two types of keys cause several problems:

• Blocking the main thread: HGETALL on a hash with a million fields, or DEL on a huge ZSet, can block Redis for hundreds of milliseconds. Every other client has to wait.
• Network bandwidth bottleneck: reading a 2 MB value with 5,000 requests/second translates to roughly 10 GB/second of traffic.
• Load imbalance in Redis Cluster: a single key always lives in one slot, meaning one node. A hot or big key concentrates load on that node, no matter how many nodes the cluster has.
• Slow replication and failover: big keys make sync operations and RDB loading take much longer.

3. Detecting big keys

3.1. Using redis-cli --bigkeys and --memkeys

The quickest approach is to use redis-cli’s built-in options:

redis-cli -h 127.0.0.1 -p 6379 --bigkeys -i 0.01

This command scans the entire keyspace using SCAN, so it doesn’t block the server like KEYS * would. The -i 0.01 parameter sleeps for 10 ms after every 100 SCAN commands, reducing pressure on production servers. The output looks like this:

[00.00%] Biggest hash   found so far '"cart:10023"' with 48211 fields
[12.35%] Biggest string found so far '"product:detail:88"' with 1048576 bytes
[45.10%] Biggest zset   found so far '"leaderboard:global"' with 920144 members

-------- summary -------
Biggest   hash found '"cart:10023"' has 48211 fields
Biggest string found '"product:detail:88"' has 1048576 bytes
Biggest   zset found '"leaderboard:global"' has 920144 members

--bigkeys measures by element count. It only reports the biggest key of each data type. If you care about actual memory usage, use --memkeys instead:

redis-cli --memkeys --memkeys-samples 0

--memkeys-samples 0 tells Redis to measure all elements instead of sampling. This is more accurate but slower.

3.2. Scanning with Java using MEMORY USAGE

redis-cli only returns the biggest key of each type. To list every key exceeding a threshold, you can scan it yourself. First, add the dependency:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-redis</artifactId>
</dependency>

Next, write a component to scan the keyspace:

@Component
public class BigKeyScanner {

    private final StringRedisTemplate redis;

    public BigKeyScanner(StringRedisTemplate redis) {
        this.redis = redis;
    }

    public Map<String, Long> findKeysLargerThan(long thresholdBytes) {
        Map<String, Long> result = new LinkedHashMap<>();
        ScanOptions options = ScanOptions.scanOptions().match("*").count(500).build();

        try (Cursor<String> cursor = redis.scan(options)) {
            while (cursor.hasNext()) {
                String key = cursor.next();
                Long bytes = redis.execute((RedisCallback<Long>) conn ->
                        (Long) conn.execute("MEMORY",
                                "USAGE".getBytes(), key.getBytes(), "SAMPLES".getBytes(), "0".getBytes()));
                if (bytes != null && bytes > thresholdBytes) {
                    result.put(key, bytes);
                }
            }
        }
        return result;
    }
}

Key points in this code:

• redis.scan(options) scans the keyspace in batches of 500 keys. The main thread isn’t locked for long.
• Cursor must be closed, so we use try-with-resources.
• MEMORY USAGE key SAMPLES 0 returns the number of bytes the key occupies, including overhead. Spring Data Redis doesn’t have a dedicated API for this, so we call it via conn.execute.

When you call findKeysLargerThan(1_000_000), you might get:

{cart:10023=3145812, product:detail:88=1048648, leaderboard:global=89231560}

Tip: Run the scanner on a replica instead of the master, and do it during low-traffic hours. Another approach is to analyze the RDB file offline using RDB parsing tools. This has zero impact on the running server.

4. Detecting hot keys

4.1. Using redis-cli --hotkeys

Redis only tracks access frequency when you enable an LFU-style eviction policy:

# redis.conf
maxmemory-policy allkeys-lfu

After enabling LFU, run:

redis-cli --hotkeys

[00.00%] Hot key '"config:homepage"' found so far with counter 255
[37.21%] Hot key '"product:stock:5001"' found so far with counter 201

-------- summary -------
hot key found with counter: 255	keyname: "config:homepage"
hot key found with counter: 201	keyname: "product:stock:5001"

The LFU counter is a logarithmic value, with a maximum of 255. It’s not the actual request count. You should only use it to compare relative hotness between keys.

If you suspect a specific key, check it directly:

redis-cli OBJECT FREQ config:homepage

(integer) 255

Tip: Don’t use MONITOR on production to find hot keys. MONITOR sends every command to the client and can cut throughput in half or more.

4.2. Counting accesses from the application

If you can’t change maxmemory-policy, you can count reads in your application. The code below tracks how many times each key is accessed within a time window:

@Component
public class HotKeyDetector {

    private final ConcurrentHashMap<String, LongAdder> counters = new ConcurrentHashMap<>();

    public void record(String key) {
        counters.computeIfAbsent(key, k -> new LongAdder()).increment();
    }

    public List<Map.Entry<String, Long>> topAndReset(int n) {
        Map<String, Long> snapshot = new HashMap<>();
        counters.forEach((k, v) -> snapshot.put(k, v.sumThenReset()));
        counters.clear();
        return snapshot.entrySet().stream()
                .sorted(Map.Entry.<String, Long>comparingByValue().reversed())
                .limit(n)
                .toList();
    }
}

You call record(key) whenever you read from Redis. Then use a @Scheduled job to call topAndReset(10) every 10 seconds and log or export the results. LongAdder reduces contention when many threads increment the same counter, making it better than AtomicLong for this use case.

5. Handling big keys

5.1. Key sharding

Suppose you store view counts for all products in a single hash called product:views. This hash will grow to millions of fields. The fix is to split it into many smaller hashes:

public final class ShardRouter {

    private ShardRouter() {}

    public static String shardKey(String baseKey, String field, int shards) {
        int shard = Math.floorMod(field.hashCode(), shards);
        return baseKey + ":" + shard;
    }
}

@Service
public class ProductViewService {

    private static final int SHARDS = 64;
    private final StringRedisTemplate redis;

    public ProductViewService(StringRedisTemplate redis) {
        this.redis = redis;
    }

    public long increment(String productId) {
        String key = ShardRouter.shardKey("product:views", productId, SHARDS);
        return redis.opsForHash().increment(key, productId, 1);
    }

    public long get(String productId) {
        String key = ShardRouter.shardKey("product:views", productId, SHARDS);
        Object value = redis.opsForHash().get(key, productId);
        return value == null ? 0 : Long.parseLong(value.toString());
    }
}

How it works:

• Math.floorMod always returns a non-negative number, even if hashCode() is negative. This is where many people use % and get burned by negative indices.
• Product "P-1001" always falls into the same shard, say product:views:17. This ensures reads and writes are consistent.
• Each hash now contains only about 1/64th of the fields. In Redis Cluster, these 64 keys live in different slots, so load is distributed across many nodes.

Tip: Don’t add a hash tag like product:views:{all}:17 if your goal is to spread load. Hash tags force all shards into the same slot. Each key gets smaller, but load still concentrates on one node.

For big String keys, like a 1 MB JSON product detail, split the rarely-changing part from the frequently-changing part into separate keys. You can also compress the value or cache only the fields you actually need.

5.2. Safely deleting big keys

DEL frees memory synchronously. Deleting a ZSet with a million members can block Redis for hundreds of milliseconds. Use UNLINK instead:

public void removeLeaderboard() {
    redis.unlink("leaderboard:global");
}

UNLINK removes the key from the keyspace instantly. Memory reclamation is pushed to a background thread. You should also enable lazy free in your config so Redis’s implicit delete operations don’t block the main thread either:

# redis.conf (Redis 6+)
lazyfree-lazy-user-del yes
lazyfree-lazy-expire yes
lazyfree-lazy-eviction yes
lazyfree-lazy-server-del yes

With lazyfree-lazy-user-del yes, even DEL commands act like UNLINK. This protects you from old code that still calls delete().

If you’re stuck on an older Redis version or want tighter control over load, delete in batches:

public void deleteHashGradually(String key) {
    ScanOptions options = ScanOptions.scanOptions().count(500).build();
    try (Cursor<Map.Entry<Object, Object>> cursor = redis.opsForHash().scan(key, options)) {
        List<Object> batch = new ArrayList<>(500);
        while (cursor.hasNext()) {
            batch.add(cursor.next().getKey());
            if (batch.size() == 500) {
                redis.opsForHash().delete(key, batch.toArray());
                batch.clear();
            }
        }
        if (!batch.isEmpty()) {
            redis.opsForHash().delete(key, batch.toArray());
        }
    }
    redis.unlink(key);
}

Each HDEL removes only 500 fields, so blocking time per command is minimal. Redis allows modifying a hash while HSCAN is running. Remaining fields are still returned. The final unlink cleans up any stragglers.

6. Handling hot keys

6.1. Local cache before Redis

The most effective approach is to prevent most requests from reaching Redis at all. Add Caffeine:

<dependency>
    <groupId>com.github.ben-manes.caffeine</groupId>
    <artifactId>caffeine</artifactId>
</dependency>

Then place a short-lived local cache in front of Redis:

@Service
public class HomepageConfigService {

    private final StringRedisTemplate redis;
    private final LoadingCache<String, String> localCache;

    public HomepageConfigService(StringRedisTemplate redis) {
        this.redis = redis;
        this.localCache = Caffeine.newBuilder()
                .maximumSize(1_000)
                .expireAfterWrite(Duration.ofSeconds(2))
                .build(key -> redis.opsForValue().get(key));
    }

    public String getConfig() {
        return localCache.get("config:homepage");
    }
}

Suppose you have 20 application instances. Each instance calls Redis at most roughly once every 2 seconds for this key, so Redis receives about 10 requests/second instead of tens of thousands. LoadingCache also coalesces concurrent loads for the same key, preventing thundering herd when the cache expires.

The trade-off is that data can be stale by up to 2 seconds. For homepage config or product categories, this latency is usually acceptable.

6.2. Key replication

When you need fresher data, or you’re in a Redis Cluster with one overloaded node, you can write the value to multiple copies and read randomly from one:

@Service
public class FlashSaleService {

    private static final int REPLICAS = 8;
    private final StringRedisTemplate redis;

    public FlashSaleService(StringRedisTemplate redis) {
        this.redis = redis;
    }

    public void publish(String saleId, String json) {
        for (int i = 0; i < REPLICAS; i++) {
            redis.opsForValue().set("flashsale:" + saleId + ":r" + i, json, Duration.ofMinutes(10));
        }
    }

    public String read(String saleId) {
        int i = ThreadLocalRandom.current().nextInt(REPLICAS);
        return redis.opsForValue().get("flashsale:" + saleId + ":r" + i);
    }
}

The 8 keys flashsale:42:r0 through flashsale:42:r7 typically fall into different slots. So read load is divided among multiple nodes. This approach works well for read-heavy, write-rarely data.

Tip: Writing to multiple replicas is not atomic. For a brief moment, replicas can diverge. Don’t use this for data requiring strong consistency, like account balances. For a hot inventory counter, consider sharding the inventory into independent “buckets” instead. Each bucket holds a portion of the quantity rather than duplicating the same value.

Besides these two approaches, you can read from replicas. With Lettuce, configure ReadFrom.REPLICA_PREFERRED to spread read load across a master–replica setup.

7. Testing key sharding logic

The sharding routing logic is the easiest place to make mistakes, so write tests for it:

class ShardRouterTest {

    @Test
    void sameFieldAlwaysMapsToSameShard() {
        String first = ShardRouter.shardKey("product:views", "P-1001", 64);
        String second = ShardRouter.shardKey("product:views", "P-1001", 64);

        assertThat(first).isEqualTo(second).startsWith("product:views:");
    }

    @Test
    void shardIndexIsNeverNegative() {
        // "polygenelubricants" has hashCode() == Integer.MIN_VALUE
        String key = ShardRouter.shardKey("product:views", "polygenelubricants", 64);
        int index = Integer.parseInt(key.substring(key.lastIndexOf(':') + 1));

        assertThat(index).isBetween(0, 63);
    }

    @Test
    void fieldsAreSpreadAcrossShards() {
        Map<String, Long> distribution = IntStream.range(0, 100_000)
                .mapToObj(i -> ShardRouter.shardKey("product:views", "P-" + i, 64))
                .collect(Collectors.groupingBy(k -> k, Collectors.counting()));

        assertThat(distribution).hasSize(64);
        assertThat(distribution.values()).allSatisfy(count ->
                assertThat(count).isBetween(1_000L, 2_200L));
    }
}

The second test checks the edge case with Integer.MIN_VALUE. If you used Math.abs(hash) % n, this test would fail because Math.abs(Integer.MIN_VALUE) is still negative. The third test confirms 100,000 fields are distributed reasonably evenly: about 1,560 fields per shard on average.

8. Common mistakes and when not to use these techniques

• Running KEYS * to find big keys: this command blocks Redis until it finishes scanning. Always use SCAN.
• Sharding but still needing to read everything: if your business logic frequently HGETALL all fields, sharding just turns one slow command into 64 commands. Reconsider the business logic first, like pagination or pre-computing aggregations.
• Changing the shard count while running: changing SHARDS from 64 to 128 causes most fields to move to different locations. You need a migration plan, or choose a large enough shard count upfront.
• Local cache for rapidly changing data: inventory levels, auction prices… don’t work with a few-second TTL. If you must use it, combine it with Pub/Sub or Redis 6+ client-side caching (CLIENT TRACKING) to invalidate the cache.
• Optimizing too early: a hash with 2,000 fields isn’t a problem yet. Measure first with --bigkeys, --hotkeys, and slowlog (SLOWLOG GET) before deciding.

9. Conclusion

In this article, you’ve seen why hot keys and big keys are dangerous for Redis, which processes commands on a single thread. To detect them, you can use redis-cli --bigkeys, --memkeys, --hotkeys, MEMORY USAGE, OBJECT FREQ, or count from the application. To handle big keys, shard them using Math.floorMod, delete with UNLINK, and enable lazy free. To reduce load from hot keys, use Caffeine for local caching or replicate keys. Include periodic scanning in your operations plan—early detection is always cheaper than firefighting when Redis gets overwhelmed.