Load Balancer: Algorithms and When to Use Them

1. Overview

When your system has multiple instances on the backend, the question is always: which instance should a new request go to? This article explains common load balancing algorithms, how they work with simulated Java code, and the criteria for choosing the right algorithm for each workload type. By the end, you’ll know when Round Robin is enough, and when you need Least Connections or Consistent Hashing.

2. What is a Load Balancer?

A load balancer (LB) is a component that sits between clients and a pool of servers (backend pool) and distributes requests such that:

• No single server gets overloaded while others remain idle.
• The system can handle one backend going down (failover).
• You can scale horizontally (add/remove instances) without clients knowing.

An LB can be software (Nginx, HAProxy, Envoy), hardware (F5), or a managed service (AWS ALB/NLB, GCP Load Balancer). In terms of where it operates, LBs fall into two main categories:

• Layer 4 (transport layer): routes based on IP/port, doesn’t read HTTP content. Fast, low overhead.
• Layer 7 (application layer): can read headers, paths, cookies… so it routes more intelligently (for example, routing by path like /api/orders) but consumes more CPU.

Note: Choosing the wrong layer is a common design mistake. If you need to route by URL path or sticky sessions based on cookies, Layer 4 can’t do it — you must use Layer 7.

3. Common Load Balancing Algorithms

3.1. Round Robin

Requests are sent to each server in a fixed order, cycling back to the beginning when the list is exhausted.

public class RoundRobinBalancer {

    private final List<String> servers;
    private final AtomicInteger index = new AtomicInteger(0);

    public RoundRobinBalancer(List<String> servers) {
        this.servers = servers;
    }

    public String nextServer() {
        int i = index.getAndIncrement() % servers.size();
        return servers.get(Math.abs(i));
    }
}

getAndIncrement increments the counter safely across multiple threads, % servers.size() keeps the index within bounds. With 3 servers [A, B, C], consecutive calls to nextServer() produce:

A, B, C, A, B, C, ...

Round Robin is simple and doesn’t need to know server state. Downside: if requests have varying processing times (one request processes in 1ms, another in 500ms), servers processing heavy requests can become overloaded over time.

3.2. Weighted Round Robin

Each server has a “weight” reflecting its processing capacity (CPU, RAM). More powerful servers receive more requests.

public class WeightedRoundRobinBalancer {

    private record WeightedServer(String name, int weight) {}

    private final List<WeightedServer> servers;
    private int currentIndex = -1;
    private int currentWeight = 0;

    public WeightedRoundRobinBalancer(Map<String, Integer> weightedServers) {
        this.servers = weightedServers.entrySet().stream()
            .map(e -> new WeightedServer(e.getKey(), e.getValue()))
            .toList();
    }

    public synchronized String nextServer() {
        int maxWeight = servers.stream().mapToInt(WeightedServer::weight).max().orElse(1);
        int gcdWeight = servers.stream().mapToInt(WeightedServer::weight)
            .reduce((a, b) -> java.math.BigInteger.valueOf(a).gcd(java.math.BigInteger.valueOf(b)).intValue())
            .orElse(1);

        while (true) {
            currentIndex = (currentIndex + 1) % servers.size();
            if (currentIndex == 0) {
                currentWeight -= gcdWeight;
                if (currentWeight <= 0) {
                    currentWeight = maxWeight;
                }
            }
            if (servers.get(currentIndex).weight() >= currentWeight) {
                return servers.get(currentIndex).name();
            }
        }
    }
}

This is the ‘smooth weighted round robin’ algorithm used in Nginx: servers with higher weights are chosen more frequently but still distributed evenly without clustering. With the configuration {A: 5, B: 1, C: 1}, the selection sequence is approximately:

A, A, B, A, C, A, A

3.3. Least Connections

Selects the server with the fewest active connections. Suitable when request processing times vary widely.

public class LeastConnectionsBalancer {

    private final Map<String, AtomicInteger> activeConnections = new ConcurrentHashMap<>();

    public LeastConnectionsBalancer(List<String> servers) {
        servers.forEach(s -> activeConnections.put(s, new AtomicInteger(0)));
    }

    public String nextServer() {
        return activeConnections.entrySet().stream()
            .min(Comparator.comparingInt(e -> e.getValue().get()))
            .map(Map.Entry::getKey)
            .orElseThrow();
    }

    public void onRequestStart(String server) {
        activeConnections.get(server).incrementAndGet();
    }

    public void onRequestEnd(String server) {
        activeConnections.get(server).decrementAndGet();
    }
}

nextServer() scans the map and gets the server with the fewest connections. onRequestStart/onRequestEnd must be called in pairs (for example, in try/finally) to keep metrics accurate.

Note: Least Connections requires the LB to track state (stateful) — adding synchronization overhead if you have multiple LB instances running in parallel. Round Robin is completely stateless.

3.4. Least Response Time

A variant of Least Connections that selects the server with the lowest (number of connections × average response time). More complex but more accurately reflects actual load because it factors in processing latency.

public record ServerStats(String name, int activeConnections, double avgResponseTimeMs) {
    double score() {
        return activeConnections * avgResponseTimeMs;
    }
}

public String pickFastest(List<ServerStats> stats) {
    return stats.stream()
        .min(Comparator.comparingDouble(ServerStats::score))
        .map(ServerStats::name)
        .orElseThrow();
}

avgResponseTimeMs is typically calculated using exponential moving average, updated after each response is returned. This algorithm works well for systems with a mix of fast APIs (cache lookups) and slow APIs (reports, heavy computations).

3.5. IP Hash / Consistent Hashing

Used when you need sticky sessions — the same client always routes to the same server (for example, when servers keep sessions in-memory or maintain local caches).

public class ConsistentHashBalancer {

    private final SortedMap<Long, String> ring = new TreeMap<>();
    private static final int VIRTUAL_NODES = 100;

    public ConsistentHashBalancer(List<String> servers) {
        for (String server : servers) {
            for (int i = 0; i < VIRTUAL_NODES; i++) {
                ring.put(hash(server + "#" + i), server);
            }
        }
    }

    public String serverFor(String clientKey) {
        long hash = hash(clientKey);
        SortedMap<Long, String> tail = ring.tailMap(hash);
        Long nodeKey = tail.isEmpty() ? ring.firstKey() : tail.firstKey();
        return ring.get(nodeKey);
    }

    private long hash(String input) {
        return input.hashCode() & 0xFFFFFFFFL;
    }
}

Each server is spread across multiple ‘virtual nodes’ on a hash ring (ring) for more even distribution. serverFor finds the nearest point on the ring in clockwise order corresponding to the client’s key (usually IP or user ID). Major advantage: when adding/removing a server, only a small fraction of keys are remapped — unlike traditional hashing (hash(key) % n) which remaps everything.

Quick test to verify the same key always maps to the same server:

@Test
void sameClientKey_alwaysRoutesToSameServer() {
    ConsistentHashBalancer balancer =
        new ConsistentHashBalancer(List.of("server-1", "server-2", "server-3"));

    String first = balancer.serverFor("user-42");
    String second = balancer.serverFor("user-42");

    assertThat(first).isEqualTo(second);
}

3.6. Random and Power of Two Choices

Random simply picks a server at random — nearly as effective as Round Robin with a large number of requests but requires no state maintenance. Power of Two Choices (P2C) improves on this: randomly select 2 servers, then choose the one with fewer connections between the two. This algorithm is proven to significantly reduce ‘tail latency’ compared to pure random selection, while not requiring monitoring of the entire cluster like Least Connections does.

public String pickWithPowerOfTwo(List<String> servers, Map<String, Integer> loadMap, Random random) {
    String candidate1 = servers.get(random.nextInt(servers.size()));
    String candidate2 = servers.get(random.nextInt(servers.size()));
    return loadMap.get(candidate1) <= loadMap.get(candidate2) ? candidate1 : candidate2;
}

4. Health Checks and Failover

Server selection algorithms only matter if the LB knows which servers are ‘alive’. Health checks typically come in two types:

• Active check: The LB itself periodically sends requests to an endpoint like /health.
• Passive check: The LB marks a server ‘down’ when it consecutively receives errors/timeouts from actual requests.

Example of active health check configuration in Nginx:

upstream backend_pool {
  server 10.0.0.1:8080 max_fails=3 fail_timeout=30s;
  server 10.0.0.2:8080 max_fails=3 fail_timeout=30s;
}

max_fails=3 means that after 3 consecutive failures, the server is removed from the pool for fail_timeout seconds before being retried. Without health checks, the LB keeps sending requests to dead servers until the client times out — a terrible user experience.

5. Algorithm Comparison: When to Use Each

Algorithm	When to Use	When to Avoid
Round Robin	Homogeneous servers, similar request weights (simple REST APIs)	Servers with different specs, varying request weights
Weighted Round Robin	Cluster with servers of different capacities (gradual migration to new instances)	Need to react to real-time load changes
Least Connections	Requests with highly variable processing times (file uploads, long-polling)	Many LB instances, high state synchronization cost
Least Response Time	Mix of fast/slow APIs, need to optimize overall latency	Small systems, insufficient data for stable response time measurements
IP Hash / Consistent Hashing	Need sticky sessions, cache locality, client-based sharding	Frequent server scaling up/down at high speed (though remapping impact is reduced, there’s still overhead)
Random / P2C	Very large systems, want to avoid centralized state tracking overhead	Need absolute even distribution in short timeframes

6. Common Mistakes When Designing Load Balancing

• Using Round Robin for non-uniform workloads: a few servers receive all the heavy requests while others sit idle — causing silent load imbalance that’s hard to spot in overall monitoring (average CPU still looks ‘normal’).
• Sticky sessions via IP Hash when clients are behind shared NAT/proxies: multiple actual clients get bundled into one IP, causing one server to receive abnormally high load. Use cookie-based sticky sessions at Layer 7 instead of IP Hash in this case.
• No health checks or wrong health check endpoint: /health returns 200 OK but doesn’t verify database connectivity — the server looks ‘alive’ to the LB but can’t actually process requests.
• Skipping connection draining during scale-in: immediately terminating servers drops in-flight requests. You need to wait for current connections to complete (graceful shutdown) before removing the server from the pool.
• Single point of failure in the LB itself: if there’s only one LB instance, it becomes the single point of failure for the entire system. You need at least 2 LB instances with DNS failover or a virtual IP (keepalived).

7. Conclusion

A load balancer is not just ‘something that distributes requests evenly’ — choosing the right algorithm depends on workload characteristics: uniform or not, whether you need sticky sessions, and whether your system can afford centralized state tracking. Round Robin and Weighted Round Robin suit most simple cases; Least Connections and Least Response Time handle variable workloads well; Consistent Hashing robustly solves the sticky session and sharding problem as the cluster changes. More than the algorithm itself, proper health checks and failover are what truly determine your system’s reliability.