Pattern card: load balancing

The full lessons, with worked examples: Load balancing and scaling out and Health checks and redundancy.

What it is

A load balancer sits in front of a group of identical servers, takes every incoming request and hands it to one server that has room.

• It enables scaling out: add ordinary machines instead of buying one bigger machine (horizontal vs vertical scaling).
• It removes a single point of failure: a dead server is noticed by health checks and skipped.
• It hides the fleet: users see one address; servers can be added, drained and replaced behind it.
• It needs stateless servers: any request may land on any server, so carts, logins and sessions live in a shared database, cache or signed token.

When to use it

Use it when:

• One server can't carry peak traffic. Work it out: peak requests a second ÷ safe requests per server.
• Losing one machine must not take the app down. Even two servers behind a load balancer beat one big one.
• Traffic swings a lot: add servers for the evening rush or a match, remove them after.
• You deploy often: drain and update servers one at a time with no downtime.

Don't reach for more servers when:

• The bottleneck is elsewhere. If app servers are mostly waiting on the database, more of them just send more queries to it. Fix the data layer (cache, indexes, replicas).
• Servers hold state only they know. Make them stateless first, or every restart loses data.
• Traffic is tiny. One server plus a standby may be all you need; a managed load balancer is still cheap insurance.

How it works

1. Pick a server for each request (see Variants for the rules).

2. Check health. Every few seconds the load balancer asks each server, say GET /health. Three settings:

• Interval: how often, say every 5 seconds.
• Timeout: how long to wait for a reply, say 2 seconds.
• Thresholds: failures in a row to remove (say 3), passes in a row to bring back (say 2).

1. 1

   Load balancerServer A

   GET /health, every 5 s

2. 2

   Server ALoad balancer

   No reply, 3 times in a row

3. 3

   UsersLoad balancer

   Next request

4. 4

   Load balancerServer B

   A skipped, sent to B

3. Watch real traffic too. Passive checks rest a server after, say, 5 errors in a row on real requests, without waiting for the next probe.

4. Ask the right question.

• Liveness: "are you stuck?" Failing means restart. Keep it shallow.
• Readiness: "can you take real requests now?" Failing means stop sending traffic, but don't restart. Check only what the server itself truly needs, such as its own database connection.

5. Remove and add gently.

• Draining: a server being removed gets no new requests but finishes the ones it has.
• Slow start: a new or returning server gets a small share, ramping up over 30 to 60 seconds, so its cold caches aren't flooded.

6. Make the balancer itself redundant. Run at least a pair that watch each other with a heartbeat; the standby takes over the same address within seconds (failover). Managed cloud load balancers do this across zones for you.

7. Layer them. DNS sends users to a nearby region; a layer 4 balancer spreads connections; a layer 7 balancer routes requests by path.

Variants

Picking rules:

• Round robin. How: take turns. Good for: identical servers and requests of similar cost. Cost: a slow server still gets its full share, and its queue grows.
• Weighted round robin. How: a server twice as big gets twice the turns. Good for: mixed machine sizes during a migration. Cost: weights are set by hand and go stale.
• Least connections. How: send to the server with the fewest requests in progress. Good for: varied request times (a 300 ms search next to a 5 ms score check); slow servers naturally get less. Cost: needs live counts; a just-restarted server with zero connections can be flooded without slow start.
• Power of two choices. How: pick two servers at random, send to the less busy. Good for: large fleets and many balancers; close to the best spread with very little bookkeeping. Cost: slightly less even than a perfect global view, which nobody has anyway.
• Hash on a key. How: the same user or cart always goes to the same server, often with consistent hashing. Good for: servers holding a warm local cache per key. Cost: hot keys overload one server; it reintroduces state that dies with the server.
• Sticky sessions. How: pin a user to one server with a cookie. Good for: legacy apps that keep sessions in memory. Cost: uneven load, and users lose their session when that server dies. Prefer a shared store.

Where it sits:

• Layer 4. How: spreads raw connections by address and port without reading them, like sorting post by pin code. Good for: very high volume, any protocol. Cost: blind to request cost; one long connection stays on one server.
• Layer 7. How: reads each web request; routes by path or header (/payments to payment servers), retries, does TLS termination. Good for: per-request balancing, canary releases. Cost: more work per request.
• DNS across regions. How: the app's name returns an address in a nearby region. Good for: geography and losing a whole region. Cost: changes take minutes because answers are cached.
• Client-side balancing. How: each service picks a server itself from a list. Good for: internal service-to-service calls. Cost: every client needs the logic and an up-to-date list.

Numbers to know

• Server ceiling = slots ÷ time per request. 8 requests at once × (1,000 ÷ 40 ms) = about 200 requests a second.
• Plan for 60–70% busy at peak. Near 100% utilization, waits climb sharply and requests time out.
• Detection time ≈ interval × failures needed: 5 s × 3 = about 15 seconds. Passive checks: about a second.
• Requests lost while detecting ≈ server's rate × detection time: 860 a second × 15 s = about 13,000, which client retries recover.
• Slow start: 30 to 60 seconds of ramp.
• Servers fail: at about one failure per server every three years, 1,000 servers lose about one a day.
• Chains multiply: five parts at 99.9% each = about 99.5% together, about 44 hours of trouble a year instead of about 9.
• Spares multiply in your favour: two independent servers at 99.9% are both down about one time in a million, about 30 seconds a year.
• N+1: if peak needs the work of 2.4 servers, 3 servers sit at 80% and hit 120% when one dies; 4 servers sit at 60% and reach 80% after a failure. Many teams run N+2 because servers are also out for updates.
• Zone sizing: peak needs 215 servers across 3 zones; to survive losing a zone, each of the other two needs about 108, so run about 325.
• Server start-up: a few minutes, so autoscaling lags sudden spikes. Pre-warm before known peaks.

Pitfalls

• Health check that always says OK. The process runs but can't reach its database, and still gets a tenth of traffic. Fix: readiness checks that prove the server can do its job, plus watching real error rates.
• Health check that depends on a shared service. A brief database blip fails every check and the balancer empties the whole fleet. Fix: check only local, essential things, and cap how much of the pool can be removed (say 50%).
• One slow server under round robin. A failing disk makes it 10 times slower and drags down the p99 for everyone. Fix: least connections or two choices, plus outlier ejection.
• Flapping on return. A restarted server gets a flood, slows, fails checks and is thrown out again. Fix: slow start, and report ready only after warming up.
• Single load balancer. Hundreds of healthy servers, one dead balancer, app down. Fix: pairs or a managed, multi-zone service.
• Long-lived connections. WebSockets opened before scale-out stay on the old servers; new ones sit idle. Fix: ask clients to reconnect after a random 10 to 30 minutes; balance per request at layer 7.
• Running hot to save money. 20 servers at 90%: lose one and the rest tip past 95%. Fix: headroom for the loss of a server or a whole zone.
• Spikes faster than start-up. Traffic goes from 5,000 to 80,000 a second in two minutes; failures trigger retries, adding more load. Fix: scheduled scale-up, load shedding with a quick "try again", and a retry budget.
• Copies that fail together. Same rack, same power, same buggy release. Fix: spread across availability zones and release in stages.

In an interview

The cue is "one server won't handle this" or "what if this dies?" Don't just draw a box labelled LB.

• Capacity sum: peak ÷ safe rate per server, plus headroom and a spare zone.
• Stateless servers behind the balancer, shared data in a cache or database.
• Picking rule: least connections (or two choices) when request times vary.
• Health: interval, thresholds, passive checks, liveness vs readiness, draining, slow start.
• No single point of failure: paired balancers, several zones, DNS across regions.
• Scaling: autoscaling plus pre-warming for known peaks.

Follow-ups and short answers:

• "What if the load balancer fails?" It runs as a pair or a managed service across zones; the standby takes the same address within seconds, and DNS can steer users to another region.
• "Round robin or least connections?" Round robin is fine for identical servers and similar requests. When request times vary, least connections or two random choices keep queues even.
• "Layer 4 or layer 7?" Layer 4 is faster and protocol-agnostic; layer 7 reads requests, so it can route by path, retry and balance per request, which matters for long connections.
• "How quickly do you notice a dead server?" About 15 seconds with 5-second checks and 3 failures; passive checks cut it to about a second.
• "How many servers?" Enough for peak with one server, or one zone, gone: 2.4 servers of work means running 4.