Pattern card: replication

The full lessons, with worked examples: Read replicas and replication lag and Quorums. Related cards: caching, sharding.

What it is

Replication keeps the same data on several machines. The common form has one primary database that takes every write, and several read replicas that copy each change and answer reads.

• Everyday version: the school office keeps the master timetable and pins photocopies on every floor. Changes happen only on the master; students read the copies.
• What it buys: more reading power, copies close to users, and a spare to promote when the primary dies.
• What it costs: copies are a moment behind (replication lag), every write is applied on every copy, and you must decide which reads can live with that.
• What it does not do: spread writes. Every replica applies all of them. Too many writes for one machine is a job for sharding.

When to use it

Use it when:

• Reads far outnumber writes. 100 reads per write is common; feeds and catalogues reach 1,000 to 1. That is most consumer apps.
• Losing the database must not mean losing the app. Even a quiet app runs one replica as a spare for failover.
• Users are spread out. A Delhi user reading from a Delhi replica waits about 5 ms instead of about 25 ms to Mumbai.
• Reports would slow the app. A separate replica for analytics keeps heavy queries away from shoppers.

Don't reach for it when:

• Writes are the bottleneck. More replicas add copying work to the primary and fix nothing. Batch writes, queue them, or shard.
• You want a backup. Replicas copy mistakes too. A dropped table is dropped everywhere within milliseconds. Keep real backups.
• Every read must be exact and current. If almost all reads are about money or stock, replicas can't take much of the load anyway.

How it works

1. The primary logs every change. Each write goes into a numbered write-ahead log: "change 48,213,907: set user 1182's name". The same log lets the database recover after a crash.
2. Replicas follow the log. Each replica streams the log and applies changes in the same order. Same start, same changes, same order: same data. A replica only remembers the last number it applied, so after a disconnect it asks for "everything after 48,213,907".
3. App servers route by kind of read. Writes go to the primary. Ordinary reads go to replicas. Reads that decide something (a balance before a payment, the last seat) go to the primary.
4. Lag is measured, not assumed. Each replica reports how far behind it is. Any replica beyond a limit, say 5 seconds, is taken out of rotation until it catches up.
5. Failover promotes a replica. When the primary dies, the most up-to-date replica becomes the new primary, usually within a minute. The old primary is fenced so it can't come back taking writes, and apps reconnect through a fixed address that moves with the role.

Reading your own writes. The usual trick when a user reloads right after saving:

1. 1

   PhoneApp server

   Save new name

2. 2

   App serverPrimary

   Write, log position 907

3. 3

   PhoneApp server

   Reload profile, 0.3 s later

4. 4

   App serverPrimary

   Saved under 10 s ago: read primary

5. 5

   App serverReplica

   Later reads: any replica past 907

• Time window: for about 10 seconds after a save, that user's reads go to the primary. Simple, but fragile when lag spikes past the window.
• Log position: return the save's log number with the write and only read from replicas that have reached it. More precise.
• Sticky replica: keep each user on one replica for the visit, so their view never goes backwards. That promise is monotonic reads.
• Show it locally: the app shows the new post straight from what the user typed while the save completes.

Variants

• Asynchronous leader-follower. How: the primary confirms at once and ships changes later (async). Good for: fast writes that don't depend on replica health. Cost: if the primary dies, the last few unsent writes are lost; replicas are only eventually consistent.
• Semi-synchronous (one sync replica). How: wait for one nearby replica to confirm, let the rest copy in their own time. Good for: the usual middle path; every confirmed write exists on two machines. Cost: each write adds a round trip, about 1–2 ms within a city, and a slow sync replica slows every save. Some systems quietly drop back to async on timeout, so alert on it.
• Fully synchronous. How: synchronous replication to every replica. Good for: small, precious data. Cost: the slowest or dead replica blocks all writes.
• Leaderless quorum. How: keep N copies (the replication factor, often 3); a write counts when W confirm, a read asks R and keeps the newest version. If R + W is more than N, every read overlaps the latest write on at least one copy. Good for: no single write point, and speed, because you wait for the W fastest copies, not the slowest. Cost: versions, read repair, background anti-entropy, and conflicts between simultaneous writes.
• Multi-leader (active-active). How: a primary in each region takes local writes and they copy to each other. Good for: fast writes in several regions, and surviving a whole region going down. Cost: two people can change the same record at once, so you need a conflict rule such as last write wins, which silently drops one change.
• Physical versus logical copying. Physical ships the raw log and gives byte-identical replicas, the default for spares. Logical ships row changes, which allows partial copies and other versions, and underpins change data capture.
• Special replicas. An analytics replica that serves no users; a delayed replica kept an hour behind as a quick undo; cascading replicas that copy from another replica to spare the primary, at the cost of extra lag.

Numbers to know

• Lag: a few ms to a few tens of ms in one city when healthy. Another city adds the travel time, about 25 ms Delhi to Mumbai.
• Sizing sum: 40,000 reads and 400 writes a second, one server comfortable at say 10,000 reads. Reads need four servers' worth, so run five replicas at about 8,000 each. If one fails, four carry about 10,000 each. The primary still takes all 400 writes, and so does every replica.
• Lag growth sum: an import writes 2,000 changes a second; a busy replica applies 1,500. It falls 500 a second behind, so after a minute it is 30,000 changes back, about 20 seconds of lag. When writes drop to 400 a second it gains about 1,100 a second and clears the backlog in roughly half a minute.
• Failover: usually under a minute. With async copying, expect to lose the last second or two of confirmed writes.
• Quorums: N = 3, W = 2, R = 2 survives 1 lost copy; N = 5, W = 3, R = 3 survives 2. Writes survive N − W failures, reads N − R.
• Storage: 2 TB of data is 6 TB with three copies, 10 TB with five. At 5,000 writes a second, three copies means 15,000 writes across the machines.
• Tail speed: copies answering in 2, 3 and 40 ms; waiting for all takes 40 ms, waiting for 2 of 3 takes 3 ms.

Pitfalls

• Deciding from a replica. A wallet checks a lagging replica, misses a payment made a second ago, and approves a second ₹2,000. Reads that decide go to the primary, ideally in the same transaction as the change.
• Big jobs on the primary. Updating 50 lakh rows at once pushes lag to minutes. Batch in chunks of say 5,000 with pauses, slow down when lag rises, run at night.
• A silently stale replica. A replica with a slow disk answers fast with three-hour-old data. Route by lag, not just "is it up"; rebuild it if it can't catch up.
• Reports on a serving replica. A 40-minute finance query starves shoppers on that copy. Give analytics its own replica.
• Flickering comments. Alternate refreshes hit an up-to-date and a lagging replica, so a comment appears and vanishes. Sticky replicas plus read-your-writes.
• Promoting without fencing. Two machines both believe they are primary, a split brain, and take conflicting writes. Fence the old one first.
• Testing with zero lag. On a laptop the "replica" is the primary. Add about 2 seconds of artificial lag in testing and every read-after-write bug shows up.
• Copies that fail together. Three copies in one rack share power and a switch. Spread them across availability zones; the extra millisecond or two is cheap.

In an interview

Bring replication in whenever reads dominate: feeds, catalogues, profiles, URL shorteners, results pages.

• Say the ratio and the count. "About 100 reads per write, one server handles about 10,000 reads, so N replicas plus one spare."
• Say what doesn't scale. Writes stay on one primary; more replicas don't help them.
• Name the lag and its fixes. Read-your-writes for the author, sticky replicas for everyone, primary for money and stock.
• Pick sync or async on purpose. One sync replica nearby for durability, the rest async for speed.
• Explain failover. Promote, fence, reconnect through a stable address, and say what can be lost.
• Separate replicas from backups.

Follow-up: "A user doesn't see their own new post." Route their reads to the primary for a few seconds, or to replicas past their write's log position, and show the post locally while it saves.

Follow-up: "The primary dies. What's lost?" With async copying, the last unsent writes. One sync replica nearby makes that zero for confirmed writes.

Follow-up: "Writes are now the bottleneck." Reduce and batch writes, move slow ones to a queue, then shard across several primaries.

Follow-up: "Why R + W > N?" Two groups that together are bigger than N must share a copy, and that shared copy has the latest write. With N = 3, W = 2, R = 1, the one copy you ask may be the one that missed it.

For experienced engineers

R + W > N gives overlap with the latest acknowledged write, not linearizability: concurrent reads can still see old and new values until read repair completes, and "decrement if above zero" updates need consensus, not quorums. For read-your-writes, return the commit log position with the write and route to a replica whose replay position has reached it; time windows break under lag spikes.