Everyday storyTwo bank branches when the line goes down
Your account has ₹6,000. Normally, the Andheri and Bandra branches of your bank see the same balance, because every withdrawal at one is sent down a line to the other within seconds. One afternoon a digging crew cuts that line.
You withdraw ₹5,000 at Andheri. An hour later you walk into Bandra and ask for another ₹5,000. The Bandra cashier has a choice. Say yes, and the branch stays open for business, but your account may end up ₹4,000 in the red with nobody noticing until the line is fixed. Say "sorry, please try later", and the bank stays correct, but it has stopped serving a customer who may well have had the money. It can't do both until the line is back.
Notice that neither choice is "wrong". A bank would rather refuse than hand out money it can't account for. A newspaper stall that keeps a running tally of sales between two shops would rather keep selling and sort the numbers out at night. The right answer depends on what being wrong would cost.
In a real productIRCTC berths versus Instagram likes
IRCTC must never sell the same berth to two people. When it can't be sure, during a Tatkal rush or a glitch, it makes you wait, shows an error, or puts you on a waiting list. That's frustrating, but it's on purpose: two families turning up for one berth on an overnight train would be far worse.
Instagram, on the other hand, can show a like count that's a few behind for a minute, or show your friend a comment a few seconds after you see it. Nobody is harmed. Two apps facing the same trade-off make opposite choices, and even inside one app the choice often differs from screen to screen: a shopping app might be relaxed about reviews and strict about the last item in stock.
The ideaHow it works
Copies, and the links between them
Big apps don't keep your data on one machine. They keep copies, called , on several machines, often in different buildings or cities, so that one broken machine doesn't lose anything and so that reads can be shared out. Usually the copies stay in step: each change is sent from one to the others within milliseconds.
But the links between them can break. A fibre cable is cut, a switch fails, a data centre loses its connection to the rest of the world while its machines keep running happily. That's a : both sides are alive, and they just can't talk to each other.
The tricky part is that partitions aren't rare. Across hundreds of machines and several buildings, some link somewhere is misbehaving most weeks. So "what do we do during a partition?" is not a theoretical question; it's something every app with copies has to answer in advance.
The choice: answer or be right
The CAP trade-off says that during a partition, a system with copies must choose between two things:
- Consistency: every answer reflects the latest change, as if there were only one copy. This is . To keep it, a copy that can't confirm it's up to date must refuse or wait.
- : every request gets an answer from whichever copy it reaches, even if that answer might be out of date.
- 1
YouAndheri branch
Withdraw ₹5,000 (balance ₹6,000)
- 2
Andheri branchBandra branch
Balance is now ₹1,000 (line is down, not delivered)
- 3
YouBandra branch
Withdraw ₹5,000 again
- 4
Bandra branchYou
Sorry, can't confirm your balance right now
Bandra can't know whether Andheri has paid out money in the last hour. If it answers, it might be wrong. If it wants to be sure it's right, it must not answer. There's no third option while the line is down.
People sometimes say "we'll choose consistency and availability, and skip partitions". That's not a real option: you don't get to decide whether the network breaks, only how you behave when it does. So the true question is narrower and more practical: when copies can't talk, which features keep working, and which ones stop?
Choosing per feature
The answer is almost never one setting for the whole app. Ask, for each kind of data: what does a wrong answer cost, and what does a refused answer cost?
- A wallet balance or a payment. A wrong answer means money spent twice. A refused answer means a customer retries in a minute. Choose consistency.
- The last train berth or the last phone in a sale. A wrong answer is an : two people, one item, and a cancellation with a refund and an apology. Choose consistency.
- Your profile or your own post. A slightly old version for others is fine, but you must see your own change, or you'll think it failed and try again. Choose a middle level, described below.
- Likes, view counts, "trending" lists, recommendations. A wrong answer is a number off by a few for a minute. A refused answer is a broken feed. Choose availability.
The pattern is simple: where being wrong is permanent and costly, stay correct and turn people away briefly. Where being wrong is temporary and cheap, stay up.
The everyday version: speed versus agreement
There's a quieter version of the same trade-off that applies every second of every day, even when nothing is broken. Keeping copies in agreement takes time, and that time grows with distance.
Say an app saves a change in Mumbai and keeps copies in Mumbai, Hyderabad and Singapore. A message takes about 1 millisecond to reach a machine in the same building, about 20 to reach Hyderabad and back, and about 60 to reach Singapore and back. If each save waits for every copy to confirm, called , every save takes at least 60 milliseconds. Do ten saves in a row for one checkout and that's more than half a second of pure waiting.
The alternative is to confirm the save as soon as the nearby copy has it and send the change to the far copies afterwards. Saves are fast, but for a short time the far copies are behind, a delay called . Normally that's a fraction of a second. Under heavy load, it can grow to seconds or even minutes.
This everyday half of the trade-off has a name, : if there's a Partition, choose Availability or Consistency; Else, choose Latency (speed) or Consistency. It's a better description of real life than CAP alone, because most of the time the network is fine, and you're still choosing.
A common middle path: wait for one or two copies close by (in other buildings in the same city), so a single fire or power cut can't lose a saved change, and let the far-away copies lag. You pay a millisecond or two per save instead of sixty.
The levels in between
Consistency isn't on or off; it's a dial with several useful settings. From strictest to loosest:
- Strong: everyone sees every change immediately, in the same order. Money, stock counts, seat bookings. Costs the most speed, and is the first thing to stop during a split.
- : things that depend on each other appear in the right order. A reply never shows up before the comment it answers; a "delivered" status never appears before "shipped".
- : you always see your own changes immediately; others catch up shortly. Profiles, posts, settings.
- : once you've seen something, you never see an older version. Your feed never jumps back in time when you refresh.
- : copies agree if you give them a little time and no new changes arrive. Like counts, view counts, search results.
The middle levels are popular because they fix exactly what users notice, at a fraction of the cost of strong consistency. Most people don't care whether a stranger in Delhi sees their new photo now or in two seconds; they care very much that they see it now.
What happens during a split
How does a system even know it's in a partition? Machines send each other a small every second or so. When heartbeats stop arriving for a few seconds, each side assumes the other is unreachable. The trouble is that a slow link and a dead link look the same from the inside: no reply. So every system needs a , a moment after which it stops waiting and acts.
What it does next depends on its choice. Systems that choose consistency usually let only the side with most of the copies carry on accepting changes, because a majority can't exist on both sides at once. The smaller side switches to read-only, or refuses. Systems that choose availability let both sides keep going and merge the two histories once the link returns, using rules like or data types designed to merge cleanly. The next lesson, on quorums, shows exactly how majorities make this work.
The big idea: when copies can't talk you must choose between answering and being right, and even when they can, agreement costs time, so choose per feature: money must be right, likes can be late, and your own changes should always show up for you.
For experienced engineers
CAP's C is linearizability, a single-object, real-time ordering guarantee; its A means every non-failing node answers, which few real systems promise anyway, so "CP versus AP" is a coarse label. Serializability is a different axis (transactions over many objects); strict serializability combines both. PACELC captures the common case: Spanner-style systems pay commit wait or cross-region consensus for external consistency, while Dynamo-style stores let you tune per request (quorum sizes, LOCAL versus EACH quorum). Session guarantees (read-your-writes, monotonic reads, monotonic writes, writes-follow-reads) can be layered on an eventually consistent store cheaply, by tracking a per-session high-water mark (a log position or version) and only reading from replicas that have caught up to it, or pinning the session to the leader for a short window after a write. Causal consistency is the strongest model that stays available under partition. For AP data, prefer CRDTs or explicit merge functions over timestamp-based last-write-wins, which silently drops concurrent updates under clock skew. Finally, partition handling needs fencing tokens or leases so a deposed leader can't keep writing; heartbeat timeouts alone don't prevent split brain.
Step by stepA flash sale when the link between two cities breaks
An online store runs a 12 PM flash sale: 500 phones at a big discount. The store serves customers from two , one in Mumbai and one in Hyderabad, and each shopper is sent to the nearer one. Let's follow what happens when the link between them breaks in the middle.
-
11:59, the setup. The stock counter for the sale phone, "500 left", lives in Mumbai, with a synchronous copy in a second Mumbai building. Hyderabad holds copies of product pages, photos and reviews, and keeps carts for its own shoppers. When a Hyderabad shopper taps Buy, Hyderabad asks Mumbai to take one phone off the counter, which adds about 20 milliseconds.
-
12:00:00, the sale opens. About 3 lakh people tap Buy in the first minute. Each buy is a tiny change to the counter: "if more than zero left, take one". Mumbai handles each in a couple of milliseconds.
-
12:00:20, the link is cut. About 340 phones have been sold; about 160 are left. Hyderabad's requests to Mumbai stop getting replies. Within about 3 seconds, missed heartbeats tell Hyderabad it's cut off.
-
Hyderabad decides, feature by feature. Product pages, photos and reviews keep loading from its own copies; they might be a minute old, and nobody minds. The stock badge switches from "160 left" to "Limited stock", because Hyderabad no longer knows the real number. Carts keep working locally and will be merged later. But Buy is held: shoppers see "Confirming your order…" for up to 30 seconds, then "We couldn't confirm this right now, please try again". Hyderabad refuses rather than guesses.
- 1
Hyderabad shopperHyderabad site
12:00:10: Buy
- 2
Hyderabad siteMumbai stock
Take one phone (about 20 ms away)
- 3
Mumbai stockHyderabad site
Done: about 160 left
- 4
Hyderabad siteMumbai stock
12:00:20: link cut, no reply
- 5
Hyderabad siteHyderabad shopper
Confirming your order… then please try again
- 6
Mumbai shopperMumbai stock
Buy: still works on this side
- 7
Mumbai stockMumbai shopper
12:00:45: the last phone is sold
-
Mumbai carries on. It still owns the counter, so its shoppers keep buying. The last phone goes at 12:00:45.
-
12:01:10, the link returns. Hyderabad catches up on the counter in a moment. Its shoppers' held and retried buys, about 9,000 of them, now all get an honest "Sold out". Their carts and wishlists, changed during the split, are merged with no loss.
-
The result. Exactly 500 phones sold, none twice. About 50 seconds of a worse experience for Hyderabad shoppers.
Now imagine the other choice. If Hyderabad had kept selling from its last known count of 160, it might have sold another 160 while Mumbai was selling the same 160. That's about 160 orders to cancel, 160 refunds to process, and 160 customers who were told "Order confirmed" and then weren't. For the stock counter, availability would have cost far more than 50 seconds of waiting.
But notice that the store didn't stop Hyderabad entirely. Pages, photos, reviews and carts stayed available the whole time. Only the one feature where a wrong answer is costly was made to wait. That is choosing per feature in practice.
What happens when…Situations
Real situations this idea runs into, and how well-built systems handle each one.
What happens when you change your profile photo and still see the old one
Kabir uploads a new profile photo. The save goes to the main copy of the database, the , and succeeds. Half a second later he refreshes, and his read happens to go to a replica that hasn't received the change yet. The old photo comes back. Kabir thinks the upload failed and does it again, or gives up on the app.
Promise read your own writes. For about a minute after someone changes something, send that person's reads of their own data to the main copy, or to a replica known to have caught up. Everyone else can keep reading from any replica; they won't notice a two-second delay.
- 1
KabirApp
Upload a new profile photo
- 2
AppMain copy
Save it: done in 10 ms
- 3
Main copyReplica
Copy the change (arrives 2 seconds later)
- 4
KabirApp
Refresh half a second later
- 5
AppReplica
Read the profile from the nearest copy
- 6
ReplicaKabir
The old photo comes back
- 7
AppMain copy
The fix: his own profile is read here for a minute
What happens when your feed goes back in time on refresh
Priya opens her feed and sees a new post from her cousin. She pulls to refresh, and the post has vanished. Refresh again and it's back. Each refresh went to a different replica, and one of them was further behind than the others. Nothing was lost, but the app feels broken.
Offer monotonic reads: keep each user's reads on the same replica for their session, a , or remember the newest version they've seen and only read from replicas that have reached it. Data may still be slightly old, but it never moves backwards.
What happens when both sides keep selling during a split
A store's two data centres each believe they can sell stock, and the link between them breaks. Each side carries on with its own count. When the link returns, they've each sold the last 50 units of a popular shoe: 100 orders for 50 pairs. When two parts of a system both act as the one in charge, it's called .
For anything that must never be sold twice, only one side may change it at a time: the side holding the majority of copies, or the one that owns that item. The other side holds or refuses those changes until the link returns. Systems also make sure a machine that has lost its role can't keep writing, a safeguard called .
What happens when the network is slow rather than cut
A link between two buildings isn't broken; it's dropping a fraction of messages and taking 2 seconds to answer instead of 2 milliseconds. Heartbeats arrive some of the time. One side can't tell whether the other is alive and slow or truly gone, and if it waits too long to decide, every user request waiting on that link is stuck too.
Set clear timeouts based on measured normal behaviour, say a few times the usual slowest reply, and treat "no answer in time" as "unreachable" for that request. Systems that choose consistency then refuse quickly with a clear message, instead of hanging; systems that choose availability fall back to their local copy. Either way users get an answer fast.
What happens when a reply appears before the comment it answers
Neha posts a comment on a photo, and Arjun replies to it. Their comments are saved in different places and copied around separately. A friend in another city sees Arjun's reply, "Haha, so true!", under a photo with no comment for it to answer. A moment later Neha's comment appears. Confusing, and in a group chat, it can change the meaning of a conversation.
Use causal consistency for things that depend on each other: each reply carries a note of what it depends on, and a copy doesn't show the reply until it has the comment it answers. It costs very little compared with strong consistency, and it removes the most confusing kind of out-of-order surprise.
What happens when a payment terminal can't reach the bank
A shop's card terminal or a small-value payment feature loses its connection. Refusing every payment would stop the shop from selling anything; accepting every payment risks spending money that isn't there. This is the bank-branch choice in miniature.
Many payment systems choose availability with a limit: they allow small payments offline up to a low cap, with a running total stored on the card or device, and settle them with the bank once the connection returns. The possible loss is small and known in advance. Anything above the cap still needs the bank to confirm.
Watch outCommon mistakes
Picking one setting for the whole app
A team decides "we're a strongly consistent system" and makes every read and write wait for full agreement. Pages that show reviews and recommendations become slow, and during a split the whole app goes down, not just checkout. Another team makes everything eventually consistent for speed, and oversells stock in its first big sale.
Decide per kind of data. List the main things the app stores, ask what a wrong answer and a refused answer each cost, and choose strong consistency only where wrong answers are expensive. Most data in most apps can be relaxed.
Reading from a replica straight after a write
A wallet app saves a payment on the primary database and then shows the new balance by reading from a replica. Most of the time the replica has caught up. During busy hours it's a second behind, so customers see their old balance, assume the payment didn't go through, and pay again.
After a user changes something, read it back from the primary, or from a replica confirmed to be up to date, for that user. Use replicas freely for reads where a slightly old answer does no harm.
Assuming eventual means within a second
"Eventually consistent" has no deadline. A team builds a feature assuming replicas are always under a second behind. One evening a heavy batch job slows replication, and replicas fall 10 minutes behind. Order statuses, read from replicas, show "preparing" for food that was delivered long ago, and support gets thousands of calls.
Measure replication lag all the time and alert when it crosses a limit. When lag is high, send important reads to the primary or show a "may be a few minutes old" note. Design features to tolerate the worst lag you've seen, not the usual lag.
Claiming the system can ignore partitions
In an interview or a design review, someone says "we'll be consistent and available, because our network is reliable". Then a cloud building loses its link for 4 minutes, both halves keep accepting writes, and the team spends a week reconciling records by hand.
Assume links will break and decide in advance what each feature does when they do: which side keeps writing, which side goes read-only, and how changes are merged afterwards. Test it by deliberately cutting links in a practice environment.
Interview angleIn an interview
CAP comes up in almost every system design interview that has more than one database copy or more than one region. It usually arrives as "what happens if this data centre is cut off?" or "is your system consistent or available?" Interviewers are checking that you know partitions are a fact of life and that you choose deliberately, per feature.
A strong answer covers:
- what a partition is, and that you can't opt out of it;
- for each main kind of data, whether you keep it strongly consistent or let it be eventually consistent, and why;
- what users see in each case during a split: a clear "try again" or slightly old data;
- the everyday cost of agreement: synchronous copies nearby, lagging copies far away;
- the session guarantees users notice, like read your own writes;
- how the system decides which side keeps writing, usually a majority.
Follow-up: "Is your design CP or AP?" Neither, as a whole: the booking and payment path is consistent and refuses during a split, while browsing, search and counts stay available with slightly old data.
Follow-up: "How would you show a user their own update immediately with replicas?" Route that user's reads of their own data to the primary for a short window after they write, or read from a replica only once it has caught up to their last write.
Follow-up: "What does PACELC add to CAP?" That even without a partition, you're choosing between speed and agreement on every request, because waiting for distant copies takes time.
Try it in the simulator
Goal: Add a read replica, then switch off the database and see which requests still work.
Add a read replica and connect both app servers to it. Run it, then use Chaos to switch off the database for a while. Reads keep coming from the replica (the app stays available, though it can't show new changes), but every save fails until the database is back. That's the trade-off, live: answering from an old copy versus refusing to change anything.
Quick checkCheck your understanding
Words in this lessonThe jargon, in plain English
Every technical word used above. In the lesson, tap any word with an ⓘ to see its meaning right there.
- Availability
- Whether the app answers at all. "Highly available" means it keeps responding even when some parts fail.
- Causal consistency
- A promise that things which depend on each other are seen in the right order, so nobody sees a reply before the message it answers.
- Data centre
- A building full of servers, with its own power and cooling. Big apps run in several, often in different cities.
- Eventual consistency
- A guarantee that copies will agree once updates stop arriving, but may briefly disagree.
- Fencing
- Cutting off a machine that has been replaced, by blocking its network or taking away its right to write, so it can't keep making changes if it turns out to be still running.
- Heartbeat
- A small "I'm alive" message a machine sends every second or so. When heartbeats stop, others assume it has failed.
- Last write wins
- A simple rule for two changes to the same thing at nearly the same time: the change that arrives last is the one that's kept.
- Monotonic reads
- A promise that once you have seen a change, you won't later see an older version, usually kept by sending each user to the same copy of the data.
- Network partition
- When the network between parts of a system breaks, so some machines can't reach others even though each is still running.
- Overselling
- Selling more of something than you actually have, like confirming 1,200 orders for 1,000 phones. Booking and sale systems are built to make this impossible.
- PACELC
- An extension of the CAP trade-off: during a network split you choose between availability and consistency, and Else, in normal times, between speed (latency) and consistency.
- Primary database
- The main copy of a database. Every change is saved here first, then copied to the replicas.
- Read replica
- A copy of a database that follows the primary's changes and serves reads, taking load off the primary.
- Read your own writes
- A promise that after you change something, you'll see your own change, even if others see it a little later.
- Replication lag
- The delay between a change on the primary database and that change appearing on a replica.
- Split brain
- When two machines both believe they are the leader and both accept changes, so the data stops agreeing.
- Sticky session
- A load balancer setting that keeps sending the same user to the same server, usually by a cookie. It lets that server keep things in memory, but unbalances load and loses them if the server dies.
- Strong consistency
- A promise that every read sees the latest saved change, as if there were only one copy of the data, even though there are several.
- Synchronous replication
- Copying each change to a replica and waiting for it to confirm before telling the app the save worked. Nothing confirmed is lost if the primary dies, but every save takes a little longer.
- Timeout
- A limit on how long to wait for something before giving up and reporting an error.
Enjoyed this? Keep going for free
This lesson is a free preview. Sign up free to save your progress and earn XP, and to unlock all 232 lessons, the simulator challenges and mock interviews.
Sign up free