Everyday storyGetting sugar: the shelf, the kirana, the next town
You need sugar for chai. If the dabba is on the kitchen shelf, it takes five seconds. If you have to walk to the kirana shop down the lane, ten minutes. If every shop nearby is shut and you drive to the wholesale market in the next town, two hours.
Same sugar, wildly different waits. And notice: walking faster barely helps. Where the sugar is matters far more than how quickly you move. A sensible household keeps a dabba on the shelf, buys from the kirana when it runs low, and makes the long trip only once a month for a big sack.
Now think of Diwali, when twenty relatives arrive. Nobody counts the exact grains of sugar the family will need. Your mother does a quick sum in her head: about two cups of tea per person per day, four days, a spoon each, plus the sweets. Round up, buy a little extra, done. Computers live by both habits: keep the things you use often close by, and plan with quick, rounded sums for the busiest days rather than the quiet ones.
In a real productA video call with family in the US, and an online game
On a call with a cousin in New Jersey, there's a small pause before they laugh at your joke. Nothing is broken. Your voice really does take about a tenth of a second to cross the ocean, and their laugh another tenth to come back. People sometimes talk over each other on these calls for exactly this reason: each of you hears the other a moment late.
Gamers know this number by name. Mobile games show a "ping" in the corner, say 30 ms when playing on a server in Mumbai, and 180 ms when matched to one in Singapore or Europe. At 180, your shots seem to land a beat late. It's also why apps used in India try to run their servers in Indian , and why shopping apps plan for the midnight sale rather than for an ordinary Tuesday afternoon. Every one of those choices starts with two questions: how far away is the data, and how many people will ask for it at once?
The ideaHow it works
The ladder of waits
is how long you wait for an answer, and it depends mostly on where the data is. Computers work in tiny units. A millisecond (ms) is a thousandth of a second. A microsecond is a thousandth of that, and a nanosecond is a thousandth again. Those are hard to feel, so imagine reading from the computer's took 1 second. Then:
- reading from its would take about 17 minutes;
- a to another computer in the same data centre, about 1.5 hours;
- a round trip across India, say Mumbai to Delhi, about 4 days;
- a round trip to the US and back, about 3 to 4 weeks.
In real units that's roughly 100 nanoseconds for memory, 100 microseconds for an SSD read, half a millisecond inside a data centre, about 30 ms across India and about 200 ms to the US. You don't need to memorise these exactly. You need the shape: each step down the ladder is roughly a hundred to a thousand times slower than the one above it. That's why a program that does a thousand memory reads is still faster than one that makes a single trip across the country.
Why distance wins
Data travels through glass fibre cables as pulses of light, and light in glass covers about 200 km every millisecond. That sounds fast until you add up the distances. Mumbai to Delhi is about 1,400 km in a straight line, but cables follow roads and railways, and the signal passes through dozens of relay machines on the way. In practice one way takes 12 to 15 ms, so a round trip is about 30.
Mumbai to the US east coast is around 13,000 km as the crow flies, and the undersea cables wind much further than that. One way takes about 100 to 110 ms. No amount of money or faster computers changes this: you can't buy faster light. The only way to cut it is to move the data closer to the user.
Here's what one request to a faraway server looks like. The work itself is quick; the trip dominates:
- 1
User in MumbaiServer in Virginia
About 110 ms across the ocean
- 2
Server in VirginiaDatabase
About 1 ms, same building
- 3
DatabaseServer in Virginia
About 5 ms to read
- 4
Server in VirginiaUser in Mumbai
About 110 ms back
Out of about 225 ms, the server and database spend 6. The other 219 are the ocean. Speeding up the database tenfold would save about 5 ms, and the user would never notice. Putting a server in Mumbai would save over 200, and everyone would.
There's one more hop people forget: the last stretch from the phone to the nearest mobile tower. On 4G it often adds 15 to 30 ms each way, more on a crowded network. So a user in Pune talking to a server in Mumbai still waits perhaps 50 to 60 ms for a round trip, even though the cities are only 150 km apart.
Averages hide the slow ones
If 100 people open your app, they don't all wait the same time. Most are quick; a few are slow because a server was busy, a disk was slow or a packet got lost. Engineers describe this spread with percentiles. The is the middle: half of requests are faster. The is the slow edge: 99 out of 100 requests are at least that fast, and the unlucky 1 in 100 are slower.
Why care about 1 in 100? Because a busy app serves lakhs of requests an hour, so 1% is thousands of real people. And because the slow edge grows when one screen needs several answers at once. A food app's home screen might ask four services for restaurants, offers, delivery times and suggestions, then wait for all of them:
Three services answer in 30 ms, but the screen can't show until the fourth arrives at 900 ms. The page is only as fast as its slowest part. This sending of one request to many places is called , and it multiplies bad luck: if each of 10 services is slow 1 time in 100, about 1 page in 10 hits at least one slow one.
The estimation toolkit
Engineers size systems with maths: quick, rounded sums that are wrong by a little but right about which parts matter. A few handy numbers do most of the work:
- A day has 86,400 seconds; round it to 1 lakh. A month is about 25 lakh seconds.
- So 1 crore requests a day is about 100 a second on average (it's really about 115).
- Traffic is never flat. Lunch, dinner, a cricket match or a sale can push the busiest minute to 2 to 5 times the average, sometimes far more.
- Sizes go up in steps of a thousand: a kilobyte (KB) is a short text message, a megabyte (MB) is a phone photo or two, a gigabyte (GB) is a film, a terabyte (TB) is a thousand films.
Rounding is the point, not a shortcut. If one server handles 100 requests a second and you need 115, the answer is "two servers, not twenty", and that's what you needed to know.
Storage works the same way. Say a photo app gets 20 lakh uploads a day at about 2 MB each. That's 40 lakh MB, or 4 TB a day. Over a year, 4 × 365 is about 1,500 TB, which is 1.5 petabytes. That one line tells you photos can't live on a single machine and need cheap . Network traffic follows too: if each photo is viewed 10 times, the app sends out 40 TB a day, and that bill may be bigger than the storage bill.
From requests to servers
The last step turns traffic into machines. Take the busiest second, divide by what one server comfortably handles, and add spare room.
Say dinner time brings 600 requests a second and one server handles about 200 flat out. Three servers would be exactly full, so you run four, each doing about 150. The fourth isn't waste: it means no server runs at its limit, and if one crashes the other three can still carry the load.
A second handy rule links speed and crowding. The number of requests in progress at once equals the arrival rate times the time each one takes. This is . At 600 requests a second, each taking 0.2 seconds, about 120 requests are being worked on at any instant. If a slow database makes each request take 2 seconds instead, 1,200 are in progress at once, and servers that were comfortable start running out of room. Slowness and overload feed each other.
The big idea: distance is the biggest delay, so keep data close to users, and size everything with rounded sums for the busiest minute, not the average one.
For experienced engineers
Keep percentiles honest. You can't average p99s across servers or minutes; merge raw latency histograms (HDR histograms or t-digests) and compute percentiles from the merged data. Beware coordinated omission: a load generator that waits for each reply before sending the next quietly stops sending during a stall and under-reports the tail by orders of magnitude, so drive tests at a fixed arrival rate. Tail amplification under fan-out is 1 − (1 − p)^n, so a page touching 100 back ends at 1% slowness each sees a slow call on about 63% of loads; the standard defences are hedged or tied requests after the p95, tight per-call deadlines with partial results, and reducing fan-out width. Queueing explains why headroom matters: for a simple single-server queue the mean wait grows like 1/(1 − ρ), so going from 50% to 90% utilisation multiplies queueing delay about five times, and tail latency degrades faster still. For capacity models, start from measured per-instance throughput at your p99 target (not the throughput at saturation), apply Little's law to size thread pools and connection pools, and plan N+1 or N+2 per zone so a zone loss doesn't push survivors past the knee.
Step by stepDinner time on a food app's menu page
It's 8:30 pm in Pune. Priya taps a biryani place in a food delivery app. Let's follow her one request, then size the system for everyone doing the same thing.
-
The tap leaves her phone. The app's servers run in a data centre in Mumbai, about 150 km away. Over 4G, the trip there takes about 25 ms: most of it is the hop from her phone to the tower, not the distance between cities.
-
The app server checks the . It asks a fast in-memory store for the menu of restaurant 812. That takes under 1 ms. This time it's a : nobody has opened this menu in the last few minutes.
-
It reads the database. The menu (80 dishes, prices, photos' addresses) comes back in about 5 ms. The server stores a copy in the cache so the next person gets it in 1 ms, and builds a reply of about 60 KB.
-
The reply travels back. Another 25 ms or so, plus a few ms to send 60 KB over her connection. The menu appears about 60 ms after her tap. People feel anything under about 100 ms as instant.
- 1
Priya in PuneApp server in Mumbai
Open the menu: about 25 ms over 4G
- 2
App server in MumbaiCache
Menu for restaurant 812? (under 1 ms)
- 3
CacheApp server in Mumbai
Miss: not stored yet
- 4
App server in MumbaiDatabase
Read the menu (about 5 ms)
- 5
DatabaseApp server in Mumbai
80 dishes, about 60 KB
- 6
App server in MumbaiPriya in Pune
Menu on screen, about 60 ms after the tap
-
What if the servers were in the US? Each direction would cost about 110 ms on top of the 4G hop, so the same tap would take about 280 ms, and the app makes several such calls when a screen opens. The page would feel sluggish even though the server did exactly the same 6 ms of work.
-
Now size it for everyone. Say the app serves 50 lakh menu views a day. 50 lakh ÷ 1 lakh seconds is about 50 a second on average. But orders bunch up at lunch and dinner, so plan for 5 times that: about 250 a second at 8:30 pm.
-
Servers. If one app server comfortably handles 100 menu requests a second, 250 needs three, and a fourth as spare room: four servers.
-
Cache and database. If 9 menus in 10 come from the cache, the database sees only 25 reads a second at the peak, an easy load. Without the cache it would see all 250.
-
Bandwidth. 250 replies a second × 60 KB is about 15 MB a second going out at the peak, well within one data centre's means.
Ten minutes of rounded sums, and the team knows where the servers go, how many to run and that the cache matters more than the database size.
What happens when…Situations
Real situations this idea runs into, and how well-built systems handle each one.
What happens when a page makes ten calls one after another
An app server in Mumbai needs data from a partner service hosted in the US, say ten items' stock levels. The code asks for item 1, waits for the answer, then asks for item 2, and so on. Each question is a round trip of about 200 ms, so the page takes 2 seconds just waiting on the ocean, though the partner's work takes 5 ms per item.
Ask once for all ten items in a single batched request, or send the ten requests at the same time instead of one after another. Either way the page pays for one round trip, about 0.2 seconds. Better still, keep a nearby copy of data that changes slowly so most pages don't cross the ocean at all.
What happens when a midnight sale brings twenty times the usual traffic
A shopping app normally sees about 300 requests a second. At midnight the big sale opens, and in the first ten minutes 6,000 a second arrive. The servers were sized for an ordinary evening's peak, so requests queue up, pages time out and people refresh, which adds even more requests.
Estimate the peak in advance from last year's sale and marketing plans, then run a at that level. Add servers and warm the caches before midnight (), since takes minutes to react. For the very busiest pages, a lets people in at a rate the system can handle.
What happens when servers run close to full
A team sees their servers at 90% busy and thinks, "Good, we're getting our money's worth." But at 90%, a new request almost always finds the server already busy and has to wait its turn. The wait doesn't grow gently: at 50% busy a request might wait 10 ms in line, at 90% about 50 ms, and at 98% several hundred. Any small burst tips it over.
Plan for servers to sit around 50 to 70% busy at the peak, and watch and p99 latency together. Scale up when the p99 starts rising, not when the machines hit 100%, because by then users are already waiting.
What happens when the storage estimate forgets the copies
A team estimates 4 TB of photos a day and buys storage for that. Three months later it's full. Their sum counted each photo once. In reality every photo is stored three times for safety, there are four resized versions for different screens, and there's a nightly .
Multiply by everything that makes copies: the (often 3), extra sizes such as , backups and indexes. Then add growth, say 30 to 50% a year, and round up. The real figure here is closer to 15 TB a day, which changes the plan.
What happens when users are in small towns on patchy networks
The team tests the app in a Bengaluru office on fast Wi-Fi: every screen loads in 150 ms. A user on a crowded 4G network in a small town in Bihar sees 2 to 3 seconds. Each trip from her phone takes 150 ms or more, and the home screen downloads 2 MB of images and code, which takes seconds over a slow link.
Make fewer trips and send fewer bytes: combine requests, use , send smaller images to slow connections and cache them on the phone. Serve images and code from a with in many Indian cities, and measure latency from real users' phones, not just from the office.
Watch outCommon mistakes
Sizing for the average day
A team reads "1 crore requests a day", divides to get about 115 a second and buys capacity for 115. Then dinner time, Sunday evenings and every festival sale arrive at 3 to 10 times that, and the app slows down at exactly the moments it earns the most.
Find the busiest minute: from past traffic graphs, or by assuming 3 to 5 times the average when there's no history. Size for that, add spare room so one machine can fail, and treat known events like sales as their own, higher estimate.
Tuning the code when the trip is the problem
Users complain the app is slow, so engineers spend two weeks shaving the server's work from 20 ms to 8 ms. The page still feels slow, because each screen makes four trips of 200 ms to a server on another continent. They saved 12 ms out of over 800.
Measure where the time goes before fixing anything: network trips, waiting in line, database and server work, each separately. Fix the biggest slice first. Very often that means fewer round trips or moving servers and data closer to users.
Mixing up bits and bytes
Network speeds are quoted in per second, file sizes in bytes, and a byte is 8 bits. A team plans to serve 100 MB a second over a "100 Mbps" link and finds it carries only about 12 MB a second, eight times less than they assumed.
Write the units on every line of a sum: MB versus Mb, per second versus per day. Divide link speeds by 8 before comparing them with file sizes, and sanity-check each result against something you know, like how long a film takes to download at home.
Chasing false precision
In an interview or a planning meeting, someone works out that they need 3.47 servers and 1,283.6 GB of storage, then spends twenty minutes on the decimals. Meanwhile they never checked whether the peak was 2 times the average or 20 times, which matters far more.
Round to one or two significant figures, state your assumptions out loud ("say 5 times the average at peak"), and spend your time on the numbers that change the design. Being roughly right quickly beats being exactly right too late.
Interview angleIn an interview
Almost every system design interview has an estimation step, usually right after you agree on what the system does. The interviewer says "let's assume 10 crore users" and watches whether you can turn that into requests a second, storage and servers without getting lost. They also listen for latency sense: do you know a trip across the world is hundreds of milliseconds while a cache read is under one?
A strong answer covers:
- daily active users × actions per user → requests a day → requests a second (using 1 lakh seconds a day);
- the read-to-write mix, since reads and writes are often sized separately;
- the peak, as a stated multiple of the average;
- storage per item × items a day × days kept × copies;
- servers needed at the peak, with spare room;
- where latency comes from: trips, queues, disk and work, and where the servers sit relative to users.
Follow-up: "Your p99 doubled but the average is unchanged. What do you look at?" The slow tail: one slow dependency in a fan-out, a busy database, servers near full, garbage collection pauses or a share of users on a far-away route. Break the slow requests down by step to find which one grew.
Follow-up: "How much storage for five years?" Items a day × size × 365 × 5, multiplied by copies (often 3), plus growth. Then say whether it fits on a few machines or needs object storage or sharding.
Follow-up: "Your users are in India and the database is in the US. What changes?" Every trip costs about 200 ms, so move servers and data to an Indian region, put a CDN in front of static content and cut the number of trips per screen.
Try it in the simulator
Goal: See how much of the latency is just distance, then cut it.
These users are far from the server: every trip across the network takes 80 ms. Press Run and note the p99. Then select the arrow between the users and the server, delete it, and connect them again: a new connection counts as "close by" (1 ms). Run again and compare how much of the wait was the trip and how much was real work.
Hint
Most of the original wait is the trip, not the server's work. Putting servers closer to users, or keeping copies of content near them, is how real apps fix this.
Quick checkCheck your understanding
Words in this lessonThe jargon, in plain English
Every technical word used above. In the lesson, tap any word with an ⓘ to see its meaning right there.
- Autoscaling
- Adding servers automatically when traffic rises and removing them when it falls, so you pay only for what you need.
- Back-of-the-envelope estimate
- A quick, rounded calculation, like 1 crore requests a day ≈ 100 a second, used to decide which parts of a design matter. Close enough beats exact.
- Backup
- A copy of your data kept somewhere else, so you can get it back if the original is lost or broken.
- Bandwidth
- How much data is sent over the network. Cloud providers charge for data sent out to viewers, so popular pictures and videos can cost more to send than to store.
- Bit
- The smallest unit of data, a single 0 or 1. Eight bits make one byte. Network speeds are quoted in bits per second, file sizes in bytes.
- Cache
- A fast store that keeps copies of frequently used data so you don't have to fetch it from the slow source every time.
- Cache miss
- A request the cache couldn't answer, so it had to go to the original source.
- CDN
- Content Delivery Network: many caches spread around the world that serve files from a location near each user.
- Compression
- Making a file smaller so it's quicker to send and cheaper to store, usually by dropping detail you won't notice, like shrinking a video to suit a phone screen.
- Data centre
- A building full of servers, with its own power and cooling. Big apps run in several, often in different cities.
- Edge server
- A server close to where you are (often in your own city) that keeps copies of popular content, so it reaches you faster than from a distant data centre.
- Fan-out
- Sending one thing to many places, like telling every follower about a new post or every viewer about a new comment.
- Latency
- How long one request takes, from sending it to getting the answer back. Usually measured in milliseconds (ms).
- Little's law
- A simple rule for any line of work: items arriving per second times seconds each item takes equals the number in progress at once. 4 jobs a second at 6 seconds each means about 24 being worked on.
- Load test
- A rehearsal where a system is flooded with fake users on purpose, to find out how much it can take and what breaks first, before real users arrive.
- Memory (RAM)
- A computer's short-term working space. Reading from it takes about a ten-millionth of a second, far faster than any disk, but it is wiped when the power goes off.
- Object storage
- A cheap, practically unlimited store for big files like photos and videos. You save and fetch each file by a key, like a cloakroom ticket.
- p50 latency (median)
- The middle response time: half of requests are faster than this and half are slower. It shows the typical experience but hides the slow ones.
- p99 latency
- The 99th percentile: 99 out of 100 requests are at least this fast. It describes the slowest experiences users actually have.
- Pre-warming
- Getting servers, caches and pages ready before a known rush, so the system isn't cold and slow when the crowd arrives.
- Replication factor
- How many copies of each piece of data are kept, often 3.
- Round trip
- A message going to another computer and the answer coming back. Its time is the wait you actually feel.
- SSD
- A solid-state drive: storage made of memory chips with no moving parts. It keeps data when the power is off and reads much faster than an old spinning hard disk, though still far slower than RAM.
- Thumbnail
- A small, low-detail version of a photo or video, used in grids and previews because it loads almost instantly.
- Utilization
- The share of time a component is busy. Near 100%, queues grow and latency climbs fast.
- Virtual waiting room
- An online queue. Instead of everyone hitting the booking system at once, people wait on a holding page and are let in a few at a time.
Enjoyed this? Keep going for free
This lesson is a free preview. Sign up free to save your progress and earn XP, and to unlock all 232 lessons, the simulator challenges and mock interviews.
Sign up free