Latency numbers cheat sheet

The full lesson, with the stories behind these numbers: Latency and estimation. For turning users into requests and servers, see the estimation cheat sheet.

Remember the shape, not the decimals. Hardware changes every few years; the ratios between the rungs barely move. Every figure here is "about".

Units

• 1 second = 1,000 milliseconds (ms).
• 1 ms = 1,000 microseconds (µs).
• 1 µs = 1,000 nanoseconds (ns).
• People feel: under about 100 ms as instant; about 1 second as a pause that keeps their flow; about 10 seconds as "it's broken", and they leave.

The ladder of waits

• Read from memory: about 100 ns.
• Random read from an SSD: about 100 µs (1,000× memory).
• Round trip inside one data centre: about 0.5 ms.
• Round trip across India (Mumbai to Delhi): about 30 ms.
• Round trip India to the US east coast: about 200 ms.

Scaled to human time, pretend a memory read takes 1 second:

• SSD read: about 17 minutes.
• Same data centre round trip: about 1.5 hours.
• Mumbai to Delhi and back: about 4 days.
• India to the US and back: about 3 to 4 weeks.

The takeaway: a thousand memory reads are still faster than one trip across the country.

Compute and memory

• CPU instruction: well under 1 ns.
• Small fast cache on the chip: about 1 ns.
• Main memory read: about 100 ns.
• Read 1 MB sequentially from memory: tens of µs.
• Compress 1 KB of text: a few µs.
• Hash a short key, or parse a small JSON message: around 1 µs to a few µs.

Storage

• SSD random read (4 KB): about 100 µs.
• Read 1 MB sequentially from SSD: about 0.5–1 ms.
• Spinning disk seek: about 5–10 ms, so random reads on spinning disks are about 100× slower than on SSD.
• Read 1 MB sequentially from a spinning disk: about 5–10 ms.
• Write that must reach disk safely (flush): about 0.1–1 ms on SSD; this is why databases batch their log writes.
• Fetch an object from object storage in the same region: tens of ms for the first byte.

Networks

Physics first. Light in glass fibre covers about 200 km every millisecond. Cables follow roads and railways and pass through relay machines, so real trips run well above the straight-line minimum.

• Same machine, between processes: tens of µs.
• Same data centre: about 0.5 ms round trip.
• Between zones in one region: about 1–2 ms round trip.
• Mumbai to Delhi: about 1,400 km straight; 12–15 ms one way, about 30 ms round trip.
• Mumbai to Singapore: about 60–70 ms round trip.
• India to Europe: about 120–150 ms round trip.
• Mumbai to the US east coast: about 13,000 km straight, longer by cable; 100–110 ms one way, about 200–220 ms round trip.

The last hop is often the biggest.

• Phone to mobile tower on 4G: about 15–30 ms each way, more on a crowded network.
• A user in Pune to a server in Mumbai (150 km apart): still about 50–60 ms round trip, mostly the tower.
• Home broadband or office Wi-Fi: a few ms for the first hop.
• Patchy small-town 4G: 150 ms or more per trip, plus slow downloads.

Bandwidth.

• Bits vs bytes: network speeds are in bits a second, files in bytes. Divide by 8: a 100 Mbps link carries about 12.5 MB a second; 1 Gbps about 125 MB a second.
• Send 1 MB over 1 Gbps: about 8–10 ms.
• Send 2 MB of images over a slow 4 Mbps phone link: about 4 seconds.

Connection costs

• DNS lookup: near zero if cached; about 20–100 ms if not.
• New TCP connection: 1 round trip before any data.
• TLS setup: 1–2 more round trips on a new connection. From India to the US that alone can be 200–400 ms.
• Reusing a connection: skips all of the above. Keep connections open and pooled.

Everyday operations

• Cache lookup (in-memory store, same data centre): about 0.2–1 ms.
• Indexed database read: about 1–10 ms.
• Database write with a durable commit: a few ms.
• Unindexed scan of a big table: hundreds of ms to seconds.
• Call to another service in the same data centre: about 1–5 ms including its own work.
• Calling a large language model: hundreds of ms before the first word, seconds for a long answer.

Percentiles and fan-out

• p50: the middle request; half are faster.
• p99: 99 in 100 are at least this fast. At lakhs of requests an hour, the slow 1% is thousands of people.
• Averages hide the tail. You can't average p99s across servers either; merge the raw data and recompute.

Fan-out multiplies bad luck. A screen that waits for several services is as slow as the slowest.

• Chance at least one of n calls is slow = 1 − (0.99 to the power n), when each is slow 1 time in 100.
• n = 1: 1% of pages slow.
• n = 10: about 10% of pages slow.
• n = 100: about 63% of pages slow.
• Example: three services answer in 30 ms, the fourth in 900 ms; the screen shows at 900 ms.
• Defences: tight per-call timeouts with partial results, fewer services per screen, and a hedged request (send a second copy if the first is slower than the usual p95).

Busy servers wait longer

Queues explain why "90% busy" is not efficient. Waiting grows roughly like 1 ÷ (1 − busy share).

• 50% busy: a request might wait about 10 ms in line.
• 90% busy: about 50 ms, roughly 5× worse.
• 98% busy: several hundred ms; any burst tips it over.
• Plan for 50–70% utilisation at the peak, and scale when the p99 starts rising, not when machines hit 100%.

Little's law: requests in progress = arrival rate × time each takes.

• 600 requests a second × 0.2 seconds = 120 in flight.
• If a slow database makes it 2 seconds: 1,200 in flight, and thread pools and connection pools run out. Slowness and overload feed each other.

Worked sums

A menu page in Pune, server in Mumbai.

1. 1

   Phone in PuneApp server in Mumbai

   Tap: about 25 ms over 4G, mostly the hop to the tower

2. 2

   App server in MumbaiCache

   Menu 812? Under 1 ms, a miss

3. 3

   App server in MumbaiDatabase

   Indexed read: about 5 ms

4. 4

   App server in MumbaiPhone in Pune

   60 KB reply: about 25 ms plus a few to send. Total about 60 ms

• 25 ms there + under 1 ms cache + 5 ms database + 25 ms back + a few ms to send 60 KB ≈ 60 ms. Feels instant.
• Same page from a US server: add about 110 ms each way ≈ 280 ms, and a screen usually makes several such calls.

A remote trip vs the work.

• 225 ms request to a faraway server, of which 6 ms is server and database work.
• Making the database 10× faster saves about 5 ms. Moving the server to Mumbai saves over 200 ms.

Ten calls one after another.

• 10 sequential calls to a US partner × 200 ms = 2 seconds of waiting, though each item takes 5 ms of work.
• Batched into one request, or sent in parallel: about 0.2 seconds.

A page's latency budget, say 300 ms at the p99.

• Phone to server and back: about 60 ms.
• Edge and load balancer: about 5 ms.
• App work: about 30 ms.
• Three service calls in parallel, each with a 100 ms timeout: about 100 ms.
• Database: about 20 ms.
• Spare for the tail: about 85 ms.

Rules of thumb

• Distance beats code. Put servers and data near users; put images and scripts on a CDN with edge servers in many cities.
• Count round trips per screen. One is fine; ten in a row across an ocean is two seconds.
• Memory is about 1,000× faster than SSD, which is why caches work.
• Measure before tuning: split time into network, queueing, database and work, then fix the biggest slice.
• Test from real phones, not only office Wi-Fi.