Pattern card: caching

The full lesson, with a worked sale-night example: Why apps remember things.

What it is

A cache is a small, fast store (usually memory) that keeps copies of data that is read often, in front of a bigger, slower source of truth such as a database.

• Why it is fast: memory instead of disk, and a plain key lookup (product:42) instead of joins, sorting and permission checks. Most caches are key-value stores.
• Hit and miss: a hit returns in about 1 ms. A miss goes to the database (about 15 ms), returns the answer and keeps a copy.
• What it buys: lower latency per read, and far fewer reads reaching the database.
• What it costs: copies can be out of date, and a new moving part can fail.
• The rule: the database stays the truth. You must be able to throw the whole cache away without losing anything that matters.

When to use it

Use it when:

• Reads far outnumber writes: product pages, menus, profiles, timetables, config.
• Popularity is lopsided: a few thousand items out of a crore get most views. A cache holding 1% of the data can often answer over 90% of reads.
• Slightly old is fine: a description that is a minute behind hurts nobody.
• One page needs many lookups: 20 lookups at 15 ms one after another is 300 ms; at 1 ms it is 20 ms.

Don't use it (or bypass it) when:

• It must be exact: bank balances, stock at checkout, the price you charge. Read the database at the moment money moves.
• It is read once: OTPs, one-off reports, log lines. Hit rate near zero, so the cache only adds a step.
• It changes on almost every read: a live seat count mid-rush.
• It is personal and the cache is shared: never put pages with personal details in a CDN.

How it works

Reads (cache-aside). The app asks the cache; on a miss it reads the database and stores a copy with a TTL.

1. 1

   App serverCache

   GET product:42

2. 2

   CacheApp server

   Miss

3. 3

   App serverDatabase

   Read product 42 (about 15 ms)

4. 4

   App serverCache

   SET product:42, TTL 5 min

Writes. Save to the database first, then delete the cached copy. The next read refills it. Delete rather than overwrite: two updates close together can finish in the wrong order and leave the older value cached with nothing to fix it.

1. 1

   App serverDatabase

   Save the new price first

2. 2

   DatabaseApp server

   Saved

3. 3

   App serverCache

   DELETE product:42

Freshness has two tools, use both:

• TTL (expiry): bounds how long any copy can be wrong. Pick it by asking "how wrong, for how long, before someone is hurt?"
• Delete on change (invalidation): keeps copies fresh in normal running. The TTL is the safety net for the code path that forgets to delete.

Eviction. A full cache drops something to make room, usually the least recently used item (LRU).

Layers. Most systems stack several caches, each catching some requests so the next sees fewer:

• Device: app code, fonts, images. Fastest, hardest to clear.
• CDN: photos, video, pages that are the same for everyone, served from a nearby city.
• In-process: a tiny copy in each app server's memory for a few seconds. Good for hot keys.
• Shared cache: a separate fleet every app server asks. This is "the cache" in most diagrams.
• Database buffer: databases keep recent pages in memory too.

Closer to the user means a faster hit but harder deletes, because there are millions of phones and thousands of CDN servers holding copies.

Variants

• Cache-aside (lazy loading). How: the app reads the cache, fills it on a miss, deletes on write. Good for: most read-heavy data; only caches what is actually asked for. Cost: the first read of each key is slow; a small race can leave an old value (see Pitfalls).
• Read-through. How: the app only talks to the cache, and the cache loads misses from the database itself. Good for: keeping loading logic in one place. Cost: needs a cache that knows your database; same freshness issues as cache-aside.
• Write-through. How: every save goes to the cache and the database together. Good for: data read soon after it is written. Cost: every write is slower, and the cache fills with things nobody reads.
• Write-back (write-behind). How: saves go to the cache and reach the database later in batches. Good for: counters such as video views. Cost: a cache crash loses the unsaved writes. Never for payments or anything you can't lose.
• Write-around. How: saves go only to the database; the cache picks them up on the next miss. Good for: data written often and read rarely. Cost: the first read after a write is a miss.
• Refresh-ahead / stale-while-revalidate. How: serve the slightly old copy while one request refreshes it in the background. Good for: hot keys where a brief stale read is fine. Cost: you knowingly serve old data for a few seconds.

Numbers to know

• Hit about 1 ms, database read about 15 ms, so a miss costs about 16 ms (cache check plus database).
• Load left on the database = reads a second × miss rate. At 20,000 reads a second:
  • 50% hits: 10,000 reach the database. The cache barely helps.
  • 90% hits: 2,000.
  • 99% hits: 200. Nine more points cut database reads tenfold.
• Average read time = hit rate × hit time + miss rate × miss time. 0.9 × 1 + 0.1 × 16 = about 2.5 ms at 90%; about 1.15 ms at 99%; about 8.5 ms at 50%.
• Size from the busy part, not the whole dataset: about 1% of a lopsided catalogue can give 90%+ hits. Past that point, more memory adds few hits.
• Below about 50% hit rate for a kind of data, ask whether it belongs in the cache at all.
• Example TTLs:
  • description and photos: about an hour;
  • price: about a minute, rechecked at checkout;
  • stock while browsing: about 2 seconds;
  • live cricket score: a few seconds;
  • bank balance: don't cache.
• Jitter TTLs: say 55 to 65 seconds instead of exactly 60, so keys written together don't expire together.
• Negative cache: keep "not found" for about 30 seconds.
• Cold cache: a database sized for 90% hits sees about 10 times its normal reads when the cache comes back empty.

Pitfalls

• Stampede. A hot key expires at peak and 5,000 requests miss at once, all asking the database the same thing. Fix: request coalescing (one request refills, others wait a few milliseconds), serve stale while refreshing, and jittered TTLs.
• Cold restart. Restarting every cache server together turns 90% hits into nearly 100% misses. Fix: restart one at a time, pre-warm popular keys before known rushes, and make sure the database (or a limit in front of it) survives the miss load for a few minutes.
• Hot key. One product gets 50,000 views a second and all of them land on the one cache server that owns its key. Fix: a short in-process copy on each app server, or several copies of the key on different cache servers, picked at random per read.
• Forgotten delete. A screen built by another team changes the price but never deletes the copy. Fix: route every change through one shared save-then-delete function, and always keep a TTL.
• Stale race in cache-aside. A slow reader misses and reads the old row; a writer updates the row and deletes the key; the slow reader then stores the old value. Fix: short TTLs, versioned keys, or compare-and-set on a version number.
• Missing user in the key. cart-page as a key shows one person's cart to everyone. Fix: put everything that changes the answer in the key, such as cart:user-1182.
• Lookups for things that don't exist. Scrapers asking for missing IDs miss every time. Fix: cache "not found" briefly and add rate limiting.
• Cache as the only copy. Likes or sessions kept only in the cache vanish on restart. Fix: the database holds anything you can't lose.
• Caching everything. Rarely read data evicts popular items and lowers the hit rate, and every miss now costs an extra hop.

For experienced engineers

A shared cache cluster is itself sharded, usually with consistent hashing so adding a node moves about 1/N of keys. If reads go to replicas, a second delayed delete covers replication lag. Decide early what happens when the cache is down: if the database cannot take the full miss load, the cache has quietly become a hard dependency and you need load shedding.

In an interview

"Add a cache" is not an answer. Say what, where, for how long, and what happens when it fails.

• What: data read often and changed rarely, with a rough hit rate.
• Where: device, CDN, in-process, shared cache, and which data sits at each layer.
• The sum: reads a second × miss rate = database load. Show it.
• Freshness: a TTL per kind of data, plus delete on change.
• Pattern: cache-aside by default; say when write-through or write-back fits.
• Failure: stampede, hot key, cold restart, and what the database survives without the cache.
• Never cached: money, stock at checkout, passwords, anything personal in shared layers.

Follow-ups and short answers:

• "How do you keep the cache consistent with the database?" Save to the database, then delete the key; keep a TTL as a backstop. Where stale is never acceptable, read the database.
• "What if the whole cache goes down?" Every read misses, so the database takes many times its load. Size it to survive for a while, or shed load and serve a simpler page while the cache warms.
• "Why not just make the cache bigger?" Past the busy part of the data, extra memory adds few hits. Measure hit rate against size first.
• "Why delete instead of updating the cached value?" Two updates can finish out of order and leave the older value cached. A delete makes the next reader fetch whatever the database holds now.
• "How do you spread a cache across many servers?" Hash keys to servers with consistent hashing, so adding a server moves only a small share of keys. See Pattern card: sharding.