Everyday storyThe railway station announcement board
At a big station like Howrah or New Delhi, every train event goes up on the board: arrived, delayed, platform changed. The board doesn't care who's reading. Porters, food stalls, taxi drivers and passengers each watch it and act in their own way. The station master doesn't phone the tea stall to say the Rajdhani is late; he puts it on the board once, and everyone who cares sees it.
Now imagine the station also keeps a printed log of every announcement for the past week, numbered line by line. A porter who went for lunch notes he'd read up to line 4,512, and when he comes back he picks up from line 4,513. A newly opened snack stall can read the whole week's log to learn which platforms get the most hungry passengers. Nobody tears a line off the log when they read it; each reader just keeps their own bookmark.
That is an event log. The board is the stream, each line is an event, each bookmark is a reader's position, and the week-long printout is what lets latecomers and newcomers catch up.
In a real productLive dashboards, fraud checks and 'trending now'
When you place an order on a big food or shopping app, a lot happens in the next second. The payment is checked for fraud. The order shows up on the restaurant's tablet and on a live sales dashboard in an office in Bengaluru. "Trending near you" quietly updates. Your loyalty points move, a delivery partner nearby gets considered, and the order is filed away for tomorrow's reports. The order service didn't call any of those systems one by one. It wrote one event, and they all read it.
You notice this design mostly when it works: live order tracking that updates without refreshing, a "someone just bought this" badge, a bank SMS that arrives seconds after a card swipe, a fraud block that stops a strange payment before it completes. You also notice its edges: the sales count that's a minute behind, or the points that show up a little after the order does. Those small delays are readers catching up with the stream.
The ideaHow it works
From queue to log
In earlier lessons, a handed each job to one and then forgot it. That's perfect for "send this email": one job, one worker, done. But an order isn't one job. Fraud wants it, analytics wants it, search wants it, and next month a team you haven't hired yet will want it too.
Without a shared stream, the order service ends up calling each of them directly:
Every new reader means a change to the order service. If the email sender is slow, ordering slows down. If search is down, the order service must decide whether to fail, retry or skip. The order team ends up responsible for systems it doesn't own.
An , like Apache Kafka, turns this around:
- append ("OrderPlaced #812") to the end of a named stream called a , in order.
- The log keeps events for a , typically days or weeks, whether or not anyone has read them.
- Any number of read the whole stream independently. Each remembers its own position, its , like the porter's bookmark.
- Reading doesn't remove anything. A new reader can start from the oldest retained event, or from now.
It's a cousin of , with one crucial difference: the history stays. That single change unlocks replay, late joiners and easy recovery.
- 1
Order serviceEvent log
OrderPlaced #812 (partition 3, offset 99,812)
- 2
Event logFraud checker
Read #812 within milliseconds
- 3
Event logAnalytics
Read #812 for the live sales dashboard
- 4
Event logSearch updater
Read #812 to update "popular now"
- 5
Fraud checkerEvent log
Write FraudScore #812 = low
Partitions: how one log handles lakhs of events
A single ordered list on one machine would hit a ceiling quickly. So a topic is split into , spread across the servers in the cluster, called . Each partition is its own ordered log with its own offsets.
When a producer writes an event, it attaches a key, such as the order ID or user ID. The key is hashed to pick a partition, so every event with the same key lands in the same partition, in the order it was written. "Order placed", "order paid" and "order delivered" for order A all go to partition 0, so any reader sees them in the right sequence.
Inside a consumer group, each partition is read by exactly one reader at a time, while one reader can handle several partitions. That gives two rules worth remembering:
- Order is guaranteed only within a partition. There is no global order across the whole topic. If two events must stay in order, give them the same key.
- The partition count caps a group's parallelism. With 12 partitions, a group can use at most 12 readers. A 13th sits idle.
When a reader joins, leaves or crashes, the group goes through : partitions are reassigned, and reading pauses for a moment.
Sizing it with rounded numbers
Say a food app handles about 50 lakh orders a day, and each order produces about six events (placed, paid, accepted, picked up, delivered, rated). That's 3 crore events a day.
- Average rate: 3 crore ÷ 86,400 seconds, about 350 events a second.
- Dinner peak during an IPL final, say six times average: about 2,000 a second.
- Each event is about 1 KB, so the peak is about 2 MB a second. That's small for a log; volume is rarely the hard part.
- Storage for 7 days: 3 crore × 7 × 1 KB, about 210 GB. Kept on 3 brokers for safety, about 630 GB.
- If one fraud reader handles about 300 events a second, the peak needs about 7 readers. Choose 12 partitions, leaving room to grow, because adding partitions later reshuffles which key goes where.
The number that matters most in daily life is : how far each group is behind the newest event. If lag is steady near zero, the reader keeps up. If it climbs during the peak and falls afterwards, the log is doing its job of absorbing the burst. If it climbs and never falls, the reader is too slow and needs more readers, faster code, or more partitions.
Safety: what happens when a broker dies
Each partition is copied to several brokers, usually three. One copy is the and takes the writes; the others are followers that copy it. A producer can choose how sure it wants to be before carrying on: wait for the leader only (faster, but a crash can lose the last few events) or wait until enough copies have it (a few milliseconds slower, but survives a broker failing). For orders and payments, teams wait for the copies.
On the reading side, a reader handles an event and then saves its new offset. If it crashes between the two, it restarts from the last saved offset and handles a few events again. So the normal promise is : nothing is lost, but repeats happen. Readers must therefore be built with , for example by recording which order IDs they've already scored. Kafka can offer for work that stays inside Kafka, but anything outside it, such as a database row or an SMS, still needs duplicate-safe readers.
What gets built on top
- computes things continuously: orders per minute per city, a rolling fraud score per card, or a join between "order placed" and "rider assigned" to spot orders waiting too long.
- turns every database change into an event by reading the database's own change log, so search, caches and the warehouse stay in sync without the app writing twice.
- keeps only the newest event for each key. A "product prices" topic then holds the latest price of every product, and a new service can load the full current list just by reading it from the start.
- goes furthest: the log of events becomes the record of truth, and current state is worked out from it.
Kafka isn't the only option. Managed cloud streams and other open-source logs work on the same ideas: append, partition, retain, and let readers keep their own place.
The big idea: write every important event once to a durable, ordered, partitioned log, and let any number of systems read it at their own pace, now or later.
For experienced engineers
Durability is a producer-plus-topic contract: acks=all with min.insync.replicas=2 on replication factor 3 tolerates one broker loss without data loss, and unclean leader election should stay off for anything financial. Enable idempotent producers (on by default in recent clients) so retries don't duplicate within a partition. Batching (linger.ms, batch.size) and compression trade a few milliseconds of latency for large throughput gains. On the consumer side, commit offsets after processing, not before, and make side effects idempotent with a dedupe key or an upsert keyed by event ID; Kafka transactions give exactly-once only for consume-transform-produce loops that stay inside Kafka. Cooperative sticky assignment and static group membership shrink rebalance pauses, which otherwise show up as lag spikes on every deploy. Partition count is hard to change for keyed topics because the hash mapping moves, so size for two to three years of peak. Watch lag in time, not only in messages, and alert on lag growth rate. For cross-service contracts, use a schema registry with backward-compatible evolution (Avro or Protobuf), and for database-to-stream publishing use an outbox table read by CDC rather than dual writes.
Step by stepOne biryani order, from tap to five systems
It's 8:41 PM on a Saturday in Hyderabad. Ananya orders a chicken biryani for ₹320. Let's follow order Z-4471 through the stream.
-
8:41:02.000 – the order is saved. The order service writes Z-4471 to its own database first. That's the record of truth.
-
8:41:02.004 – the event is appended. Using an outbox row saved alongside the order, the service publishes
OrderPlacedto theorderstopic with the key Z-4471. The key hashes to partition 7 of 12. The event is about 1 KB: order ID, restaurant, items, amount, city, time. -
8:41:02.009 – it's safe. The leader for partition 7 writes it at offset 52,10,331 and waits until two followers on other brokers have copied it. Only then does it confirm. Five milliseconds have passed, and Ananya's app already shows "Order placed".
- 1
Order serviceKafka
OrderPlaced Z-4471, key = Z-4471
- 2
KafkaOrder service
Saved on 3 brokers: partition 7, offset 52,10,331
- 3
KafkaFraud checker
Read at 8:41:02.020, score = low
- 4
Fraud checkerKafka
FraudScore Z-4471 = low, bookmark moved on
- 5
KafkaDashboard
Counted in the 8:41 minute for Hyderabad
- 6
KafkaWarehouse loader
Picked up in the 8:45 batch of 3 lakh events
-
8:41:02.020 – fraud reads it. The fraud group's reader for partition 7 had reached offset 52,10,330, so this is next. It checks the card, the address and how many orders this phone has placed today: score low. It writes a
FraudScoreevent and saves its offset as 52,10,331. Elapsed: 20 milliseconds. -
8:41:02.050 – the dashboard counts it. A stream-processing job adds ₹320 to the "Hyderabad, 8:41" minute. When the minute closes, the city team's screen shows about 1,800 orders that minute.
-
8:41:02.300 – search and loyalty catch up. "Biryani" gets a nudge in "trending near you", and Ananya earns 32 points. These readers run a little behind, a few hundred milliseconds, and nobody minds.
-
8:45:00 – the warehouse loader takes a batch. It reads every five minutes, picking up about 3 lakh events in one go and writing them as a single file. Tomorrow morning, the finance team's reports include Z-4471.
-
8:46 – a hiccup. The fraud reader handling partitions 6 and 7 crashes after scoring 200 more orders but before saving its offset. The group rebalances in a couple of seconds, and another reader picks up partition 7 from offset 52,10,331, the last saved bookmark. Those 200 orders get scored again, but because the fraud service stores scores keyed by order ID, the second pass simply overwrites with the same result. Nothing is lost and nothing is doubled.
One write by the order service fed five systems, each at its own speed, and a crash cost two seconds of delay rather than a lost order.
What happens when…Situations
Real situations this idea runs into, and how well-built systems handle each one.
What happens when one restaurant gets 40% of the orders
A celebrity chef launches a ₹99 meal with a big promotion, and for an hour, 40% of all orders go to one restaurant. If the topic is keyed by restaurant ID, every one of those events hashes to the same partition. That partition's reader falls further and further behind while the other eleven readers sit almost idle. Fraud checks for that restaurant's orders arrive minutes late. This is a .
Pick keys for even spread as well as for order. Most readers only need order per order, not per restaurant, so keying by order ID spreads the load across all partitions. Where per-restaurant order really matters, teams split a known hot key into a few sub-keys, such as restaurant ID plus a number from 1 to 8, and merge the results downstream.
What happens when a reader falls hours behind
During a big sale, the analytics readers can't keep up. Lag grows from seconds to three hours. Nothing breaks immediately, because the log simply holds the events. But dashboards show stale numbers, and if the lag ever exceeds the retention period, the oldest unread events are deleted before the reader gets to them. That data is gone for good.
Alert on lag in minutes, and on whether it's growing. Add readers up to the partition count, make each reader faster (batch database writes instead of one per event), and set retention well beyond the worst lag you expect, often 7 days for a stream whose readers normally lag by seconds. For big, rare peaks, size the partitions ahead of time so there's room to add readers.
What happens when one broken event crashes the reader
A new app version sends an order with the amount as text, "three hundred", instead of a number. The search reader crashes on it, restarts from its last offset, reads the same event, and crashes again. Because order within a partition is strict, every event behind it on that partition is stuck too. This is a .
Catch errors per event, retry a small number of times, then move the bad event to a and carry on. Someone reviews the dead-letter queue later. Better still, stop bad events at the door: producers validate events against a shared before writing them.
What happens when the order team renames a field
The order team renames amount to total_amount and ships on Tuesday afternoon. Their own code is fine. But six teams read the orders topic, and three of them start writing zero rupees into reports, quietly, for two days, until finance spots that revenue has dropped to nothing.
Treat events as a public contract. Register each event's format in a that refuses changes that would break existing readers. Add new fields freely, keep old ones until every reader has moved, and for a truly breaking change publish a new version of the event alongside the old one.
What happens when a broker machine dies at peak
At 9 PM on a Friday, one of six brokers loses its disk. It held the leader copies of several partitions, so writes to those partitions briefly can't land there.
Because each partition has copies on two other brokers, the cluster promotes a follower to leader within seconds, and producers retry automatically. Writes confirmed only after the copies had them survive intact. Ordering apps see a short stall, not lost orders. The team replaces the machine, and the new broker copies its partitions back in the background.
Watch outCommon mistakes
Writing to the database and the log separately
The order service saves the order, then publishes the event in a second step. One day the service crashes between the two. The order exists, the customer has paid, but no event was ever published, so the restaurant never hears about it. Or the reverse: the event goes out, the save fails, and fraud scores an order that doesn't exist.
Use the : save the order and an "event to publish" row in the same database transaction, and let a separate process, often change data capture, publish from the outbox. Either both happen or neither does.
Assuming the whole topic is in order
A developer builds a wallet balance from a payments topic keyed by payment ID. A top-up and a spend for the same wallet land on different partitions, and the reader handles the spend first. The balance dips below zero and the payment is wrongly declined.
Key events by the thing whose order matters, here the wallet ID, so all of one wallet's events share a partition. Treat order across different keys as unknown, and design readers so it doesn't matter.
Too few partitions, chosen on day one
A team creates its topic with 3 partitions because traffic is small. A year later, peak traffic needs 20 readers, but only 3 can work. Raising the partition count changes which partition each key maps to, which breaks per-key order during the switch.
Size partitions for two to three years of peak, using rounded sums like the ones above, and leave headroom. A few dozen partitions cost little; hundreds per topic without need add overhead, so don't overshoot wildly either.
Using the log as a request–reply system
A checkout service writes "please check stock" to a topic and then waits for an answer on another topic before it replies to the user. Under load, lag creeps up, and checkout now takes seconds instead of milliseconds, with timeouts that are hard to trace.
Use the log for things that have happened and that others react to in their own time. When the user is waiting for an answer, call the other service directly with a timeout, and publish the outcome as an event afterwards.
Interview angleIn an interview
Event streaming comes up whenever a design has one action that many systems react to: "design a food delivery backend", "design a ride-hailing app", "design a real-time analytics dashboard" or "design a notification system". Interviewers want to see when you'd reach for a log instead of a queue or direct calls, and whether you understand partitions, order and failure.
A strong answer covers:
- why a log fits: one write, many independent readers, replay for new features and bug fixes;
- the topic design: what the events are, what key you partition by, and why that key keeps the right order;
- rough sizing: events per second at peak, event size, retention, partition count;
- delivery promises: at-least-once by default, idempotent readers, the outbox pattern for publishing from a database;
- operations: consumer lag as the health signal, dead-letter queues, schema rules, broker replication.
Follow-up: "How do you guarantee an order's events are processed in sequence?" Key them by order ID so they share a partition, and have a single reader per partition within the group; order across different orders isn't guaranteed and shouldn't need to be.
Follow-up: "Your consumer crashed and restarted. What about duplicates?" It resumes from its last saved offset, so a few events repeat; the reader makes its effects idempotent, for example by upserting results keyed by event or order ID.
Follow-up: "Queue or log?" A queue suits one-off jobs that one worker should do once and forget; a log suits facts that several systems need, possibly later, in order.
Try it in the simulator
Goal: Absorb a burst of events with a queue and a reader that catches up.
Shoppers send about 200 orders a second, and halfway through, the rate spikes to three times normal. Add a queue and a worker (app server → queue → worker → database) and run: the burst lands in the queue and the worker catches up afterwards. Watch the queue length rise during the spike and fall once it passes; that rise and fall is consumer lag. The simulator's queue is simpler than a Kafka log (one reader, no replay), but it shows the same buffering.
Quick checkCheck your understanding
Words in this lessonThe jargon, in plain English
Every technical word used above. In the lesson, tap any word with an ⓘ to see its meaning right there.
- At least once
- A delivery promise: every message arrives, but some may arrive twice, so receivers must handle repeats.
- Broker
- One server in an event-streaming cluster. Brokers store the partitions, accept writes and serve reads; a cluster usually has several so it survives one failing.
- Change data capture
- Turning every change in a database into an event, so other systems can follow along without the app writing twice.
- Consumer group
- A set of readers that share the work of reading an event stream, each group keeping its own position in it.
- Consumer lag
- How far a reader is behind the newest event in a stream, counted in events or in seconds. Growing lag means the reader can't keep up.
- Dead-letter queue
- A separate holding queue for jobs that keep failing after a few retries, so they stop blocking other work and someone can look at them later.
- Event
- A small record that something happened, like "order placed" or "post liked", that other parts of the system can react to.
- Event log
- An append-only, ordered record of events that is kept for days or weeks, so many systems can read it independently and even read it again from the start. Apache Kafka is the best-known example.
- Event sourcing
- Storing every change as an event (like passbook entries) and calculating the current state from them, instead of overwriting it.
- Exactly once
- The ideal delivery promise: every message arrives once and only once. Networks can't guarantee it on their own, so apps get the same effect from retries plus receivers that ignore repeats.
- Hot key
- One item that everyone wants at once, like a celebrity's post. All the traffic lands on the one place it's stored and can overload it.
- Idempotency
- Doing something twice has the same effect as doing it once, which makes retries safe.
- Leader
- The one machine in a group that is allowed to accept changes; the others follow and copy it.
- Log compaction
- A setting that keeps only the latest event for each key and drops older ones, so the log works as an always-current list, like the latest price of every product.
- Message queue
- A buffer that holds messages until a consumer is ready to process them, so producers don't have to wait.
- Offset
- An event's position number inside one partition of an event log. A reader remembers the offset it has reached, like a bookmark, so it can carry on from there after a restart.
- Outbox pattern
- Saving "send this message" in the same database transaction as the change itself, then sending it from there, so it can't be forgotten.
- Partition
- One slice of an event stream or table, so work can be spread across machines; order is kept within each slice.
- Poison message
- A job or message that fails every time it's handled, for example because its data is broken. Retrying it never helps, so it has to be set aside.
- Producer
- Any service that writes events into an event log or queue, like an order service writing "order placed".
- Publish–subscribe
- Senders publish messages to a named channel, and everyone subscribed to that channel receives them. The sender doesn't need to know who is listening.
- Rebalancing
- Reassigning which reader in a consumer group handles which partitions, whenever a reader joins, leaves or crashes. Reading pauses briefly while it happens.
- Retention period
- How long an event log keeps events before deleting the oldest ones, for example seven days. Readers must catch up within it or miss events.
- Schema
- A description of the shape data must have: which fields exist, what type each one is and which are required. Programs use it to check data before trusting it.
- Schema registry
- A shared service that stores the agreed format of each kind of event and rejects changes that would break the readers already relying on it.
- Stream processing
- Computing results continuously as events arrive (counts per minute, fraud scores) instead of in nightly batches.
- Topic
- A named stream of events inside an event log, such as "orders" or "payments". Producers write to it and readers subscribe to it.
- Worker
- A background process that takes jobs from a queue and does them.
Enjoyed this? Keep going for free
This lesson is a free preview. Sign up free to save your progress and earn XP, and to unlock all 232 lessons, the simulator challenges and mock interviews.
Sign up free