Design: a notification system

The full lesson, with a worked example: Design: a notification system. The user's side is in Push notifications.

Requirements

Functional

• Any service can notify a user: it names the user, a message type and the details. It never talks to providers itself.
• Channels: push, SMS, email and an in-app inbox.
• Preferences: per type and per channel opt-outs, quiet hours in the user's own time zone, and a frequency cap such as "at most 3 offers a day".
• Scheduling and campaigns: "remind at 7 pm", and offers to crores of users at once.
• Tracking: sent, delivered, opened, failed, for support and analytics.

Non-functional

• Urgent is fast: an OTP arrives within a few seconds, even mid-campaign.
• Nothing important is lost: accepted means stored.
• No repeats: one alert, one arrival.
• Survives provider failure: an SMS gateway going down must not stall OTPs.
• Out of scope (say so): message content, A/B testing of copy, the marketing tool that builds audiences.

Estimates

Say 20 crore users, about 5 notifications each a day.

• Normal load: 100 crore a day ÷ 86,400 s ≈ 12,000 a second, about 35,000 at the evening peak.
• A campaign: 2 crore offers paced over 20 minutes = 2 crore ÷ 1,200 s ≈ 17,000 a second, on top of normal load.
• OTPs: about 50 lakh a day ≈ 60 a second. A tiny share of traffic, but each has someone waiting.
• Tap-through spike: if 5% of 2 crore tap within a minute, that is 10 lakh app opens in 60 s. Your own campaign can flood your own app.
• SMS cost: say 10 to 20 paise each. 2 crore SMS at 15 paise ≈ ₹30 lakh, so offers go by push and SMS is kept for OTPs, payments and fallbacks.
• Log storage: about 500 bytes per send × 100 crore a day ≈ 500 GB a day; 60 days ≈ 30 TB.

The shape that matters: lumpy traffic, and the most important messages are the smallest share. That argues for separate lanes.

API

• POST /v1/notifications: fields userId, type (for example order.picked_up), params (values for the template), optional channels, sendAt and an Idempotency-Key header. Returns 202 Accepted with a notificationId. A repeat with the same key returns the first result, not a second send.
• POST /v1/campaigns: audienceId, type, params, startAt, paceSeconds. Returns a campaignId; the service expands the audience itself.
• GET /v1/notifications/:id: status history (accepted, suppressed, scheduled, sent, delivered, failed).
• PUT /v1/users/:id/preferences: opt-outs per type and channel, quiet hours, time zone.
• POST /v1/devices: register or refresh a device token for a user.
• POST /v1/provider-events: the webhook providers call with delivery receipts and "token invalid" replies.

Priority comes from the type, in a central list the platform team owns, never from the caller. Otherwise every team marks its messages urgent.

Data model

• notifications: id, idempotencyKey (unique per sender), userId, type, priority, channel, status, createdAt, sentAt, providerMessageId. About 500 bytes; partition by day so old data drops cheaply.
• preferences: userId, opt-outs per type and channel, quietStart, quietEnd, timeZone, language. Read on every request, so cache it.
• devices: userId, platform, token, lastSeenAt. A user often has two or three.
• templates: type, channel, language, body with placeholders, priority. Small and cached in every API server.
• cap counters: userId + type + day → count, in an in-memory store with a one-day expiry.
• campaigns: id, audienceId, type, startAt, pace, progress (batches sent).

The sent log (the notifications table) is both the audit trail and the de-duplication record.

High-level design

1. 1

   Order serviceNotification API

   Notify user 5521: order picked up (key n-81)

2. 2

   Notification APIPreferences

   Opted in? Quiet hours? Under the cap?

3. 3

   PreferencesNotification API

   Allowed, push, Hindi

4. 4

   Notification APIOrder service

   202 Accepted (saved before replying)

5. 5

   Notification APIUrgent queue

   n-81, push, high priority

6. 6

   Urgent queuePush worker

   Next job, ahead of any campaign

7. 7

   Push workerPush provider

   Send to device token, key n-81

8. 8

   Push providerPhone

   Delivered

9. 9

   Push providerPush worker

   Receipt: delivered (log updated)

• One front door. The API saves the request with its idempotency key before replying, so a crash after "accepted" cannot lose it.
• Rules first. Preferences, quiet hours and caps are checked before anything is queued. A message that won't be sent should cost almost nothing.
• Templates. The type picks a message template in the user's language and fills in the details.
• Scheduler. Holds anything not for now: quiet hours, send-later, and campaign batches.
• Lanes. Separate queues for urgent, normal and bulk, usually split per channel too, each with its own workers.
• Channel workers. Talk to push providers, an SMS gateway and an email service, each wrapped in rate limits and a circuit breaker.
• Feedback. Receipts update the log; "token invalid" deletes the token at once.

Deep dives

Lanes, not one queue

A queue is first come, first served. Drop 2 crore offers into it at 8 pm and an OTP at 8:01 waits behind all of them. Separate queues with reserved workers keep the urgent lane nearly empty, so it moves in milliseconds whatever bulk is doing. A priority flag on one queue is not enough: a backlog still starves urgent work. Rate-limit the urgent lane per sender, so one team's bug cannot fill it.

Pacing campaigns

• Why: providers throttle senders who go too fast, and every send causes taps that hit your own servers.
• How: split the audience into batches (say 1 lakh), release them at a steady rate with a token bucket per provider, add jitter, and warn the app's server teams beforehand.
• Re-check at send time: a user may have opted out since the campaign was scheduled.
• "At 9 pm sharp" becomes "starting at 9 pm" over 15 to 30 minutes. Nobody notices.

Exactly once, in practice

Queues promise at-least-once delivery, and a worker whose request timed out cannot know if the SMS went. So:

• Every notification carries its key from the start.
• The worker checks the sent log before sending and records "sent" straight after.
• Providers that accept a key get the same key, so a retry that slips through is dropped at their end.
• Retries use exponential backoff with jitter (1 s, 2 s, 4 s), a cap on tries, then a dead-letter queue.

That is deduplication: at-least-once delivery that looks exactly-once to the user.

Channels and fallback

• Push is nearly free but not guaranteed: phone off, app removed, notifications muted.
• Escalate for things that matter: push first, and if the app hasn't reported it seen within about 5 minutes, send an SMS.
• Provider failure: a circuit breaker stops calling a sick gateway, jobs stay queued, and urgent SMS switches to a backup gateway. Bulk SMS just waits.
• Stale tokens: delete on the first "invalid" reply, or you pay to send into the void and providers may mark you as careless.

Noise control

• Grouping: collect "liked your photo" events for a short window, then send one "Rahul and 49 others liked your photo".
• Per-user, per-type limits catch a buggy sender firing 50 alerts a minute, which deduplication misses because each has its own key.
• Alert the owning team when a sender jumps far above its normal rate.

Trade-offs

• Central service vs each team on its own: more to build and run, but one place for preferences, caps, retries and the log, and consistent behaviour for users.
• Checking the sent log vs speed: a few milliseconds per send. Worth it for SMS and payments; some skip it for low-value pushes and accept rare repeats.
• Push vs SMS: push is cheap and rich but unreliable; SMS is reliable and costs money. Use push by default, SMS for urgent and fallback.
• Fast campaigns vs safe ones: a burst reaches everyone sooner but gets throttled and floods the app; pacing is slower but smooth.
• Reach vs annoyance: more sends reach more people, until they mute the app and you lose even the OTP channel.

Interview talking points

• Start with channels and the lumpy load: a steady base, campaign spikes, a few urgent OTPs.
• One API with idempotency keys; save before acknowledging.
• Preferences, quiet hours and caps checked before queuing, in the user's time zone.
• Separate lanes by priority and channel, with reserved workers; priority set by type, not by caller.
• Workers: backoff with jitter, dedupe by key, dead-letter queue.
• Provider failure: circuit breakers, a backup SMS gateway, deleting dead tokens.
• Campaign pacing and the tap-through spike, which most candidates forget.
• A status history per notification for support ("did Meera get her OTP?").

For experienced engineers

Model each notification as a state machine (accepted, suppressed, scheduled, sent, delivered, failed) and keep lanes as separate consumer groups with their own capacity. If a calling service must not lose a request across its own crash, it writes to an outbox in its database and a relay publishes it, rather than calling the API inside its transaction.

Follow-up questions

How do you make sure an OTP isn't delayed by a campaign? Give OTPs their own queue and reserved workers, keep that lane lightly loaded on purpose, and route urgent SMS to a backup gateway when the main one's circuit breaker trips.

Can you promise exactly-once delivery? Not over a network. You promise at-least-once and deduplicate by key in the sent log and at the provider, which looks exactly-once to the user.

How would you send 2 crore notifications in 20 minutes? Split the audience into batches, release them through a per-provider token bucket at about 17,000 a second, re-check preferences at send time, and warn the app's server teams about the tap spike.

A user in Dubai got an offer at 6 am. What went wrong? Quiet hours were checked on the company's clock. Store each user's time zone from the phone, check quiet hours in it, and let the scheduler hold the message until the user's own morning.

How do you stop one buggy service sending a user 50 alerts a minute? A per-user, per-type rate limit at the front door, grouping similar alerts in a short window, and an alarm when any sender's volume jumps far above normal.

Where would the sent log live, and how big is it? About 500 GB a day at this scale, so a store partitioned by day with a 60-day retention, keyed by notification id for de-duplication and by user for support queries.