Pattern card: message queues

The full lessons, with worked examples: Queues and background work and Delivery guarantees and the outbox.

What it is

A queue holds jobs or messages between the service that creates them (the producer) and the workers that handle them (consumers).

• Asynchronous work: the request saves what it must, drops a job and replies at once. Slow steps happen later (async).
• Decoupling: producers don't know how many workers exist or whether one is restarting. If the SMS provider is down for ten minutes, orders still go through; SMS jobs wait.
• Shock absorber: bursts pile up and drain while workers run at a steady pace.
• Fan-out: with publish–subscribe, one event ("order 812 placed") reaches every subscriber, each with its own copy.
• The catch: most queues promise at-least-once delivery. Nothing is lost, but some jobs arrive twice.

When to use it

Use it when the work:

• Is slow: notifying a restaurant tablet, sending email or SMS, resizing photos, building reports.
• Can fail and be retried: calls to outside providers.
• Arrives in bursts: dinner rush, end of a match, a noon flash sale.
• Has several listeners: restaurant, rider search, SMS and reports all react to one order.

Don't use it when:

• The user needs the answer before moving on: "is this seat free?", "did my payment go through?" That belongs in the request.
• Arrivals beat drain rate all day: a queue absorbs bursts, not a permanent shortage. Add workers instead.
• The extra moving part isn't worth it: a queue is one more thing to run and watch, and cause and effect end up seconds and machines apart.

How it works

1. Save and record the event together (outbox). Writing to the database and then to the queue is a dual write: a crash between them loses the event or announces a ghost order. The outbox pattern saves the order and an "event to send" row in one transaction; a relay publishes unsent rows and marks them sent. Change data capture can replace polling.

2. Hand out jobs on a lease. A worker takes a job and it is hidden from others for, say, 30 seconds (a lease). On success the worker sends an acknowledgement and the job is deleted. No ack before the lease ends, and the job reappears for another worker.

1. 1

   QueueWorker

   Job sms-812, hidden for 30 s

2. 2

   WorkerDatabase

   Insert key + do work, one transaction

3. 3

   DatabaseWorker

   Committed

4. 4

   WorkerQueue

   Ack: delete the job

3. Make handlers idempotent. A worker can do the work and crash before acking, or be slow enough that its lease runs out. So:

• every job carries an idempotency key, such as sms-812-placed;
• the worker inserts the key into a handled-keys table (with a unique rule) in the same transaction as the work; a repeat fails the insert and does nothing;
• prefer "set status to Paid" over "add ₹500", because setting twice is harmless.

4. Retry transient errors, park permanent ones. A transient error (provider overloaded) gets exponential backoff with jitter: wait about 1, 2, 4, 8 seconds plus a random extra. A poison message (malformed number, deleted order) moves to a dead-letter queue after, say, 5 tries, for someone to fix and replay.

5. Watch two numbers: jobs waiting, and age of the oldest job. Scale workers on age; apply backpressure or load shedding if it keeps rising.

Variants

• Work queue (competing consumers). How: each job goes to exactly one of many workers, acked and deleted. Good for: background jobs such as emails and thumbnails. Cost: no replay once a job is deleted; order across workers is only rough.
• Pub/sub. How: one event, many subscribers, each with its own copy. Good for: one action that triggers several services; adding a new listener needs no change to the producer. Cost: every subscriber must handle repeats; harder to see who depends on an event.
• Partitioned log (stream). How: events are appended to a log split into partitions and kept for days (a retention period); each consumer group tracks its own offset. Good for: high volume, replaying history, stream processing, keeping order per key. Cost: parallelism capped by partition count; one stuck message blocks its partition.
• Priority queues. How: a priority queue, or separate queues with their own workers. Good for: OTPs in seconds while marketing email waits an hour. Cost: low-priority work can starve; watch its age too.
• Delayed / scheduled jobs. How: a job becomes visible only after a set time. Good for: reminders, retries with backoff, "cancel unpaid order after 10 minutes". Cost: timing is approximate; jobs still run at least once.
• Delivery promises:
  • At most once: send once, never retry. Some loss, no repeats. Fine for analytics pings and live score ticks.
  • At least once: retry until acked. No loss, some repeats. The default for orders and money.
  • Exactly once: only inside a closed system. End to end you build "effectively once" from at-least-once plus idempotent handlers.

Numbers to know

• Reply time: doing five slow steps inline takes over 3 seconds; save, enqueue and reply takes about 40 ms.
• Queue growth = arrivals − drain rate.
  • Burst: 1,000 jobs in a second, workers finish 200 a second, about 800 wait and drain in about 5 seconds.
  • Overload: 200 a second arrive, 150 finish, the queue grows by 50 a second, about 1.8 lakh after an hour.
• Wait for a new job ≈ queue length ÷ drain rate: 12,000 waiting ÷ 400 a second = about 30 seconds.
• Workers needed (Little's law): in flight = arrival rate × time per job. 300 emails a second × 0.2 s = 60 in flight; at 10 per worker that's 6 workers minimum, run 8 or 9.
• Lease: longer than the slowest normal job (about 30 seconds is common); extend it with heartbeats for long jobs.
• Backoff: 1, 2, 4, 8 seconds plus jitter; dead-letter after about 5 tries.
• Duplicates: at 1 in 10,000 deliveries and 50 lakh events a day, about 500 a day.
• Dedup table: keep keys longer than the longest redelivery gap, say 7 days. 2 crore events a day × 50 bytes × 7 days = about 7 GB.
• Outbox delay: polling every 100 to 500 ms, usually well under a second. One relay sending 200 rows every 200 ms moves about 1,000 events a second.
• Message size: keep messages small, well under a kilobyte; put files in object storage and send the key.

Pitfalls

• Queue hiding a capacity gap. Timeouts vanish but the queue grows all day; password resets arrive two hours late. Fix: compare arrival and drain rates, alert on oldest-job age, cap queue size so overload is a visible error.
• "Done" when only queued. "Payment successful" shown before the bank declines. Fix: honest wording ("Payment processing") and a pending state that updates.
• Handlers not safe to repeat. "Add 50 points" runs twice after a lease expires. Fix: idempotency key checked in the same transaction; set final values instead of adding.
• Dedup in a separate store. Checking keys in a cache, then writing to the database, reopens the gap on a crash. Fix: key and effect in one transaction.
• Ack before work. A crash after the ack loses the job: at-most-once by accident. Fix: ack after the work is committed.
• Lease shorter than the job. Slow jobs get handed to a second worker while still running. Fix: set the lease above the slowest normal time and extend it while working.
• Retrying poison messages forever. They burn workers and block ordered partitions. Fix: separate permanent from transient errors; dead-letter early.
• Retry storms. Thousands of failed jobs retry at the same instant. Fix: backoff with jitter and a retry budget.
• Out-of-order events. "Cancel" processed before "place". Fix: route by order ID to one partition, and carry version numbers so older events are ignored.
• Whole files in messages. A 4 MB photo per job slows the queue and hits size limits. Fix: object storage plus a key.

In an interview

The cue is any step that is slow, can fail, or arrives in bursts. Separate what the user must wait for from what can happen later.

• Split the request: what stays inline, what goes on the queue, and why.
• Size it: arrival rate, time per job, workers by Little's law, how long a burst takes to drain.
• Delivery: at-least-once, idempotency keys, ack after work, leases.
• Failure: backoff with jitter, dead-letter queue, outbox for the save-and-publish gap.
• Operations: queue length and oldest-job age, scaling on age, separate queues for urgent work.

Follow-ups and short answers:

• "How do you process a message exactly once?" You can't guarantee it end to end. Accept at-least-once and make the handler idempotent with a unique key checked in the same transaction as the change.
• "Workers can't keep up during a sale. What now?" Scale workers on oldest-job age, split urgent from deferrable work, and if the backlog still grows, apply backpressure or shed the least important jobs.
• "How do you keep events for one order in order?" Send all events with the same order ID to one partition, handled in sequence, and check version numbers so a late, older event is ignored.
• "Why not one distributed transaction across the database and the queue?" Many queues don't support it, and it is slow and ties both systems' availability together. The outbox gets the same result with one local transaction.
• "How do you know the outbox is healthy?" Alert on the age of the oldest unsent row past a few seconds, plus dead-letter size and the rate of duplicates rejected.