Pattern card: idempotency

The full lessons, with worked examples: Retries, backoff and jitter, Tapping Pay twice and Delivery guarantees and the outbox. Related: the queues card and the payments write-up.

What it is

An operation is idempotent when doing it twice has the same effect as doing it once. "Set my address to X" is; "add ₹50 to the wallet" is not.

• Why it matters: a timeout tells you no answer came back, not that nothing happened. The work may have run and only the reply was lost. A blind retry of "charge ₹300" charges ₹600.
• The tool: an idempotency key, a unique name for one intended action, made once and sent with every attempt. The receiver remembers keys it has handled and replays the stored answer for a repeat.
• The honest contract: networks give at-least-once delivery at best. Exactly-once delivery can't be promised between two machines, but at-least-once delivery plus idempotent handling gives "effectively once": the message may arrive twice, but it counts once.

When to use it

Use it when:

• A write can be retried: create order, pay, transfer, send OTP, book a seat. Repeats come from a second tap, the app's own retry, and the network resending.
• A queue delivers work: any consumer of an at-least-once queue will see duplicates. At 1 in 10,000 deliveries repeated and 50 lakh events a day, that's about 500 a day.
• You receive webhooks or callbacks: providers resend until you acknowledge.
• A job may be picked up twice: a worker whose lease expires while it's merely slow hands the job to a second worker.

You don't need keys for:

• Reads: fetching a menu twice changes nothing.
• Naturally repeatable writes: "set status to delivered" is safe, though a version number stops an old update overwriting a newer one.
• Fire-and-forget data: an analytics ping can be at most once; losing one is fine.

How it works

1. Make the key before the first try. The client creates a long random UUID when the user reaches the pay page, so it means "this checkout". Made fresh on each tap, it means nothing. Send it in an Idempotency-Key header.
2. Claim it atomically before any side effect. Insert the key into a table with a unique constraint, in the same transaction as the payment record. Two simultaneous copies collide; only one wins.
3. Answer by state. Never seen: go ahead. In progress: don't start a second charge; tell the client to wait and check back. Done: return the saved answer word for word, success or decline.
4. Store a fingerprint. Keep a hash of the request (amount, order) with the key. Same key with a different amount is refused, not guessed at.
5. Pass a reference downstream. The payment service sends its own reference to the bank, so a retried debit is spotted there too. After a crash mid-call, ask the bank for the status of that reference; never re-charge blind.
6. Expire keys after the retry horizon. At least a day for payments; longer than the longest possible redelivery for queues.

One checkout on a slow train:

1. 1

   AppPayment service

   Pay ₹349, key K-71

2. 2

   Payment serviceKey records

   Insert K-71 as in progress

3. 3

   Payment serviceBank

   Debit ₹349, reference K-71

4. 4

   AppPayment service

   Timeout: retry with K-71

5. 5

   Payment serviceApp

   K-71 in progress: wait, then check

6. 6

   BankPayment service

   Debited

7. 7

   Payment serviceKey records

   K-71 done, reply saved

8. 8

   AppPayment service

   Check K-71

9. 9

   Payment serviceApp

   Paid, reference #8812 (charged once)

The sending side. Saving an order and publishing "order placed" are two writes to two systems, a dual write; a crash between them loses the event or announces a ghost order. The outbox pattern saves the event in the same transaction as the order, and a sender publishes it later. A crash after publishing but before marking it sent means a repeat, which the receivers already ignore.

Variants

• Natural idempotency. How: phrase changes as "set X to Y" with a version, or a conditional update such as UPDATE orders SET status = 'paid' WHERE status = 'awaiting_payment'. Good for: state changes; only one payment can move an order to paid. Cost: doesn't fit "add" or "send" actions.
• Client key plus server records. How: the steps above. Good for: public APIs, payments, order creation. Cost: a table of keys and a lookup on every write; correct only if claim and work commit together.
• Consumer deduplication. How: every message carries an ID; the consumer inserts "#5501 handled" and does the work in one transaction, with a unique constraint on the ID. This is deduplication. Good for: queue consumers, webhooks. Cost: retention sized to the redelivery horizon.
• Outbox or change data capture. How: the event row commits with the business change; a poller (every 100–500 ms) or change data capture publishes it. Good for: never losing events. Cost: one more process, and an outbox table to keep trimmed.
• Keys in a fast in-memory store only. How: a key with a TTL in a cache. Good for: cheap protection for low-stakes actions. Cost: a crash or eviction forgets keys, so it can't protect money.
• Reconciliation as the net. How: daily reconciliation matches your records with the bank's and refunds extras. Good for: what keys can't catch, such as a fresh key for the same order. Cost: slow; fixes the next day.

Numbers to know

• Why retry at all: if 1 call in 100 fails independently, one retry makes it 1 in 10,000; two retries, 1 in 10 lakh.
• Backoff: 100, 200, 400, 800 ms, doubling, with a cap of 5–10 seconds, plus jitter so clients that failed together don't return together.
• Caps: 2–3 tries in total; a deadline of 5–10 seconds when a person is waiting; a retry budget of about 10% of normal traffic.
• Layer multiplication: three layers each making 3 tries is up to 27 calls for one tap. Retry at one layer only.
• Key storage sum: 50 lakh payments a day at about 1 KB each is about 5 GB a day, kept a day or two.
• Dedupe sum: 2 crore events a day at about 50 bytes each, kept 7 days, is about 7 GB.
• Outbox throughput: one sender publishing 200 rows every 200 ms moves about 1,000 events a second, with a delay well under a second.
• Leases: a queue typically gives a worker about 30 seconds before handing the message to another.

Pitfalls

• Making the key inside the retry loop. Every attempt looks new and the protection vanishes. Create it once per user intent.
• Check, then work, in separate steps. Two copies both see "never seen" and both charge. Claim with a unique insert in the same transaction as the work.
• Dedupe table in a different store from the effect. A crash between "remember" and "do" reopens the gap. Keep them in one transaction.
• Caching failures that happened before any work. If validation failed or the service crashed before calling the bank, release the key so a retry can run. Cache only final outcomes.
• Retrying errors that won't change. 400, 401, 404 and "insufficient balance" fail the same way again. Retry timeouts, dropped connections, 503 and 429, honouring Retry-After.
• Retries at every layer. Multiplied retries land exactly when the downstream service is weakest, a retry storm. One layer retries; inner layers fail fast.
• Acknowledging before the work. A crash after the "got it" loses the message: at-most-once by accident. Acknowledge after the work.
• Trusting "exactly once" from the queue. Such promises hold inside the queue system only. A database write, SMS or bank call still needs a key.
• No way out after the last try. Use a circuit breaker for a failing service, a dead-letter queue for jobs, and a clear "confirming, don't pay again" screen for users.

In an interview

It comes up the moment you draw an arrow and the interviewer asks "what if that call fails?", and in every payment, booking or queue design.

• Say the contract: at-least-once delivery plus idempotent receivers.
• Show the key: made per checkout before the first tap, claimed atomically, three states, saved reply, fingerprint, expiry.
• Cover both ends: outbox on the sending side, dedupe in the consumer's transaction.
• Bound the retries: which errors, backoff with jitter, a few tries, a deadline, one layer, a budget.
• Add the net: one-way status changes and daily reconciliation.

Follow-up: "How do you make POST /orders safe to retry?" The client sends a key; the server stores the key and the new order in one transaction and returns the stored order if the key comes again.

Follow-up: "The service crashes after the bank debits but before saving 'paid'." The key stays "in progress"; on recovery or the next retry, the service asks the bank for the status of its reference and finishes the record from the answer.

Follow-up: "Where do you store keys?" In the same database as the records they protect, with a unique index; an in-memory store alone can lose them in a crash.

Follow-up: "Why not one distributed transaction across the database and the queue?" Many queues don't support it, and it's slow and ties the two systems' availability together. The outbox gets the same result with one local transaction.

For experienced engineers

The impossibility behind all this is the Two Generals problem. Store a hash of the canonicalised request with the key (same key, different body is a 422; a concurrent duplicate gets a 409 or waits), and size retention to the maximum redelivery horizon, including replays from a rewound offset.