Design: a payment system

The full lesson, with a worked example: Design: a payment system. The customer's side is in UPI: money debited but not received and the double tap on Pay.

Requirements

Functional

• Pay: accept UPI, cards, wallets and netbanking for an order.
• Refund: full and partial refunds, each linked to the original payment.
• Notify: tell the order service when a payment succeeds or fails.
• Record and match: keep a record of every movement of money and match it daily against what providers and banks report.

Non-functional

• Never charge twice, even when customers tap twice, phones resend or servers retry.
• Never lose a record of money that moved.
• Auditable: every rupee can be explained years later.
• Secure: card numbers never touch our servers; callbacks are verified.
• Correct before fast: a few seconds of waiting is fine. A wrong answer is not.

Out of scope unless asked: fraud scoring, currency conversion, payouts to sellers (mention them as separate services).

Estimates

Say a big Indian shopping app.

• Orders: 20 lakh payments a day ≈ 23 a second on average.
• Peak: a sale evening at about 20 times average ≈ 500 payments a second.
• Writes: about 10 records per payment (payment, state changes, ledger lines, outbox) ≈ 5,000 writes a second at peak.
• Storage: about 2 KB per payment ≈ 4 GB a day, under 1.5 TB a year.
• Money: 20 lakh × about ₹800 ≈ ₹16 crore a day. A bug that double-charges 0.1% of payments hurts 2,000 customers every day.

The point to say aloud: the load is modest; the hard part is correctness. One well-run relational database with a synchronous replica handles this, and it gives you real transactions.

API

• POST /v1/payments with header Idempotency-Key: ord-812-pay. Body: orderId, amountPaise, currency, method. Returns 201 with paymentId and status (usually PENDING, plus what the app must do next, such as approve in a UPI app).
• Same key, same body again: returns the original payment and its current status, never a new charge.
• Same key, different amount or method: 409 Conflict. That's a bug, not a retry.
• Same key while the first request is still running: 409 with "in progress", or the caller waits. Never let two run at once.
• GET /v1/payments/:id returns status, amount and state history. Checkout polls this while pending.
• POST /v1/payments/:id/refunds with its own Idempotency-Key and amountPaise. Refunds for a payment can't add up to more than was captured.
• POST /v1/webhooks/:provider receives results. Check the signature first, record the event by the provider's event ID, reply 200 quickly, and do the work afterwards.
• Events out (through the outbox): payment.succeeded, payment.failed, refund.succeeded, each carrying paymentId so receivers can ignore repeats.

Data model

• idempotency_keys: key (unique per client), request_hash, payment_id, stored_response, expires_at. Keep keys longer than any client retries, say 24 hours to a few days.
• payments: payment_id, order_id, amount_paise (a whole number), currency, method, status, version, provider, provider_ref, created_at, updated_at.
• payment_transitions: payment_id, from_status, to_status, cause (reply, webhook, status check, timeout), at. The history of every move.
• ledger_accounts: one per bucket of money: due from provider, owed to seller, platform commission, provider fees, refunds.
• ledger_entries: entry_id, txn_id, account_id, side (debit or credit), amount_paise, created_at. Append only. The entries of each txn_id sum to zero.
• refunds: refund_id, payment_id, amount_paise, status, its own key.
• outbox: id, topic, payload, created_at, sent_at.
• recon_items: one row per mismatch, with what we have, what the provider has, and how it was resolved.

Money is whole paise as integers: ₹799.99 is 79,999. Never a decimal floating-point number, or the books drift by fractions of a paisa.

High-level design

1. 1

   CheckoutPayment service

   Pay Rs 799, key ord-812-pay

2. 2

   Payment serviceLedger

   p-77 created, then PENDING (no entries yet)

3. 3

   Payment serviceProvider

   Charge Rs 799, reference p-77

4. 4

   ProviderPayment service

   Timed out: result unknown

5. 5

   CheckoutPayment service

   Retry, same key: here is p-77, pending

6. 6

   Payment serviceProvider

   Status of p-77?

7. 7

   ProviderPayment service

   Succeeded, Rs 799

8. 8

   Payment serviceLedger

   One transaction: SUCCEEDED + balanced entries + outbox row

9. 9

   Payment serviceOrder service

   p-77 paid (outbox, at least once)

• Payment service: the front door. Checks the key, records the payment, talks to the provider, applies every state change.
• Payment provider: a payment gateway that connects to UPI, the card networks and banks. Most companies use one or two instead of integrating every bank.
• Provider callbacks: results arrive by webhook, signed by the provider.
• Ledger: the append-only record of every movement of money.
• Outbox sender: delivers "paid" and "failed" messages to the order service.
• Reconciliation: a daily job that matches the ledger against the provider's settlement file.

Card details go straight from the app to the provider's certified vault, which returns a card token. Our systems only ever store the token.

Deep dives

Idempotency keys

• The caller makes the idempotency key once, when the customer commits to paying, often from the order ID, and sends the same key on every retry from every device.
• The service inserts the key under a uniqueness rule. New key: create the payment. Existing key with the same request hash: return the stored payment. Different hash: refuse.
• A server-made key per request defeats the point; every retry would look new.
• Our own call to the provider also carries a key made from our payment ID (p-77), so our retries can't double-charge there either.
• See Pattern card: idempotency for the general recipe.

The state machine and unknown results

• A payment is always in exactly one named state; only listed moves are allowed. Results arrive from replies, webhooks and status checks, often twice and out of order. A second "succeeded" changes nothing; "failed to succeeded" is never applied silently, it raises an alarm.
• Each move is one short transaction with a version check (optimistic locking), so two updates arriving together can't both win.
• A timeout means "we don't know", never "failed". The payment stays pending. Wait for the webhook and ask the provider for the status with exponential backoff: after 5 seconds, then 10, 20, 40.
• After a set window, say 30 minutes, mark it failed and release the order. A late success is still caught: complete the order if possible, otherwise refund it automatically as a compensating action.
• Never call the provider inside an open database transaction. Save pending and commit, call the provider, then save the result in a second short transaction.

The double-entry ledger

• Every movement is a ledger transaction of debits and credits that sum to zero. A ₹799 payment with a ₹40 platform fee: debit "due from provider" ₹799; credit "owed to seller" ₹759; credit "platform commission" ₹40.
• Append only. Mistakes are fixed with a new correcting transaction. A refund is a new transaction in the other direction, not a deletion.
• Balances are sums of entries, not a number someone overwrites. Any balance can be traced to its lines.
• Check "this transaction sums to zero" in the same database transaction as the insert.
• No entries while pending: no money has moved yet.

Telling other services, without a distributed transaction

• The provider is another company's system, so one distributed transaction across it, or two-phase commit, isn't available and would hold locks while waiting.
• Instead each side commits alone, and the whole checkout runs as a saga: reserve stock, take payment, confirm order, with an undo (release stock, refund) for each step.
• The outbox pattern joins the steps: the state change and a "send this" row commit together, and a sender delivers it. If the service crashes after committing, the sender still finds the message.
• Delivery is at least once, so the order service ignores a paymentId it has already handled. At-least-once delivery plus idempotent handling gives an exactly-once effect.

Webhooks and reconciliation

• Anyone can post to the webhook address. Verify the provider's digital signature before reading anything, and confirm amount and status with a direct status check before acting on large payments.
• Each morning the provider sends yesterday's payments and the settlement paid into our bank. Reconciliation matches it to the ledger by payment ID and amount. Say 99.9% match.
• The rest go to a mismatch queue: we think it succeeded but the provider doesn't list it; they list it but we never recorded it; or the amounts differ. Each fix is a new ledger entry, never an edit.
• Match at three levels: each payment, each settlement batch, and the bank statement. Provider fees are booked here too.

Trade-offs

• Correctness over speed: keys, states, status checks and daily matching add delay and code. Worth it.
• Correctness over availability: if the payment database is down, say "try again in a minute" rather than accept payments you can't record.
• One strong database first: relational, with transactions and a synchronous replica. Later, shard by merchant or payment ID so one payment's records and ledger lines stay together.
• One provider or two: a second provider keeps payments flowing in an outage, behind a circuit breaker, but doubles integration and reconciliation work.
• Pending window: a short window releases stuck orders sooner but means more late successes to refund; a long one keeps customers waiting.

Interview talking points

• State "never charge twice" and "never lose a record" first, then show with the estimate that load is small.
• Idempotency key from the caller, unique in the database, with a request hash.
• The state machine, pending for unknown results, status checks to settle them.
• Double-entry, append-only ledger in integer paise.
• Outbox plus idempotent receivers instead of a distributed transaction.
• Signed webhooks, daily reconciliation, card data only as tokens.
• Chargebacks: when a customer disputes a card payment, record a chargeback as a new ledger transaction linked to the original payment, never edit the sale.

Follow-up questions

Can you guarantee exactly-once delivery? Not over a network. You get at-least-once delivery with idempotent handling at every receiver, which together give an exactly-once effect.

The provider times out on half its requests at 8 pm on sale day. What now? A circuit breaker stops new payments going there and routes them to a second provider with fresh references. Pending ones are settled by status checks once it recovers, and customers are offered another method instead of a spinner.

How do you stop two concurrent requests with the same key from both charging? The key insert is unique, so only one wins. The loser either waits for the winner's stored response or gets "in progress"; it never starts its own charge.

Why not keep a balance column per seller and update it? Overwrites hide which update went wrong and can be lost under concurrency. Derive balances from ledger entries; cache them as a rebuildable view if reads are heavy.

How would you scale the database? Read-heavy screens use replicas. For writes, shard by merchant or payment ID so all of one payment's rows sit on one shard and every transaction stays local.

How long do you keep idempotency keys? Longer than the longest retry window of any caller, say a few days, then expire them. The payment itself is kept for years for audits.