The happy path is easy: request arrives, work succeeds, response returns. Production is what happens when the client retries, the payment provider times out, the webhook arrives twice and somebody deploys during all of it.
A reliable API assumes the network will eventually have a bad Tuesday.
The network is not a function call
Application code can make an API call look like an ordinary function call: await chargeCard(). That one line hides two machines and a network between them. The request may reach the provider even if your client never receives the response. The provider may succeed while your process crashes. The caller may retry because it cannot know what happened.
Designing only for the perfect request-response sequence turns ordinary network behaviour into duplicate orders, double charges and mysterious states.
Idempotency is a business feature
An idempotent operation can be retried without creating a second business effect: five identical requests still create one order. For payment, order and booking flows, that usually means accepting an idempotency key — an identifier the caller sends and reuses on every retry. The server remembers the result of the first successful attempt and returns that again.
Do not confuse idempotency with simply returning 200 twice. The important question is whether the business action happened once.
Timeouts need a next state
A timeout means 'I stopped waiting.' It does not necessarily mean 'the other system failed.' That distinction should shape the states you let an order or a payment sit in.
After a payment timeout, the product may need a pending state, a status check against the provider, or a webhook confirmation. Any of those beats trying again immediately and hoping the provider is also confused.
Webhooks should be assumed to repeat
A webhook is a call the other system makes to your API when something happens at their end. They retry those calls, because delivery is never certain. Your handler should expect duplicates, events arriving out of order, and the occasional event about data you cannot see yet through their API.
Store the event identifiers so a repeat can be recognised. Keep handlers safe to run twice. Where you can, acknowledge quickly and do the slow work separately.
Use retry budgets, not infinite optimism
Retries are useful for transient failure. They are terrible for permanent failure. A malformed payload does not become valid after 37 attempts.
Classify errors first. For the failures likely to recover, retry a limited number of times, waiting a little longer — and slightly randomly — after each attempt. When the retries run out, park the work somewhere a person can pick it up, with enough context to understand it.
Make consistency visible to users
Some workflows are eventually consistent: the result becomes correct in a moment, but not the instant the user clicks. The interface should admit that rather than pretend every action is instant. 'Payment received — confirming your booking' is better than showing success before the booking exists or failure while the payment actually succeeded.
Good design can make the truth feel calm rather than broken.
The boring rulebook
Use request IDs. Use idempotency keys where business actions can repeat. Make webhook handlers safe to run twice. Set explicit timeouts. Distinguish retryable from permanent failures. Check back on external states you are unsure about. Log how long each dependency took, and the identifiers you will need to search for later.
None of this is glamorous. That is why it is often missing from the first version — and why it becomes extremely glamorous during an incident.