A customer clicks "pay," the request reaches your server, the charge succeeds, and then the connection drops before the response gets back. The client, seeing no answer, retries. Now the customer is charged twice. This exact failure is why idempotency exists, and why payment companies like Stripe built their APIs around it.
Key Takeaways
- An operation is idempotent if doing it many times has the same effect as doing it once (Stripe on idempotency).
- Networks drop and clients retry, so without idempotency, retries cause double charges and duplicate side effects (Stripe).
- The standard fix is an idempotency key: a unique ID the server uses to recognize a retry (Stripe API).
- "Exactly-once" is really at-least-once delivery plus idempotent processing (idempotent APIs).
The Problem
The trouble is a direct consequence of the distributed-computing fallacies: the network is not reliable, so requests get lost and clients retry. When a client sends a payment and the connection drops after the server processed it but before the response arrives, the client genuinely does not know whether it worked (Stripe on idempotency). Its only safe-feeling option is to retry, and if the server treats the retry as a fresh request, the customer pays twice. The same problem hits any operation with side effects: sending an email, creating an order, provisioning a server.
The Idempotency Key
The clean solution is to make the operation safe to repeat. The client generates a unique idempotency key, a random UUID with enough entropy to avoid collisions, and attaches it to the request (Stripe API reference). The first time the server sees that key, it does the work and stores the result against the key, in the same transaction. If a retry arrives with the same key, the server recognizes it and returns the stored result instead of doing the work again. The customer is charged once, and the retry is harmless. Stripe expires these keys after 24 hours, which is plenty for retry storms.
| Without idempotency | With an idempotency key |
|---|---|
| Retry re-runs the operation | Retry returns the stored result |
| Customer charged twice | Charged exactly once |
| Duplicate orders, emails | Duplicates suppressed |
| "Did it work?" is dangerous | Retrying is always safe |
The Truth About "Exactly-Once"
Vendors love to advertise "exactly-once delivery," and strictly speaking it's a bit of a myth (idempotent APIs). What you actually get is at-least-once delivery (the system guarantees a message arrives one or more times) combined with idempotent processing (your application guarantees that handling the same message twice has the same effect as once). Put those together and the observable result is exactly-once. The delivery guarantee lives in the infrastructure; the idempotency has to live in your application. Teams that assume the broker gives them exactly-once, and skip the idempotency, get duplicates.
A Concrete Version
A startup's order service works fine until a traffic spike causes timeouts. Clients retry the timed-out requests, and because the order endpoint isn't idempotent, a batch of customers get duplicate orders, duplicate charges, and duplicate confirmation emails. Support drowns, refunds go out, trust takes a hit. The fix is small in hindsight: an idempotency key on the order request plus a unique database constraint, so a retry with the same key returns the original order instead of creating a new one. An afternoon of work would have prevented the whole incident.
The Honest Counterpoint
Not everything needs an idempotency key, and adding them everywhere is its own tax. Read-only operations (a plain GET) are naturally idempotent already. Some operations are safe to repeat by their nature (setting a value to X, rather than incrementing it). The machinery matters most for operations with real, non-repeatable side effects: money, provisioning, notifications. The skill is spotting which endpoints are dangerous to retry and protecting those, rather than bolting keys onto everything or, worse, onto nothing.
What This Means for Teams
Idempotency is one of those concepts that separates engineers who've run payment or ordering systems in production from those who haven't. The reflex to ask "what happens if this gets called twice?" on every side-effecting operation is hard-won, and it directly prevents the kind of incident that costs real money and customer trust. It's part of the reliability discipline, alongside safe retries against the network's unreliability, that we look for in senior engineers. See available engineers.
Frequently Asked Questions
What does idempotent mean?
An operation is idempotent if performing it multiple times has the same effect as performing it once. That makes it safe to retry, which matters because networks drop requests and clients retry.
How does an idempotency key work?
The client sends a unique key with the request. The server does the work once and stores the result against that key; if a retry arrives with the same key, it returns the stored result instead of repeating the operation.
Isn't "exactly-once delivery" a solved thing?
Not on its own. What you get is at-least-once delivery plus idempotent processing in your application. The infrastructure guarantees the message arrives; you guarantee handling it twice is harmless. Together they look exactly-once.
Does every endpoint need idempotency?
No. Reads are naturally idempotent, and some writes are safe to repeat by nature. Focus the machinery on operations with real side effects that can't be safely repeated: payments, orders, provisioning, notifications.
The Bottom Line
Networks drop requests, clients retry, and without protection those retries double-charge customers and duplicate orders. Idempotency, most simply an idempotency key plus a database constraint, makes a repeated operation harmless. Treat "exactly-once" as at-least-once delivery plus your own idempotent processing, and protect every side-effecting endpoint that would hurt if it ran twice.
Roberto Espinoza is CEO of Ruzora, which helps US startups hire pre-vetted senior LATAM engineers, with a vetted shortlist in 72 hours. See available engineers.
