API Integrations Masterclass · Chapter 3 · 14 min
Idempotency keys done right
Make retries safe: key generation, storage windows and the 409 you should actually expect.
Why networks lie
A request can succeed on the server and still look like a failure to the client due to timeout. The TCP connection might drop after the provider committed the message but before the response reached you. If you retry blindly, the recipient gets two messages, which is annoying and potentially expensive. Idempotency keys resolve this dilemma by making repeated requests safe for the system to process.
The Idempotency Contract
You attach a unique Idempotency-Key header to a mutating request on the ub3r API for safety. The server stores the key alongside the response for a window of usually twenty-four hours. A second request with the same key and the same body returns the stored response immediately. The same key with a different body returns 409 Conflict to signal a bug in your logic.
−async function sendOtp(user) {for (let attempt = 0; attempt < 3; attempt++) {− const key = crypto.randomUUID();try {return await api.post("/messages", body(user), { key });} catch (e) { await sleep(2 ** attempt * 500); }}}
Generating keys correctly
The key must be derived from the *intent* of the business action, not the *attempt*. Generating a fresh UUID inside the retry loop defeats the mechanism entirely because the server sees a new request. Create it once when the business event happens and persist it with the job record. This ensures that even if your worker crashes, the next worker to pick up the task will use the same key.
Server-side storage logic
If you are building the API side, store the key and a hash of the request body. The lock matters because two concurrent requests with the same key must not both execute at once. Take a row-level lock or use an atomic insert-if-absent to handle concurrent attempts gracefully. This prevents race conditions that could lead to double-sending even with idempotency keys in place.
interface IdempotencyRecord {
key: string;
request_hash: string;
response_code: number;
response_body: string;
locked_at: Date | null;
}Performance Considerations
Storing idempotency keys adds a small amount of overhead to every write request in your system. You should ensure that the database table used for these keys is properly indexed on the key column. Using a fast key-value store like Redis for temporary idempotency state can also improve performance. Balance the need for absolute reliability with the latency requirements of your application to find the right solution.
Discussion (4)
The callout about alerting on queue age instead of depth changed how we page people. Thank you.
Same experience here. Per-route metrics were the only way we found it.
The distinction between 'sent' and 'delivered' finally clicked for me here. We had dashboards treating them as the same thing for a year.
Would love a follow-up on how this behaves under carrier throttling. We see very different numbers per route.