Retries, DLQs and backoff

Backoff with jitter

Retrying immediately on failure turns a struggling endpoint into a dead one by overwhelming it with load. Exponential backoff spaces attempts out like 1s, 2s, 4s, 8s, and so on for efficiency. Jitter randomises each delay so that thousands of failed deliveries do not all retry in the same second. This simple technique significantly improves the stability of both the producer and the consumer during outages.

function delay(attempt, base = 1000, cap = 3600_000) {
  const exp = Math.min(cap, base * 2 ** attempt);
  return Math.random() * exp; // "full jitter"
}

Retry budgets

Decide how long an event stays alive before it is considered permanently undeliverable for your app. A typical producer schedule retries over roughly three days with around fifteen attempts to ensure delivery. After that, the event moves to a dead-letter queue (DLQ) for manual inspection by operators. This prevents transient issues from losing data while ensuring your system doesn't spend resources on dead endpoints.

Consumers retry too

When your consumer pulls work from its own queue and a downstream call fails, the same rules apply. Keep the retry state on the job object and make your handlers idempotent for safety. Route poison messages to a DLQ instead of letting them block the queue head for other messages. Proper error handling in the consumer is just as important as it is in the producer.

Handling poison messages

A poison message is one that causes your consumer to crash or fail every time it is tried. These messages can be caused by bugs in your code or unexpected data shapes from the API. Detecting these early and moving them to a DLQ is vital for maintaining the flow of other messages. You should have a clear process for investigating and replaying messages from the DLQ once fixed.

Tuning your Retry Parameters

The optimal retry strategy depends on the nature of your events and the cost of delay. For time-sensitive events like OTPs, you might want shorter intervals and a smaller total retry window. For background synchronization, a longer window with more spaced-out attempts is usually more appropriate. Experiment with different parameters in your staging environment to find the right balance for your specific use case.