RabbitMQ retries on Node.JS
In this article I am going to show you how to implement a robust retry topology using node.js and amqplib npm library. The solution is inspired by RabbitMqDlxDelayStrategy.php strategy from Enqueue library.
First of all, here is our topology:
Lets have a brief overview on how retrying works.
1) Imagine something goes wrong in a comments queue consumer. Say, there is an exception or reject.
2) We acknowledge an original message, create its clone and send it to TTL EXCHANGE with routing key = retry-1
3) Message goes to corresponding TTL queue — comments-retry-1-30s (ttl queue1 30s on image above) with TTL = 30s
4) This queue does nothing, even doesn’t have any consumers connected. The message is expired there and therefore moved to corresponding DLX (Dead Letter Exchange).
5) DLX is configured to send all messages to our payload queue comments, so we have a second try now.
6) If it fails again, do repeat, but this time publish the cloned message to comments-retry-2-10m queue.
The consumer might lost the connection with RabbitMQ (consumer crashed/killed or connection issues). In this case we should do next:
- RabbitMQ notices that connection has been lost and therefore return the message to the queue, marking it as redelivered.
You can read more about redelivery pit falls in the article
2) When the consumer recovers it handles that message again. Now it checks the redelivered flag. If it’s true it DOSN’T EVEN TRY TO HANDLE IT - just sends it to retry flow. After it goes through the whole retry flow, the message will come back without redelivered mark and will be handled as always.
Why don’t we handle redelivered messages and just send them to retry? Because we don’t know for sure what caused that redelivery. What if this message is broken (say we try to handle an old message with a new code) and each time it is handled a consumer crashes? So instead of having endless loop of crashes we have maximum 3 crashes and then the message is erased.
Each work (payload) queue (like comments and so) has its own set of retry queues and exchanges: 1 TTLX, 3 ttl queues and DLX
Implementation
First of all let’s create our comments queue and set up retry queues and exchanges:
Next, if we have a failed message we can send it to retry loop like this:
Keep in mind that we send to retry loop failed messages for both reasons: failing inside handler (because of reject OR exception) or failing because of connection issues (in such case AMQP will mark a message as redelivered). This is what happens in sendMsgToRetry: we acknowledge an original message, determine which attempt it is and send a new message (clone) to the next retry loop.
You can find a full implementation with descriptive comments here.
Also, you can visit our corporate blog to learn more about up-to-date IT trends and download some useful insights.
