Message queues are a great way to distribute data across microservices and for asynchronous communication. Our team uses queues to handle the following:
Queues are inevitable if you want highly-scalable applications and a fault tolerant infrastructure, but they can also be potentially disastrous.
In this post, I’ll outline the top 3 most common pitfalls based on our experience. Keep in mind that we work with AWS and this article is from SQS perspective. If you are using other tools, the situation might be different.
AWS promises that Standard queues provide “best-effort ordering” which means that messages may occasionally be delivered in a different order than the one they were originally sent in. In many use cases that’s not a problem, but if you application depends on persistence (e.g. a message is dependent on another message’s delivery), you will eventually end up in trouble.
Two options here:
If it’s possible, always stick with option 1 as it would ensure high throughput.
Standard SQS queues guarantee at least one delivery which means that from time to time a message will be delivered twice. This occurs because of AWS’s severe parallelization for the sake of high-availability. It may sound a little innocent at first if you are using queues to send emails and convert documents, but now imagine a customer being charged twice for the same order.
Depending on the situation there are couple scenarios that might be appropriate:
When a messages is sent to the queue, the submitter would always receive a success response for the message submission, but never a response that indicates whether the message has been successfully processed or not. That can be limiting, but at the end, it’s part of the asynchronous communication paradigm.
If you need a delivery confirmation, consider the following potential solutions: