Return to Answer

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database based queue system for middleware solution. I agree with the article. A database is not the right tool for queue based system, but it may be good enough for you.

Queue based system like RabbitMq are built around massive scale on moderate hardware. Their architecture is able to achieve this by avoiding processes which make ACID compliant database system slow by their nature. Since a message bus only needs to ensure a message is stored and successfully processed, it doesn't need to bother with locking and writing transaction logs. Both of these concepts are absolutely required for an ACID system but are often a cause of contention.

Performance wise it comes down to, you have a SQL table. Lots of reads and lots of writes. Both require some sort of locking to update rows, pages and indexes. Your polling mechanisms is constantly locking an index to do look ups on it. This prevents writes from happening, at best they are queued. The code doing the processing is also locking to update the status on the queue as they complete or fail. Yes, you can do query optimization after optimization to get this to work, or you can use a system specifically designed for the work load you are asking. A RabbitMq eats up this type of workload without even breaking a sweat on top of that you get to save your database from the workload giving it more room to scale doing other things.

Another thing to consider is most queue system typically do not use a polling technique (some allow for HTTP, but recommend to avoid using for the receive side). RabbitMq uses network protocols specifically designed for message buses such as AMPQ.

Edit: Adding usage case.

The way I have used Rabbit is I have had an API endpoint which accepts a change which requires a heavily used database table. This table is under constant contention and at times will not be able to save a change in a timely fashion from the API. What I do instead is write the change request to a queue and then have a service which handles these messages as they are able. If database contention occurs the queue simply grows and message processing is delayed. Typically processing time down in the 14ms range, but in times of high contention we get up to 2-3 seconds.

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely that not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database-based queue system for a middleware solution. I agree with the article. A database is not the right tool for queue based system, but it may be good enough for you.

Queue based system like RabbitMq are built around massive scale on moderate hardware. Their architecture is able to achieve this by avoiding processes which make ACID compliant database system slow by their nature. Since a message bus only needs to ensure a message is stored and successfully processed, it doesn't need to bother with locking and writing transaction logs. Both of these concepts are absolutely required for an ACID system but are often a cause of contention.

Performance-wise it comes down to: you have an SQL table. Lots of reads and lots of writes. Both require some sort of locking to update rows, pages and indexes. Your polling mechanism is constantly locking an index to do lookups on it. This prevents writes from happening; at best they are queued. The code doing the processing is also locking to update the status on the queue as they complete or fail. Yes, you can do query optimization after optimization to get this to work, or you can use a system specifically designed for the work load you are asking. A RabbitMq eats up this type of workload without even breaking a sweat; on top of that, you get to save your database from the workload giving it more room to scale doing other things.

Another thing to consider is most queue systems typically do not use a polling technique (some allow for HTTP, but recommend to avoid using for the receive side). RabbitMq uses network protocols specifically designed for message buses such as AMPQ.

Edit: Adding usage case.

The way I have used Rabbit is I have had an API endpoint which accepts a change which requires a heavily used database table. This table is under constant contention and at times will not be able to save a change in a timely fashion from the API. What I do instead is write the change request to a queue and then have a service which handles these messages as they are able. If database contention occurs the queue simply grows and message processing is delayed. Typically processing time down in the 14ms range, but in times of high contention we get up to 2-3 seconds.

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database based queue system for middleware solution. I agree with the article. A database is not the right tool for queue based system, but it may be good enough for you.

Queue based system like RabbitMq are built around massive scale on moderate hardware. Their architecture is able to achieve this by avoiding processes which make ACID compliant database system slow by their nature. Since a message bus only needs to ensure a message is stored and successfully processed, it doesn't need to bother with locking and writing transaction logs. Both of these concepts are absolutely required for an ACID system but are often a cause of contention.

Performance wise it comes down to, you have a SQL table. Lots of reads and lots of writes. Both require some sort of locking to update rows, pages and indexes. Your polling mechanisms is constantly locking an index to do look ups on it. This prevents writes from happening, at best they are queued. The code doing the processing is also locking to update the status on the queue as they complete or fail. Yes, you can do query optimization after optimization to get this to work, or you can use a system specifically designed for the work load you are asking. A RabbitMq eats up this type of workload without even breaking a sweat on top of that you get to save your database from the workload giving it more room to scale doing other things.

Another thing to consider is most queue system typically do not use a polling technique (some allow for HTTP, but recommend to avoid using for the receive side). RabbitMq uses network protocols specifically designed for message buses such as AMPQ.

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database based queue system for middleware solution. I agree with the article. A database is not the right tool for queue based system, but it may be good enough for you.

Queue based system like RabbitMq are built around massive scale on moderate hardware. Their architecture is able to achieve this by avoiding processes which make ACID compliant database system slow by their nature. Since a message bus only needs to ensure a message is stored and successfully processed, it doesn't need to bother with locking and writing transaction logs. Both of these concepts are absolutely required for an ACID system but are often a cause of contention.

Performance wise it comes down to, you have a SQL table. Lots of reads and lots of writes. Both require some sort of locking to update rows, pages and indexes. Your polling mechanisms is constantly locking an index to do look ups on it. This prevents writes from happening, at best they are queued. The code doing the processing is also locking to update the status on the queue as they complete or fail. Yes, you can do query optimization after optimization to get this to work, or you can use a system specifically designed for the work load you are asking. A RabbitMq eats up this type of workload without even breaking a sweat on top of that you get to save your database from the workload giving it more room to scale doing other things.

Another thing to consider is most queue system typically do not use a polling technique (some allow for HTTP, but recommend to avoid using for the receive side). RabbitMq uses network protocols specifically designed for message buses such as AMPQ.

Edit: Adding usage case.

The way I have used Rabbit is I have had an API endpoint which accepts a change which requires a heavily used database table. This table is under constant contention and at times will not be able to save a change in a timely fashion from the API. What I do instead is write the change request to a queue and then have a service which handles these messages as they are able. If database contention occurs the queue simply grows and message processing is delayed. Typically processing time down in the 14ms range, but in times of high contention we get up to 2-3 seconds.

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database based queue system for middleware solution. I agree with the article. A database is not the right tool for queue based system, but it may be good enough for you.

Queue based system like RabbitMq are built around massive scale on moderate hardware. Their architecture is able to achieve this by avoiding processes which make ACID compliant database system slow by their nature. Since a message bus only needs to ensure a message is stored and successfully processed, it doesn't need to bother with locking and writing transaction logs. Both of these concepts are absolutely required for an ACID system but are often a cause of contention.

Another thing to consider is most queue system typically do not use a polling technique (some allow for HTTP, but recommend to avoid using for the receive side). RabbitMq uses network protocols specifically designed for message buses such as AMPQ.

If you are building a simple application with low traffic, there is something to be said about keeping another component out of your system. It is very likely not using a message bus is the right answer for you. However, I would suggest building your system in a way you could swap out the database based queue system for middleware solution. I agree with the article. A database is not the right tool for queue based system, but it may be good enough for you.

Queue based system like RabbitMq are built around massive scale on moderate hardware. Their architecture is able to achieve this by avoiding processes which make ACID compliant database system slow by their nature. Since a message bus only needs to ensure a message is stored and successfully processed, it doesn't need to bother with locking and writing transaction logs. Both of these concepts are absolutely required for an ACID system but are often a cause of contention.

Performance wise it comes down to, you have a SQL table. Lots of reads and lots of writes. Both require some sort of locking to update rows, pages and indexes. Your polling mechanisms is constantly locking an index to do look ups on it. This prevents writes from happening, at best they are queued. The code doing the processing is also locking to update the status on the queue as they complete or fail. Yes, you can do query optimization after optimization to get this to work, or you can use a system specifically designed for the work load you are asking. A RabbitMq eats up this type of workload without even breaking a sweat on top of that you get to save your database from the workload giving it more room to scale doing other things.

Another thing to consider is most queue system typically do not use a polling technique (some allow for HTTP, but recommend to avoid using for the receive side). RabbitMq uses network protocols specifically designed for message buses such as AMPQ.