All Questions
Tagged with message-queue design
25 questions
1
vote
1
answer
149
views
Background thread processing vs queue based processing for relatively short tasks (max 30-40 seconds)
I need suggestion / recommendation on the design approaches mentioned below.
UseCase: I have a usecase where a client uses my system to generate some recommendations. Now, these recommendations are ...
- 19
0
votes
1
answer
85
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 119
0
votes
0
answers
98
views
Event based update of In memory Hashmap in multiple pods of kubernetes
I have a service (S1) that is deployed on kubernetes having multiple replicas (pods).
S1 requires some data from database (Cosmos Cassandra Api) for calculation.
This data is constantly updated by ...
- 101
3
votes
4
answers
424
views
Organize and maintain a lot of cron jobs
I am working on a fairly large project written in PHP (Yii2) in which we are increasingly using cron for background tasks, such as generating caches, reports, etc.
We have started to move the hourly ...
- 39
0
votes
2
answers
262
views
Is it a good/bad practice for a service to send events to itself via an external queue?
Imagine there is an Order entity that has a lot of states and conditional transitions between them. At the moment it is stored in a relational DB.
There are multiple consumers of Orders that perform a ...
- 307
1
vote
0
answers
109
views
How to have only one consumer handle a long running task at a given point?
I need to design an application which makes requests to an external API. External API has a hierarchy of entities: User which contains ListOfItemss which contain Items. I need to get data on all Items ...
- 167
0
votes
0
answers
54
views
How to handle maintenance to your queuing server (e.g. rabbitMQ)
I'm new to queuing and looking into options that could be done operationally. I'm sorry if I have any conceptual misunderstandings related to queuing.
I watched this video where the presenter made a ...
- 311
0
votes
0
answers
38
views
Design a Program which Forwards Work Pieces to Corresponding Modules
I am working on a program that will listen to a queue of incoming message and will process each message differently depending on the metadata of that message. The messages must be processed in order ...
- 101
1
vote
1
answer
1k
views
Handling multiple events in Event Driven Architecture
Let's say I have
Service A that publishes Message 1 to be processed by Consumer A
The same service A that publishes Message 2 to be processed by Consumer B
I need to make sure that consumer B ...
- 27
1
vote
1
answer
352
views
Distributed message queue, propagating queue leader/follower information
I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design:
I have the following components:
FrontEnd: request ...
- 249
5
votes
2
answers
5k
views
Architecture for message processing with scheduling, at scale
I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
- 153
1
vote
1
answer
100
views
Approach to handle network breakdowns in application dealing with database?
I have an application that writes data to database. In case of network breakdowns, the connection to database gets lost & my db queries freezes the whole application.
So, in order to make it ...
- 111
8
votes
3
answers
6k
views
Message Queues and Multiple Subscribers
My team is discussing using RabbitMQ for microservice to microservice communication (among other things). We are also supporting a SAP backend that will be pushing data to us in a method undecided at ...
- 570
0
votes
1
answer
141
views
Where should I store statistics about sent emails?
my application sends a batch of custom emails every day using mailgun (an email API service). I have a producer worker scheduled to run daily, and all does is query the database for the unique ...
- 109
2
votes
2
answers
109
views
AMQP broker and happen-before guarantees
I am trying to figure out how an AMQP broker, such as RabbitMQ, can be integrated in our architecture. This should allow to easily scale horizontally in the future.
For the sake of simplicity, let me ...
- 23