Questions tagged [message-queue]
A message queue is a service that receives events or messages from producers and provides them to consumers.
259 questions
3
votes
3
answers
415
views
Design question for exactly-once processing in a message-driven system using a unique ID
To achieve exactly-once processing where messages are consumed from a queue with at-least-once delivery, many sources (e.g. here and here and here) suggest attaching a unique ID to messages in the ...
- 139
0
votes
0
answers
143
views
How does each application replica read a unique message from the message broker
In RabbitMQ streams or Kafka, messages are not deleted after being consumed. If you have a consumer application replicated across multiple Kubernetes pods, how can you ensure that each pod picks up a ...
- 9
0
votes
5
answers
138
views
How can I measure which messages are causing queue backups?
Consider the following multi-producer, multi-consumer system:
For simplicity, assume that we only have one queue
Messages are dequeued on a first-in-first-out basis
Oldest-message-pickup-latency has ...
- 119
1
vote
1
answer
149
views
Background thread processing vs queue based processing for relatively short tasks (max 30-40 seconds)
I need suggestion / recommendation on the design approaches mentioned below.
UseCase: I have a usecase where a client uses my system to generate some recommendations. Now, these recommendations are ...
- 19
3
votes
3
answers
212
views
Handling rate limits / delays in consumers without affecting performance of other operations
I have a producer that generates a batch job that consists of multiple operations (approx. 100 - 10000). These operations can be processed in any order, ideally as fast as possible.
The processing of ...
- 141
0
votes
1
answer
108
views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
7
votes
5
answers
402
views
What can I do to get a message processor to slow down the rate of writes that it is making to a database?
We have this architecture:
queue -> message processor (horizontal scaling) -> RDBMS
Sometimes external systems dump 10k messages onto the queue and the message processor of course dutifully ...
- 229
0
votes
3
answers
2k
views
Backend to client communication: WebSocket or Message Queue?
I have a hobby project, for simplicity imagine a poker game, consisting of a backend (Spring) and a client application (Flutter) where the client can join rooms in which he can take a seat. The client ...
- 9
2
votes
4
answers
232
views
Ensure concurrent processing based on a message key
I've been wondering about the following: Say you have an async message/task and you want that message only processed one at the time. for example: Only process one order for each customer at the time. ...
- 174
0
votes
2
answers
277
views
How to implement event payload isolation in an event driven architecture?
I have an event, let's say order.placed The payload consists of some general information about the order and 3 confidential fields. For example part of the message should only be visible to consumer A,...
- 129
0
votes
1
answer
85
views
Design a sequential processing of records
About 10 to 15 records are processed per day with the time interval of 5 minutes between each record.
System A inserts a record in DB and sends id of that record to active mq.
System B Listener ...
- 119
1
vote
0
answers
117
views
golang: pattern for handling message queues? Are named functions anti-idiomatic somehow?
Had a discussion today in how to implement services that work with messages coming in from event queues. We call these services processors. One of us argues for using several functions, while the ...
2
votes
0
answers
118
views
parent correlation id in message with rabbit queue and open telemetry
I have a rabbit queue that I use to communicate between applications.
I have a chain of three message.
App 1 generate and push a message in rabbit(message 1)
App 2 catch the message and generate ...
- 307
3
votes
2
answers
420
views
Can multiple producers cause events with the same key to be enqueued out of order within a specific partition of a topic?
I'm aiming to create a simple event-driven system where each microservice operates with its own database. The idea is to share database changes across microservices through events. To ensure proper ...
0
votes
1
answer
127
views
Is RabbitMQ suitable for allowing SaaS customers to subscribe to events?
I work on a SaaS system which is generally a single-tenant environment. Customers (whom the SaaS instances are for) would like a way to subscribe to events, such as updates of certain data entities. ...
- 101