Comments (12)
Hi @zek
You mean that same event could be published from different sources at the same time?
from rabbitevents.
Actually no. I need to block tasks within the same group and make sure they processed in order.
(Group key included in task data)
Let me tell you my use case.
I have a microservice which collects data and pushes this data to rabbitmq and data handled by queue workers. There are 400 instances of this microservice.
For scaling purposes I have multiple queue workers on multiple machines.
In my case task order is very important.
I'm afraid if task order may be mixed up.
Right now I use cache lock feature and block task as following.
Cache::lock('group:'.$groupKey, 10)->block(...)
So multiple events may pushed at the same second or even milisecond and I need to make sure they're executed in order by multiple queue workers.
from rabbitevents.
The order issue is clear for me. And I still don't realise how to solve this without the scan of the full list of current tasks. It should be some dispatcher which will be the only point to return messages or so.
But this isn't the issue of the Uniqueness.
from rabbitevents.
I'm truly believe that the Rabbitevents works in FIFO principle. So if many listeners are listening to one queue messages will be handled in the right order.
from rabbitevents.
I'm truly believe that the Rabbitevents works in FIFO principle. So if many listeners are listening to one queue messages will be handled in the right order.
Indeed but in my case each task within the same group must wait for previous tasks to be completed.
I guess I mention wrong Interface, should be WithoutOverlapping
.
https://laravel.com/docs/8.x/queues#preventing-job-overlaps
from rabbitevents.
Even Cache::lock
doesn't solve my issue :/
I have 10 workers and have 2 events that queued at the same second with the same group key.
Event A (group X) queued at 16:14:46.7181
Event B (group X) queued at 16:14:46.7237
I just need Event B waits for Event A to be completed. But in my case Event B processed before Event A.
from rabbitevents.
I came up with a feature in RabbitMQ. Might be a solution
https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_consistent_hash_exchange
from rabbitevents.
Hi @zek
Thank you, I'll take a look on it.
from rabbitevents.
Regarding your issue: how a listener should work? They both has taken their messages in the correct order but for some reason (Database load, server etc.) second listener could finish it's job faster. But this is the problematic which could be solved by the atomic processing.
I see that an issue you have could be solved like this:
- worker has got a message
- worker creates a mutex
- no new messages could be taken until a mutex exists
from rabbitevents.
I've tried atomic lock but it didn't solve my issue. I've read a few blog posts and seems like this is a common problem in tech firms such as e-commerce firms. They mostly points to rabbitmq_consistent_hash_exchange.
Regarding to consistent hash exchange docs,
Tasks in the same group goes to same worker therefore worker must finish the task 1 then it can receive task 2.
That means none of the listeners should never get the task 2 even if queue has only task 2.
It's really easy to implement however it requires routing key as integer. But rabbitevents uses "routing-key" as an event name. So requires to get event-name from the data or properties.
from rabbitevents.
the routing key - is the part of a queue name too
from rabbitevents.
Close
from rabbitevents.
Related Issues (20)
- Question about graceful restarting. HOT 5
- Write more information on Exception Ocurred
- When the listener starts without an event name then with large numbers of events it returns an error HOT 3
- Getting "PhpAmqpLib\Exception\AMQPInvalidArgumentException: String too long" on message delay when I start listener without event names HOT 3
- Give ability to explicitly set queue name in `rabbitevents:listen` command HOT 2
- Use amqp-ext for the faster work
- Do payloads have to be JSON? HOT 8
- [QUESTION]: Deploying to productioin HOT 5
- Support php 8 HOT 3
- Config not properly read HOT 2
- sync or dummy driver for testing environment HOT 5
- exception Illuminate\Container\EntryNotFoundException when try using command HOT 4
- Configuration file is ignored
- Add protobuf as the transporting content-type
- Add option --message-ttl HOT 2
- Handlers could be executed twice after a Message release
- Worker stoped after 3 tries with error: "[INFO] Worker has been stopped with the status code 1"
- Connection option is not working HOT 1
- Horizon Support HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitevents.