[Feature request] Add extra ttl argument for counter reset after last deduplicated message about rabbitmq-message-deduplication HOT 6 CLOSED

Hello,

can you please explain better what is you are trying to achieve?

Why just not using longer TTL values?

Please note the x-cache-ttl is not a mandatory field. If you omit it, messages with the same x-deduplication-header will be deduplicated virtually forever(*).

(*) if the cache fills up, then some old entry will be removed to free space to new ones.

from rabbitmq-message-deduplication.

Hello,
The use case is filtering the messages, sent between two of our components - REST API and Core, via RabbitMQ. The server, which makes requests to my REST API component has a bug and sometimes it makes a high flow of requests with same data (usually 1-2 per second), which I want to filter by your very useful plugin. Unfortunately fixing the bug of that server, which makes the requests, is not an option for now. The same request issue is rare, but when it happen, the period is usually different - sometimes 1 min, sometimes 5 min.

I can't use a longer TTL value, because for instance if I set the TTL for 5 mins and the flow could last for only 1 minute. Then if the user creates real event and the server makes a new request (again with same value), the message will be deduplicated.

My plan was to setup the TTL for 60 seconds and every next same message to reset the period for new 60 minutes, to make sure that we will not miss actual user actions. The predefined period is 60 seconds, because usually the creation of new event by a user takes at least 60 seconds.

I hope I explained well my use case.

Greetings,
Stoyan

from rabbitmq-message-deduplication.

I am not sure I understand correctly.

If you receive a message at the second 0 you will forward it with a TTL of 60s. If you receive another message at the second 12, you want to reset such TTL. This means the message will be deduplicated for at least 72 seconds. Let's say that you receive another duplicated at the second 24. This makes 60 + 12 + 24 = 96s.

You know a User takes minimum 60 seconds to generate a request and you don't want to loose user data.

Yet, with the above strategy, you will indeed loose a request if made in, say for example, 72 seconds.

In your above example you also mention "60 minutes", I guess you meant 60 seconds or am I understanding incorrectly?

from rabbitmq-message-deduplication.

Yes, you have understood it correctly.
The bug, caused by the external server is rare, but when it happens it's a disaster so I would like to protect our server and the users by deduplicating the messages. The combination of the bug and a new user request with same message value is for the TTL period would be a quite rare event, so I would take the risk.

The "60 minutes" is mistake, I meant 60 seconds.

If you decide that such feature would be inappropriate, please close the issue. Even without this feature your plugin is very useful for me and I will use it with TTL period of 60 seconds.

Thanks for your efforts and time!

from rabbitmq-message-deduplication.

The reason why I am reluctant implementing this feature is that it would be hard to control by the user. A message recurring frequently enough could be virtually deduplicated forever and the thing would easily go unnoticed.

Let's imagine a malicious actor which would publish the same request to your system every 50 seconds for example. It would literally deny such request for an undefined amount of time. If the load on the system is high enough, it would be hard for you to detect such occurrence.

A middle ground I would recommend you would be to set the TTL to a mid-value (for example 3 minutes) and notify the users that to prevent the overloading of systems it must take 3 minutes between a request and the next one with equal data.

An even simpler solution would be adding the request timestamp to the data the server sends to your REST interface. Duplicated requests would have the same timestamp and would be very simple to detect and handle properly. This would obviously imply you can intervene on the offending server code base.

from rabbitmq-message-deduplication.

I'm closing the issue. Thank you for the tips! :)

from rabbitmq-message-deduplication.