it seems when the cache is overflowed some problem happens about rabbitmq-message-deduplication HOT 4 CLOSED

Hello,

when the cache overflows, old elements are dropped to make place to the new ones. Which type of problem are you experiencing?

The reason why the cache has a maximum size is to protect the broker from running out of memory. Dropping old elements to make space to new ones is a pretty common implementation. Do you have something else in mind?

Regarding the second question, the quick answer is "nothing" :). The message de-duplication is performed on the exchange and not on the queue. An exchange could be bound to multiple queues or other exchanges. Therefore, flushing the cache when one of the queues is purged would affect all other bindings.

I am planning in the future to add queue level de-duplication as well. I first implemented de-duplication on the exchange because allows a more flexible design.

You can, for example, bind a de-duplication exchange to a prioritized queue and get at once priority and de-duplication support. A single exchange could de-duplicate messages for multiple queues etc..

from rabbitmq-message-deduplication.

Noxdafox, thank you for quick response

When the cache overflows old elements are dropped in a FIFO manner? (I mean, first element that has been saved into cache will be dropped first ?)

Is there any way to flush the cache ?

from rabbitmq-message-deduplication.

Unfortunately, the dropped elements are randomly chosen. This is a limitation of the implementation due to the fact the mnesia DB used for the cache is not allowing an easy FIFO implementation (some note in regards here).

If you think a FIFO is required let's open a separate ticket and I'll look closer into that.

A possible mitigation could be storing the cache on disk instead of memory (issue #2). This would allow much bigger cache size reducing the risk of overflow.

Also setting TTL for the cached elements might help. TTLs allow to de-duplicate a message for a given amount of time. After that time, the element is removed making space for the cache.

To flush the cache, you could destroy and recreate the exchange. This will actually drop the entire cache content. You shall not worry about the bound queues as they will keep their messages.

from rabbitmq-message-deduplication.

@rusyasoft the new release supports caching on disk which should allow much larger cache size. This should mitigate your issue.

I will close this issue. If you need more specific functionalities, feel free to open new issues for them.

from rabbitmq-message-deduplication.