Is it possible to add max_water_mark to the queue? about yaque HOT 7 CLOSED

Sorry for the late response. Working on other problems atm.

So... the current implementation is unbounded. You sugges being possible to limit the size of the queue? Similar to what happens to the channels in std.

We would need two methods:

1. A fn try_send which fails on queue lmit and
2. An async fn send to wait for the queue to clear.

Well... that is perfectly reasonable! Could you share details on your use case for context? I developed yaque to unload insane amounts of incoming data from memory and avoid OOM. So, a bounded channel didn't make much sense.

from yaque.

So.... @hh9527 I have been thinking about this on the back burner... something like this is indeed needed. However, I will not be able to give fine grained control on the number of elements, since that would require lots of sync with the Receiver, which looked suuuuper tricky.

On the other side, something like limiting the space used by controlling the creation of new segments is "easy" to do. In fact, that is the strategy used in queues such as Postgres' WAL log. I will start working on that today already. Got some time to spare!

from yaque.

Man, what reaaaly made my mind in this feature was spending up to 2am fixing pglogical issues exaclty because it didn't implement something like this. Thank God is looks like they solved it in Postgresql 13!

from yaque.

Merged pull request. I'll leave it there soaking before publishing....

from yaque.

@hh9527 are the mechanisms implemented in v0.6.0 what you were looking for? They are a bit coarse, but do help to stop the queue from growing uncontrollably.

from yaque.

@tokahuke thanks very very much to provide the mechanism.

in my use case: an agent is deployed beside a event source. event source emit events in push mode to agent. And agent is polled by a centric service. Agent may do some pre-processing. To avoid events make persistent storage too large, it will be necessary to limit the bound of event queue.

I've read the API docs, it was nice.

from yaque.

You are welcome! Nice that the docs were useful for something... about your use... looks a lot like the problem of database replication. However, there the event source will not wait for space to clear; therefore you are forced to abort the replication and start all over again.

... that is better than doing it manually, as I was forced to do in my job with postgresql logical replication. Thankfully, I was informed they corrected it in the most recent version..

from yaque.