View update backlog may not be read from gossip if it drops to 0 about scylladb HOT 4 OPEN

I only have vague recollections about this code (it was Duarte who wrote it six years ago), but this may have been deliberate, a (permature?) optimization that if there's nothing interesting to gossip, we won't gossip, and when the other nodes want to know our backlog, they see they don't have any new information and assume (even without this gossip) the value is zero. I remember (again, vaguely) that we remember the time when the backlog information was received (via gossip or piggy-backed on a response), and after a short time we "forget" this value and it becomes zero anyway.

So although I share you puzzlement on why there is this weird maxint corner-case, I'm afraid I don't understand what is the actual problem you are reporting here. Is the problem that we only "forget" (zero) the outdated gossiped value after a long time (?) and not quickly enough ? Don't we get the zero value almost immediately in responses? Did you see an actual problem with this in a real test case, and can you say something about what you did and what you saw?

from scylladb.

So although I share you puzzlement on why there is this weird maxint corner-case, I'm afraid I don't understand what is the actual problem you are reporting here. Is the problem that we only "forget" (zero) the outdated gossiped value after a long time (?) and not quickly enough ? Don't we get the zero value almost immediately in responses? Did you see an actual problem with this in a real test case, and can you say something about what you did and what you saw?

The incorrect values that we had because of this issue were interfering with my testing of #18334.
This issue doesn't have any big consequences in the current codebase because we're only using the backlogs for calculating delays for mv flow control, so the worst that could happen is using too long of a delay.
However, what we have is also just incorrect. Because of the ULLONG_MAX condition, we simply cannot reduce the tracked view update backlog size to 0 through gossip. Many times, it will be quickly updated through a response, but it doesn't have to - if the requests were sent in one big burst, the backlog returned in their responses may have grown considerably, and if it managed to drop to 0 before the gossip timer finished, we will be stuck with the bigger value

from scylladb.

So although I share you puzzlement on why there is this weird maxint corner-case, I'm afraid I don't understand what is the actual problem you are reporting here. Is the problem that we only "forget" (zero) the outdated gossiped value after a long time (?) and not quickly enough ? Don't we get the zero value almost immediately in responses? Did you see an actual problem with this in a real test case, and can you say something about what you did and what you saw?

However, what we have is also just incorrect. Because of the ULLONG_MAX condition, we simply cannot reduce the tracked view update backlog size to 0 through gossip. Many times, it will be quickly updated through a response, but it doesn't have to - if the requests were sent in one big burst, the backlog returned in their responses may have grown considerably, and if it managed to drop to 0 before the gossip timer finished, we will be stuck with the bigger value

I had a vague recollection that we had sort of expiration time on backlogs, so if we remember some old value from a minute ago and never got a new value, we ignore the old value and default to 0. But now that I tried to look for this code, I couldn't find so I guess it doesn't exist, and you're right - gossip should be allowed to send the value 0.

One consequence of this will be that a database that has no materialized views at all will still gossip these zero values all the time. I guess this negligible, but maybe this was the reason why this "optimization" was added originally?

from scylladb.

I had a vague recollection that we had sort of expiration time on backlogs, so if we remember some old value from a minute ago and never got a new value, we ignore the old value and default to 0. But now that I tried to look for this code, I couldn't find so I guess it doesn't exist, and you're right - gossip should be allowed to send the value 0.

One consequence of this will be that a database that has no materialized views at all will still gossip these zero values all the time. I guess this negligible, but maybe this was the reason why this "optimization" was added originally?

Actually, this issue doesn't really concern sending the value, just receiving them (the problematic if is in view_update_backlog_broker::on_change()) - we can get the 0 value through gossip but we ignore it.
We have a problem with sending backlog values through gossip as well but it's slightly different: #18461

from scylladb.