stale brick process when volume stop operations are done in parallel in brick multiplexing mode about glusterd2 HOT 8 OPEN

@atinmu I am having trouble understanding how a list might help. The portmapper we use currently is kind of an in-memory multi list itself.

from glusterd2.

I am having trouble understanding how a list might help. The portmapper we use currently is kind of an in-memory multi list itself.

portmapper is an in-memory data structure, however the values in the data structure are (de)populated asynchronously based on the Sign In/Out events. So determining the number of bricks attached a glusterfsd process by looking through the same port can go for a toss depending on how fast/slow the events are received/processed by GD2 on the subsequent brick attach/detach requests. Instead if we maintain a list which gets (de)populated when a brick is attached/detached, there's no asynchronous nature in it which is the issue described here. I hope this clarifies the problem statement and why a need for the separate data structure for the bricks per process data. In case it doesn't, please feel free to ask.

from glusterd2.

@atinmu @aravindavk I have one idea which doesn't include adding a whole new data-structure. Instead of just having a mapping of port->brickpath->pid, can we add some sort of current state like pending-delete. So as soon as I send a detach request I can update the port mapper to change the state of brickpath to pending-delete and decrease the number of active bricks on a port(or when calculating IsLastBrickInProc() consider bricks with this state as well), when signOut req comes it can match pending-delete state and delete the brick entry.

from glusterd2.

@atinmu @aravindavk I have one idea which doesn't include adding a whole new data-structure. Instead of just having a mapping of port->brickpath->pid, can we add some sort of current state like pending-delete. So as soon as I send a detach request I can update the port mapper to change the state of brickpath to pending-delete and decrease the number of active bricks on a port(or when calculating IsLastBrickInProc() consider bricks with this state as well), when signOut req comes it can match pending-delete state and delete the brick entry.

How about removing brickpath from portmap registry, signout coming after this can be safely discarded. Port collision will not happen since glusterd2 is not choosing the port.

from glusterd2.

@aravindavk Yes, I totally agree on this. We just need to think if this could have any negative effect in any use case, once we scale up.

from glusterd2.

only problem I could see is when glusterd2 restarts, this registry is reset. So if a Volume delete comes after the glusterd2 restart and before all bricks signed in, then the check will say that is the last brick and glusterd2 may terminate the process even though other bricks exists in that process.

from glusterd2.

That is one issue. Unless we handle signin in GD2 as well.

from glusterd2.

Taking this out from GCS/1.0 tag considering we're not going to make brick multiplexing a default option in GCS/1.0 release.

from glusterd2.