about fmoe forward implement about fastmoe HOT 3 CLOSED

Yes. We assume that in mp_group, the input tensor is duplicated.

from fastmoe.

1.We think it is not necessary to use slice and allgather in the fmoe forward implement. Is it feasible if each child process(member of mp_group) just reads its own data instead of communicating? What's problem it may cause?
2. Usually layers before fmoe and after fmoe are data parallel. the data is different but not duplicated.

from fastmoe.

The mp_group feature is based on the model parallel method of Megatron-LM. The attention layer requries inputs of the whole sequence to be duplicated across all workers, and also provides duplicated output on all workers. Therefore, we have to perform Slice and AllGather. Otherwise, outputs and gradients are all numerically incorrect.

from fastmoe.