How does FastMoE compare to the Switch Transformer about fastmoe HOT 4 CLOSED

FastMoE is a training system, while Switch Transformer is a model. You can implement a equivalent switch transformer model with the functions provided by FastMoE (some of them like routing strategy are still being developed).

For performance metric (training speed), I think the only metric that can bridge different hardware is utilization, i.e. training throughput / peak throughput of the cluster. Unfortunately, I do not see the switch tansformer team report their utilization.

For figure 7, the T5 is 6-7x faster is compared on per step basis, which equals to figure 7 left. We did not calculate the actual speed up because we do not want to fix the loss value which means the model is okay. You may find out the speedup from figure 7 left, which should be similar to the result in switch transformer.

from fastmoe.

hi @laekov ,

Thank you for the answer! Should have made the question clearer: how do you compare the MoE model implemented in the FastMoE paper to switch transformer implemented in mesh-tensorflow?

Are you suggesting that switch transformer (considering both the model and the system) because it has a better routing strategy?

I got th 6-7x number from the figure 5 in the switch transformer paper, where the x-axis is time, not steps.

from fastmoe.

hi @laekov ,

Thank you for the answer! Should have made the question clearer: how do you compare the MoE model implemented in the FastMoE paper to switch transformer implemented in mesh-tensorflow?

Are you suggesting that switch transformer (considering both the model and the system) because it has a better routing strategy?

I got th 6-7x number from the figure 5 in the switch transformer paper, where the x-axis is time, not steps.

FastMoE is of course not that strong as switch transformer and mtf, as we are only beginning to implement the system that can train an MoE model.

We mentioned the model in our arxiv article to show that FastMoE can train real models, but we do not suggest that either the model or the system is stronger or comparable with Google's. We do hope that FastMoE can be stronger in the future, but currently, we have neither strong human resource nor abundant hardware resource.

I was looking at figure 8 in switch's article when I said training step. You are right that switch transformer shows 7x speedup in training time against its baseline.

from fastmoe.

Got it! thank you : )

from fastmoe.