Training details? about internlm HOT 1 CLOSED

Hi, thank you for your interest in our project. As you mentioned, a global batch of 16M is indeed quite large. We commonly use configurations such as 512 GPUs with a global batch size of 4M, or 8M with a global batch size on 1024 GPUs for training. We have included information in our readme file regarding the performance testing under a certain global batch sizes.

As the number of GPUs increases, the batch size per GPU will gradually decrease, resulting in a higher proportion of communication overhead. This inevitably leads to a decrease in TGS or tflops. Additionally, enabling options such as pack_sample_into_one, bfloat16/float16, and reduce_bucket_size can cause fluctuations in TGS, and the network status of the cluster can also introduce minor disturbances. We are currently working continuously to address the communication overhead issue in small batches. We will keep you updated on any new developments.

from internlm.