FMEngine is a utility library for training very large foundation models. The goal of fmengine is to provide the following:

• Ergonomic interface for training foundation models. It is sufficient easy for a beginner to use, but also provides enough flexibility for advanced users to customize their training.
• Efficient optimizations built in. FMEngine is equipped with Flash Attention and various fused ops to accelerate training.
• HPC-friendly installation with pre-built docker and singularity/apptainer containers. FMEngine is mainly designed and tested on Slurm clusters. We provide starter scripts for running FMEngine on Slurm clusters.
• Compatible with existing frameworks and tools, particularly with HuggingFace. Since FMEngine is built with DeepSpeed, it is also compatible with all DeepSpeed features.

FMEngine is primarily implemented and maintained by the Efficient Architecture and Systems Labs @ ETH Zurich.

We thank our friends for their generous support:

• @xzyaoi
• @LorrinWWW
• @fishiu: Scaling up Low Rank Adapters + Longer Contexts
• @Taishi-N324: Benchmarking and Scaling Experiments
• @chiennv2000: Mistral Integration