Delta-compression framework for diverging branches in model training using Low-Rank Approximation (LoRA) and delta-encoding.

Built with guidance and support from Prof. Ooi Wei Tsang (NUS).

Design of the mechanism inspired by the following works:

1. Yu Chen, Zhenming Liu, Bin Ren & Xin Jin's On Efficient Construction of Checkpoints.

2. Haoyu Jin, Donglei Wu, Shuyu Zhang, Xiangyu Zou, Sian Jin, Dingwen Tao, Qing Liao and Wen Xia's Design of a Quantization-Based DNN Delta Compression Framework for Model Snapshot and Federated Learning

3. Shuyu Zhang, Donglei Wu, Haoyu Jin, Xiangyu Zou, Wen Xia & Xiaojia Huang's QD-Compressor: a Quantization-based Delta Compression Framework for Deep Neural Networks

4. Amey Agrawal, Sameer Reddy, Satwik Bhattamishra, Venkata Prabhakara Sarath Nookala, Vidushi Vashishth, Kexin Rong & Alexey Tumanov's DynaQuant: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang & Weizhu Chen's LoRA: Low-Rank Adaptation of Large Language Models

6. Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen & Tuo Zhao's LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation

7. Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang, Kam-Fai Wong, Lei Zhang's Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices