How to support other layers? about agd HOT 2 OPEN

Hey @zq-OwO-qz, thanks for your interest in this project. I've currently shifted focus and am currently actively developing this project: https://github.com/jxbz/modula. There we support more layer types including GPT. The Modula project focuses mainly on scalability and learning rate transfer, but the plan is to return to AGD once Modula is built out enough.

We didn't support biases in AGD mainly for simplicity. We found in some early experiments that biases didn't really help performance, so just wanted to reduce the number of degrees of freedom we were considering in the project. If I remember correctly, this paper from NVIDIA made the same decision: https://arxiv.org/abs/2312.02696, so we are not alone in doing this.

from agd.

Thank you for bringing this project to my attention! I will follow this.

from agd.