Request for Example on Full Parameter and Training for LLM Model about mlx-examples HOT 5 CLOSED

full parameters and training?

Do you mean fine-tuning all the parameters of an LLM? There is a PR for that you could check out if so: #645

from mlx-examples.

Not fine-tuning. I mean pre-train the model.
For example, I need to use the model in certain professional fields.
Like llama model, I want to add some tokens and retrain it.

from mlx-examples.

Like llama model, I want to add some tokens and retrain it.

Based on your comment it sounds like you want to fine tune the model. But are you saying you want an example which trains something like a Llama model from scratch? That is very computationally expensive.

from mlx-examples.

Yes. Because mac studio is much cheaper than nvida
On a Mac with 192G memory, I want to try to see if I can train a model like llama from scratch.

from mlx-examples.

Training something like Llama from scratch on a single Mac Studio is a tall order. Llama 3 was trained on 15 trillion tokens on a cluster with 24K H100. The difference in compute from that to a single Mac studio is probably well over 100000x.

If you can train a much smaller model or fine tune that would be much more feasible. Here is an example in MLX of training a Transformer LM from scratch: https://github.com/ml-explore/mlx-examples/tree/main/transformer_lm

from mlx-examples.