Comments (3)
Can you show the log of the command you ran? It should save the adapters by default every 100 iterations to a file adapters.npz
. Most likely either your command did not run for 100 iterations or you are looking in the wrong place.
Also I'd encourage you to use MLX LM for a more featured fine-tuning package. Here is a complete guide.
from mlx-examples.
Below, I share the fine tuning code I performed and the output recorded during training.
run : Jupyter Notebook with python=3.11, macos 14.5(Mac pro M1 max 64GB)
*** Code :
!python -m mlx_lm.lora \
--model "google/gemma-2b-it"
--train \
--iters 600 \
--data data \
--steps-per-eval 100 \
--max-seq-length 2400 \ #Due to the nature of the training data I had, max-seq-length was set to 2.4k.
--learning-rate 2e-4 \
# --resume-adapter-file checkpoints/600_adapters.npz
*** Output:
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
Loading pretrained model
Fetching 9 files: 100%|████████████████████████| 9/9 [00:00<00:00, 85792.58it/s]
Loading datasets
Training
Trainable parameters: 0.033% (0.819M/2506.172M)
Starting training..., iters: 600
Iter 1: Val loss 3.936, Val took 215.383s
Iter 10: Train loss 2.987, Learning Rate 2.000e-04, It/sec 0.028, Tokens/sec 186.170, Trained Tokens 65620, Peak mem 69.060 GB
Iter 20: Train loss 1.504, Learning Rate 2.000e-04, It/sec 0.023, Tokens/sec 178.319, Trained Tokens 142273, Peak mem 69.060 GB
Iter 30: Train loss 0.448, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 172.839, Trained Tokens 220139, Peak mem 69.060 GB
Iter 40: Train loss 0.280, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 173.244, Trained Tokens 297254, Peak mem 69.060 GB
Iter 50: Train loss 0.237, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 172.502, Trained Tokens 374350, Peak mem 69.060 GB
Iter 60: Train loss 0.252, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 175.211, Trained Tokens 445673, Peak mem 69.060 GB
Iter 70: Train loss 0.238, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 176.555, Trained Tokens 517723, Peak mem 69.060 GB
Iter 80: Train loss 0.139, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 176.730, Trained Tokens 600930, Peak mem 69.060 GB
Iter 90: Train loss 0.132, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 169.215, Trained Tokens 683801, Peak mem 69.060 GB
Iter 100: Val loss 0.148, Val took 200.288s
Iter 100: Train loss 0.163, Learning Rate 2.000e-04, It/sec 0.218, Tokens/sec 1688.886, Trained Tokens 761390, Peak mem 69.060 GB
Iter 100: Saved adapter weights to adapters/adapters.safetensors and adapters/0000100_adapters.safetensors.
Iter 110: Train loss 0.174, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 165.624, Trained Tokens 839339, Peak mem 69.060 GB
Iter 120: Train loss 0.143, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 166.529, Trained Tokens 922932, Peak mem 69.060 GB
Iter 130: Train loss 0.214, Learning Rate 2.000e-04, It/sec 0.023, Tokens/sec 165.572, Trained Tokens 994731, Peak mem 69.060 GB
Iter 140: Train loss 0.162, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 170.473, Trained Tokens 1071710, Peak mem 69.060 GB
Iter 150: Train loss 0.165, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 159.002, Trained Tokens 1149892, Peak mem 69.060 GB
Iter 160: Train loss 0.160, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 166.218, Trained Tokens 1228023, Peak mem 69.060 GB
Iter 170: Train loss 0.181, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 172.606, Trained Tokens 1299179, Peak mem 69.060 GB
Iter 180: Train loss 0.191, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 174.946, Trained Tokens 1370992, Peak mem 69.060 GB
Iter 190: Train loss 0.175, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 177.959, Trained Tokens 1441732, Peak mem 69.060 GB
Iter 200: Val loss 0.123, Val took 214.330s
Iter 200: Train loss 0.145, Learning Rate 2.000e-04, It/sec 0.174, Tokens/sec 1354.205, Trained Tokens 1519404, Peak mem 69.060 GB
Iter 200: Saved adapter weights to adapters/adapters.safetensors and adapters/0000200_adapters.safetensors.
Iter 210: Train loss 0.171, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 169.563, Trained Tokens 1590632, Peak mem 69.060 GB
Iter 220: Train loss 0.168, Learning Rate 2.000e-04, It/sec 0.023, Tokens/sec 166.420, Trained Tokens 1662813, Peak mem 69.060 GB
Iter 230: Train loss 0.147, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 163.872, Trained Tokens 1741135, Peak mem 69.178 GB
Iter 240: Train loss 0.138, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 168.465, Trained Tokens 1818984, Peak mem 69.178 GB
Iter 250: Train loss 0.152, Learning Rate 2.000e-04, It/sec 0.023, Tokens/sec 166.229, Trained Tokens 1890048, Peak mem 69.178 GB
Iter 260: Train loss 0.156, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 179.677, Trained Tokens 1960832, Peak mem 69.178 GB
Iter 270: Train loss 0.156, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 177.873, Trained Tokens 2031622, Peak mem 69.178 GB
Iter 280: Train loss 0.135, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 166.591, Trained Tokens 2108826, Peak mem 69.178 GB
Iter 290: Train loss 0.104, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 165.676, Trained Tokens 2192247, Peak mem 69.178 GB
Iter 300: Val loss 0.124, Val took 191.932s
Iter 300: Train loss 0.128, Learning Rate 2.000e-04, It/sec 0.218, Tokens/sec 1687.373, Trained Tokens 2269755, Peak mem 69.178 GB
Iter 300: Saved adapter weights to adapters/adapters.safetensors and adapters/0000300_adapters.safetensors.
Iter 310: Train loss 0.101, Learning Rate 2.000e-04, It/sec 0.019, Tokens/sec 161.071, Trained Tokens 2353171, Peak mem 69.178 GB
Iter 320: Train loss 0.135, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 168.836, Trained Tokens 2431368, Peak mem 69.178 GB
Iter 330: Train loss 0.183, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 169.026, Trained Tokens 2497720, Peak mem 69.178 GB
Iter 340: Train loss 0.137, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 169.529, Trained Tokens 2575376, Peak mem 69.178 GB
Iter 350: Train loss 0.130, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 170.451, Trained Tokens 2652845, Peak mem 69.178 GB
Iter 360: Train loss 0.132, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 159.830, Trained Tokens 2730910, Peak mem 69.178 GB
Iter 370: Train loss 0.123, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 165.865, Trained Tokens 2808359, Peak mem 69.178 GB
Iter 380: Train loss 0.131, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 160.952, Trained Tokens 2886630, Peak mem 69.178 GB
Iter 390: Train loss 0.147, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 177.275, Trained Tokens 2957472, Peak mem 69.178 GB
Iter 400: Val loss 0.107, Val took 203.717s
Iter 400: Train loss 0.125, Learning Rate 2.000e-04, It/sec 0.202, Tokens/sec 1573.826, Trained Tokens 3035261, Peak mem 69.178 GB
Iter 400: Saved adapter weights to adapters/adapters.safetensors and adapters/0000400_adapters.safetensors.
Iter 410: Train loss 0.095, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 169.285, Trained Tokens 3118145, Peak mem 69.178 GB
Iter 420: Train loss 0.146, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 172.076, Trained Tokens 3189678, Peak mem 69.178 GB
Iter 430: Train loss 0.154, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 170.905, Trained Tokens 3261046, Peak mem 69.178 GB
Iter 440: Train loss 0.119, Learning Rate 2.000e-04, It/sec 0.023, Tokens/sec 177.243, Trained Tokens 3337667, Peak mem 69.178 GB
Iter 450: Train loss 0.151, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 168.843, Trained Tokens 3409339, Peak mem 69.178 GB
Iter 460: Train loss 0.202, Learning Rate 2.000e-04, It/sec 0.034, Tokens/sec 202.335, Trained Tokens 3468201, Peak mem 69.178 GB
Iter 470: Train loss 0.118, Learning Rate 2.000e-04, It/sec 0.022, Tokens/sec 170.467, Trained Tokens 3545649, Peak mem 69.178 GB
Iter 480: Train loss 0.096, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 165.786, Trained Tokens 3628853, Peak mem 69.178 GB
Iter 490: Train loss 0.148, Learning Rate 2.000e-04, It/sec 0.025, Tokens/sec 179.053, Trained Tokens 3700354, Peak mem 69.178 GB
Iter 500: Val loss 0.110, Val took 201.131s
Iter 500: Train loss 0.091, Learning Rate 2.000e-04, It/sec 0.189, Tokens/sec 1565.547, Trained Tokens 3783274, Peak mem 69.178 GB
Iter 500: Saved adapter weights to adapters/adapters.safetensors and adapters/0000500_adapters.safetensors.
Iter 510: Train loss 0.099, Learning Rate 2.000e-04, It/sec 0.019, Tokens/sec 156.952, Trained Tokens 3867084, Peak mem 69.178 GB
Iter 520: Train loss 0.095, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 165.927, Trained Tokens 3950769, Peak mem 69.178 GB
Iter 530: Train loss 0.100, Learning Rate 2.000e-04, It/sec 0.019, Tokens/sec 163.905, Trained Tokens 4034831, Peak mem 69.178 GB
Iter 540: Train loss 0.095, Learning Rate 2.000e-04, It/sec 0.021, Tokens/sec 174.305, Trained Tokens 4118042, Peak mem 69.178 GB
Iter 550: Train loss 0.161, Learning Rate 2.000e-04, It/sec 0.027, Tokens/sec 176.647, Trained Tokens 4183819, Peak mem 69.178 GB
Iter 560: Train loss 0.148, Learning Rate 2.000e-04, It/sec 0.023, Tokens/sec 168.409, Trained Tokens 4255725, Peak mem 69.178 GB
Iter 570: Train loss 0.094, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 167.652, Trained Tokens 4339301, Peak mem 69.178 GB
Iter 580: Train loss 0.138, Learning Rate 2.000e-04, It/sec 0.024, Tokens/sec 169.197, Trained Tokens 4410586, Peak mem 69.178 GB
Iter 590: Train loss 0.094, Learning Rate 2.000e-04, It/sec 0.020, Tokens/sec 167.730, Trained Tokens 4493852, Peak mem 69.178 GB
Iter 600: Val loss 0.102, Val took 215.875s
Iter 600: Train loss 0.126, Learning Rate 2.000e-04, It/sec 0.214, Tokens/sec 1653.713, Trained Tokens 4570994, Peak mem 69.178 GB
Iter 600: Saved adapter weights to adapters/adapters.safetensors and adapters/0000600_adapters.safetensors.
Saved final adapter weights to adapters/adapters.safetensors.
from mlx-examples.
Ah I think the problem is its saving adapters/adapters.safetensors
not adapters.npz
. Maybe the tutorial you are referring to is somewhat out of date. But either way you should have access to the adapters in adapters/adapters.safetensors
from mlx-examples.
Related Issues (20)
- CLIP Tokenizer unable to take text of unequal length (or token length) HOT 7
- Package 'mlx_whisper.assets' is absent from the `packages` configuration HOT 1
- [Feature Request] Add support for logprobs to the mlx_lm server HOT 3
- Request for Example on Full Parameter and Training for LLM Model HOT 5
- Phi-3 128K Context Variants' `su` RoPE Scaling HOT 12
- [REGRESSION] Some MoE models display 0% GPU utilization with mlx-ops 0.14.0 HOT 3
- I would like to inquire about a solution to the following problem. HOT 1
- link to Phi-2 example in readme broken HOT 1
- SPMStreamingDetokenizer sometimes outputs incorrect multi-byte characters HOT 1
- Why change the module decomposition of whisper HOT 3
- A simple enhancement, in dataset creation time HOT 1
- [QUESTION] Is there a way to provide a Huggingface access token for downloading models that are private? HOT 1
- [Model Request] Add support for IBM's Granite model HOT 2
- [Feature] Export Lora Adapters as GGML HOT 3
- Error when running inference on newly converted OpenELM MLX model, ValueError(f"Received parameters not in model: {extras}.") HOT 1
- LLMEvaluator : libc++abi: terminating due to uncaught exception of type std::invalid_argument: [matmul] Last dimension of first input with shape (1,916,2048) must match second to last dimension of second input with shape (256,32000)
- Unable to allocate memory
- Proposal: Add mypy to .pre-commit-config.yml HOT 2
- Fusing adapters with llama3 cause bad performances HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mlx-examples.