openmoss / say-i-dont-know Goto Github PK

When training with the DPO script, it was found that if training with four GPUs, the code seems to process four times the training data, as shown in the following, I used 64 preference data pairs to train a llama-7b (already sft ) , batch_size = 64, outputs:

`train stats after 64 examples: {'rewards_train/chosen': '0.0012727', 'rewards_train/rejected': '0.015093', 'rewards_train/accuracies': '0.375', 'rewards_train/margins': '-0.013821', 'logps_train/rejected': '-60.575', 'logps_train/chosen': '-87.107', 'loss/train': '1.5716', 'examples_per_second': '1.2032', 'grad_norm': '86.947', 'counters/examples': 64, 'counters/updates': 1}

2it [02:31, 75.90s/it]train stats after 128 examples: {'rewards_train/chosen': '-0.0066001', 'rewards_train/rejected': '0.0045876', 'rewards_train/accuracies': '0.46875', 'rewards_train/margins': '-0.011188', 'logps_train/rejected': '-67.253', 'logps_train/chosen': '-87.786', 'loss/train': '1.5771', 'examples_per_second': '0.83495', 'grad_norm': '79.911', 'counters/examples': 128, 'counters/updates': 2}

3it [03:43, 74.06s/it]train stats after 192 examples: {'rewards_train/chosen': '0.0057964', 'rewards_train/rejected': '-0.00096887', 'rewards_train/accuracies': '0.65625', 'rewards_train/margins': '0.0067653', 'logps_train/rejected': '-73.114', 'logps_train/chosen': '-74.139', 'loss/train': '1.4316', 'examples_per_second': '0.89487', 'grad_norm': '53.688', 'counters/examples': 192, 'counters/updates': 3}

4it [05:04, 76.91s/it]train stats after 256 examples: {'rewards_train/chosen': '-0.010614', 'rewards_train/rejected': '0.0054293', 'rewards_train/accuracies': '0.375', 'rewards_train/margins': '-0.016043', 'logps_train/rejected': '-73.208', 'logps_train/chosen': '-83.685', 'loss/train': '1.5388', 'examples_per_second': '0.78822', 'grad_norm': '112.48', 'counters/examples': 256, 'counters/updates': 4}`

Judging from the output, the model was updated 4 times in total, processing 64 examples each time. Could you please tell me what I did not run correctly? Why is it far more than the 64 examples of data I provided?

I just submitted a download application for data.zip. Can you help me review it?