Comments (4)
Hi thanks so much for reporting! I have created #1743 to close this issue.
However, the num_train_epochs
and num_ppo_epochs
are actually two different things. The num_train_epochs
means how many epochs do we go over the dataset, the num_ppo_epochs
means the number of epochs we perform PPO updates on a batch of data. So, there is a subtle but meaningful difference here.
from trl.
@vwxyzjn Thanks for your comments. Just to clarify, is mini_batch_size
the batch size used when we perform PPO updates on a batch of data? If so, does per_device_train_batch_size
actually control the batch size of a mini-batch in PPOv2Trainer
?
from trl.
Just to clarify, is mini_batch_size the batch size used when we perform PPO updates on a batch of data?
The terminology goes like this:
import numpy as np
batch_size = 8
nminibatches = 2
gradient_accumulation_steps = 2
mini_batch_size = batch_size // nminibatches
micro_batch_size = mini_batch_size // gradient_accumulation_steps
data = np.arange(batch_size).astype(np.float32)
print("data:", data)
print("batch_size:", batch_size)
print("mini_batch_size:", mini_batch_size)
print("micro_batch_size:", micro_batch_size)
for epoch in range(4):
batch_inds = np.random.permutation(batch_size)
print("epoch:", epoch, "batch_inds:", batch_inds)
for mini_batch_start in range(0, batch_size, mini_batch_size):
mini_batch_end = mini_batch_start + mini_batch_size
mini_batch_inds = batch_inds[mini_batch_start:mini_batch_end]
# `optimizer.zero_grad()` set optimizer to zero for gradient accumulation
for micro_batch_start in range(0, mini_batch_size, micro_batch_size):
micro_batch_end = micro_batch_start + micro_batch_size
micro_batch_inds = mini_batch_inds[micro_batch_start:micro_batch_end]
print("____⏩ a forward pass on", data[micro_batch_inds])
# `optimizer.step()`
print("⏪ a backward pass on", data[mini_batch_inds])
# data: [0. 1. 2. 3. 4. 5. 6. 7.]
# batch_size: 8
# mini_batch_size: 4
# micro_batch_size: 2
# epoch: 0 batch_inds: [6 4 0 7 3 5 1 2]
# ____⏩ a forward pass on [6. 4.]
# ____⏩ a forward pass on [0. 7.]
# ⏪ a backward pass on [6. 4. 0. 7.]
# ____⏩ a forward pass on [3. 5.]
# ____⏩ a forward pass on [1. 2.]
# ⏪ a backward pass on [3. 5. 1. 2.]
# epoch: 1 batch_inds: [6 7 3 2 0 4 5 1]
# ____⏩ a forward pass on [6. 7.]
# ____⏩ a forward pass on [3. 2.]
# ⏪ a backward pass on [6. 7. 3. 2.]
# ____⏩ a forward pass on [0. 4.]
# ____⏩ a forward pass on [5. 1.]
# ⏪ a backward pass on [0. 4. 5. 1.]
# epoch: 2 batch_inds: [1 4 5 6 0 7 3 2]
# ____⏩ a forward pass on [1. 4.]
# ____⏩ a forward pass on [5. 6.]
# ⏪ a backward pass on [1. 4. 5. 6.]
# ____⏩ a forward pass on [0. 7.]
# ____⏩ a forward pass on [3. 2.]
# ⏪ a backward pass on [0. 7. 3. 2.]
# epoch: 3 batch_inds: [7 2 4 1 3 0 6 5]
# ____⏩ a forward pass on [7. 2.]
# ____⏩ a forward pass on [4. 1.]
# ⏪ a backward pass on [7. 2. 4. 1.]
# ____⏩ a forward pass on [3. 0.]
# ____⏩ a forward pass on [6. 5.]
# ⏪ a backward pass on [3. 0. 6. 5.]
from trl.
per_device_train_batch_size
should really be named forward_batch_size
. It's the size of the data that goes in to model.foward(data)
, so per_device_train_batch_size
effectively helps with OOMs.
from trl.
Related Issues (20)
- DataCollatorForCompletionOnlyLM not working with MPS HOT 3
- How to dpo llava?
- How to Add PEFT to PPO Trainer or PPO Config
- Error in AutoModelForCausalLMWithValueHead.load_pretrained("path/to/my/ppo_model")
- Adding Space caused " This instance will be ignored in loss calculation. Note, if this happens often, ..." HOT 1
- losses = losses * self.args.rpo_alpha + policy_nll_loss TypeError: only integer tensors of a single element can be converted to an index HOT 1
- how to use kto_pair loss in the latest version ? HOT 2
- Additional Optional Loss to Center Reward Models HOT 4
- How to Save the PPOTrainer? HOT 3
- Last layer of the Llava-1.5 visual tower is not training
- Is there a way to finetune a finetuned model? HOT 1
- save error HOT 4
- Unexpected Keyword Argument 'add_special_tokens' in LlavaProcessor with LLAVA-1.5-7B HOT 3
- PPOTrainer + AutoModelForCausalLMWithValueHead + Gemma 2 2B -> `RuntimeError: probability tensor contains either `inf`, `nan` or element < 0` HOT 6
- A bug in `__init__` method of class `DPOTrainer`? HOT 4
- Epoch number not right HOT 4
- Error in PPOv2
- How to fine-tune LLaVA using PPO HOT 1
- llama3 tokenizer compatibility of DPOTrainer
- issue with precompute_ref_log_probs and dpo training loss stucks at 0.6931
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from trl.