Comments (4)
hijkzzz
commented on July 30, 2024
@Ashura5 就是这个？https://github.com/OpenRLHF/OpenRLHF/blob/main/examples/scripts/train_iterative_dpo_llama.sh
from openrlhf.
Ashura5
commented on July 30, 2024
这个是在训练之前从reference model中推理数据，reward model打分
我想实现的是在训练时，每训练一步policy model，都从policy model中重新采样数据
from openrlhf.
hijkzzz
commented on July 30, 2024
这个是在训练之前从reference model中推理数据，reward model打分 我想实现的是在训练时，每训练一步policy model，都从policy model中重新采样数据
这个实现的就是每轮用最新的checkpoint 推理数据打分，只不过batch size 默认设置的比较大。如果要实现PPO那种完全online的会复杂一点，可以参考train_ppo_ray.py的设计
from openrlhf.
Ashura5
commented on July 30, 2024
这个是在训练之前从reference model中推理数据，reward model打分 我想实现的是在训练时，每训练一步policy model，都从policy model中重新采样数据
这个实现的就是每轮用最新的checkpoint 推理数据打分，只不过batch size 默认设置的比较大。如果要实现PPO那种完全online的会复杂一点，可以参考train_ppo_ray.py的设计
好的，谢谢，我再仔细看看这一段
from openrlhf.
Related Issues (20)
- DPO Finetuning constantly gives preference loss as 0.6931 HOT 8
- Difference between `DeepSpeedEngine.save_checkpoint()` and `DeepSpeedStrategy.save_model()` HOT 2
- DPO后的模型推理出的结果都是无序符号 HOT 1
- Support training from breakpoint HOT 3
- llama3 70B DPO example script
- where is gradient_accumulation HOT 1
- Support RLOO HOT 1
- 现在Train_PPO_llama_ray 过程中会把Actor Model切分到不同卡上吗 HOT 4
- ConnectionRefusedError: [Errno 111] Connection refused HOT 5
- packing的问题 HOT 2
- "right" padding hardcoded HOT 3
- Error while saving the model under 4bit lora HOT 2
- multinode ppo training extremely slow HOT 15
- 使用ray的时候Request Entity Too Large HOT 3
- dpo 训练显存 OOM HOT 1
- Feature: add DPO-P
- Zero stage 3 error HOT 1
- Performance of Iterative DPO? HOT 1
- Why multiplying rstd instead of dividing by rstd? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openrlhf.