Comments (4)
[2023-12-01 05:53:04,949] [INFO] [comm.py:662:init_distributed] Distributed backend already initialized
[2023-12-01 05:53:15,433] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
[2023-12-01 05:53:15,436] [INFO] [logging.py:96:log_dist] [Rank 0] Using client Optimizer as basic optimizer
[2023-12-01 05:53:15,436] [INFO] [logging.py:96:log_dist] [Rank 0] Removing param_group that has no 'params' in the basic Optimizer
[2023-12-01 05:53:15,453] [INFO] [logging.py:96:log_dist] [Rank 0] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
[2023-12-01 05:53:15,453] [INFO] [utils.py:56:is_zero_supported_optimizer] Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2023-12-01 05:53:15,453] [INFO] [logging.py:96:log_dist] [Rank 0] Creating torch.bfloat16 ZeRO stage 2 optimizer
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:146:__init__] Reduce bucket size 500,000,000
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:147:__init__] Allgather bucket size 500,000,000
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:148:__init__] CPU Offload: True
[2023-12-01 05:53:15,453] [INFO] [stage_1_and_2.py:149:__init__] Round robin gradient partitioning: False
[2023-12-01 05:53:49,783] [INFO] [utils.py:802:see_memory_usage] Before initializing optimizer states
[2023-12-01 05:53:49,784] [INFO] [utils.py:803:see_memory_usage] MA 12.86 GB Max_MA 12.86 GB CA 12.86 GB Max_CA 13 GB
[2023-12-01 05:53:49,784] [INFO] [utils.py:810:see_memory_usage] CPU Virtual Memory: used = 52.58 GB, percent = 41.9%
[2023-12-01 05:54:12,066] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 2440
[2023-12-01 05:54:12,099] [ERROR] [launch.py:321:sigkill_handler] ['/usr/bin/python', '-u', '../train_sft.py', '--local_rank=0', '--max_len', '200', '--dataset', 'OpenOrca', '--dataset_probs', '1.0', '--train_batch_size', '50', '--micro_train_batch_size', '1', '--max_samples', '500', '--pretrain', 'Llama-2-7b-hf', '--save_path', './ckpt/7b_llama', '--save_steps', '-1', '--logging_steps', '1', '--eval_steps', '-1', '--ckpt_path', './ckpt/7b_llama/checkpoints_sft', '--zero_stage', '2', '--max_epochs', '1', '--bf16', '--flash_attn', '--learning_rate', '5e-6', '--adam_offload', '--gradient_checkpointing'] exits with return code = -9
from openrlhf.
And i use the provided nvidia-docker and modified train_sft_llama.sh
is
set -x
read -r -d '' training_commands <<EOF
../train_sft.py \
--max_len 200 \
--dataset OpenOrca \
--dataset_probs 1.0 \
--train_batch_size 50 \
--micro_train_batch_size 1 \
--max_samples 500 \
--pretrain Llama-2-7b-hf \
--save_path ./ckpt/7b_llama \
--save_steps -1 \
--logging_steps 1 \
--eval_steps -1 \
--ckpt_path ./ckpt/7b_llama/checkpoints_sft \
--zero_stage 2 \
--max_epochs 1 \
--bf16 \
--flash_attn \
--learning_rate 5e-6 \
--adam_offload \
--gradient_checkpointing
EOF
# --wandb [WANDB_TOKENS]
if [[ ${1} != "slurm" ]]; then
export PATH=$HOME/.local/bin/:$PATH
CUDA_VISIBLE_DIVICES=0 deepspeed $training_commands
fi
I have downloaded dataset and model locally.
from openrlhf.
I have tested it, and it works well on my side, perhaps because your CPU memory is small.
Ideally your CPU should have at least 128GB of RAM。
from openrlhf.
thx for your reply:)
from openrlhf.
Related Issues (20)
- PPO OOM 8*A100 40G HOT 11
- 使用ray训练PPO时,需要把模型都上传到集群吗,为什么我每次运行的时候ray都会因为要传输的东西太大而断开连接 HOT 2
- No module named 'vllm' HOT 3
- an unexpected error while SFT HOT 2
- ValueError: invalid literal for int() with base 10: 'MIG-cfb2a8ae-864b-50df-94a5-98983023f29d' HOT 3
- Qlora load model error HOT 1
- 无法用Qwen2-70b给 Qwen2-1.5b进行蒸馏 报错 tensor size 不能match HOT 1
- ppo 错误 HOT 2
- 8卡 4090 ppo ray 训练问题 HOT 1
- 模型参数量与显存占用关系 HOT 1
- ppo 错误 HOT 7
- Data Preprocess Speed Up HOT 1
- torch.distributed.broadcast timeout HOT 2
- 版本冲突问题 HOT 1
- AssertionError: Session name does not match persisted value HOT 1
- ppo错误 HOT 1
- flash_attn问题 HOT 3
- 请问有蒸馏的loss曲线可以参考吗 HOT 1
- PPO takes very long HOT 10
- 请问这里循环处为什么设置dist.barrier()? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from openrlhf.