ConnectionRefusedError: [Errno 111] Connection refused about openrlhf HOT 5 CLOSED

set -x
export PATH=$HOME/.local/bin/:$PATH

ray job submit --address="http://127.0.0.1:8266"
--runtime-env-json='{"working_dir": "/openrlhf", "pip": "/openrlhf/requirements.txt"}'
-- python3 examples/train_ppo_ray.py
--ref_num_nodes 1
--ref_num_gpus_per_node 1
--reward_num_nodes 1
--reward_num_gpus_per_node 1
--critic_num_nodes 1
--critic_num_gpus_per_node 1
--actor_num_nodes 1
--actor_num_gpus_per_node 1
--vllm_num_engines 2
--vllm_tensor_parallel_size 2
--colocate_critic_reward
--colocate_actor_ref
--ref_reward_offload
--pretrain /openrlhf/OpenLLMAI/Llama-3-8b-sft-mixture
--reward_pretrain /openrlhf/examples/scripts/checkpoint/llama3-8b-rm
--save_path ./checkpoint/llama-3-8b-rlhf
--micro_train_batch_size 1
--train_batch_size 128
--micro_rollout_batch_size 1
--rollout_batch_size 1024
--max_samples 100000
--max_epochs 1
--prompt_max_len 1024
--generate_max_len 1024
--zero_stage 3
--bf16
--actor_learning_rate 5e-7
--critic_learning_rate 9e-6
--init_kl_coef 0.01
--prompt_data /openrlhf/OpenLLMAI/prompt-collection-v0.1
--input_key context_messages
--apply_chat_template
--normalize_reward
--adam_offload
--flash_attn
--gradient_checkpointing
--use_wandb {wandb_token}
这是配置

from openrlhf.

Traceback (most recent call last):
File "/root/.local/lib/python3.10/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 262, in _check_connection_and_version_with_url
r = self._do_request("GET", url)
File "/root/.local/lib/python3.10/site-packages/ray/dashboard/modules/dashboard_sdk.py", line 303, in _do_request
return requests.request(
File "/root/.local/lib/python3.10/site-packages/requests/api.py", line 59, in request
return session.request(method=method, url=url, **kwargs)
File "/root/.local/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
File "/root/.local/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
File "/root/.local/lib/python3.10/site-packages/requests/adapters.py", line 700, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8266): Max retries exceeded with url: /api/version (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f0fca25a3e0>: Failed to establish a new connection: [Errno 111] Connection refused'))

from openrlhf.

要先运行：ray start --head --node-ip-address 0.0.0.0
然后nodes 卡数根据GPU数量配置

from openrlhf.

感谢...没用过ray

from openrlhf.

--colocate_critic_reward
--colocate_actor_ref
--ref_reward_offload

--colocate_critic_reward
--colocate_actor_ref
是用来合并节点的

from openrlhf.