I created the environment as README says, and train on the SR3D,here are the errors:
/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/launch.py:164: DeprecationWarning: The 'warn' method is deprecated, use 'warning' instead
"The module torch.distributed.launch is deprecated "
The module torch.distributed.launch is deprecated and going to be removed in future.Migrate to torch.distributed.run
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
WARNING:torch.distributed.run:--use_env is deprecated and will be removed in future releases.
Please read local_rank from os.environ('LOCAL_RANK')
instead.
INFO:torch.distributed.launcher.api:Starting elastic_operator with launch configs:
entrypoint : train_dist_mod.py
min_nodes : 1
max_nodes : 1
nproc_per_node : 2
run_id : none
rdzv_backend : static
rdzv_endpoint : 127.0.0.1:3333
rdzv_configs : {'rank': 0, 'timeout': 900}
max_restarts : 3
monitor_interval : 5
log_dir : None
metrics_cfg : {}
INFO:torch.distributed.elastic.agent.server.local_elastic_agent:log directory set to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3
INFO:torch.distributed.elastic.agent.server.api:[default] starting workers for entrypoint: python
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/elastic/utils/store.py:53: FutureWarning: This is an experimental API and will be changed in future.
"This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=0
master_addr=127.0.0.1
master_port=3333
group_rank=0
group_world_size=1
local_ranks=[0, 1]
role_ranks=[0, 1]
global_ranks=[0, 1]
role_world_sizes=[2, 2]
global_world_sizes=[2, 2]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_0/0/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_0/1/error.json
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1972860) of binary: /home/sd/anaconda3/envs/EDA/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 3/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=1
master_addr=127.0.0.1
master_port=3333
group_rank=0
group_world_size=1
local_ranks=[0, 1]
role_ranks=[0, 1]
global_ranks=[0, 1]
role_world_sizes=[2, 2]
global_world_sizes=[2, 2]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_1/0/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_1/1/error.json
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1972896) of binary: /home/sd/anaconda3/envs/EDA/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 2/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=2
master_addr=127.0.0.1
master_port=3333
group_rank=0
group_world_size=1
local_ranks=[0, 1]
role_ranks=[0, 1]
global_ranks=[0, 1]
role_world_sizes=[2, 2]
global_world_sizes=[2, 2]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_2/0/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_2/1/error.json
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1972938) of binary: /home/sd/anaconda3/envs/EDA/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:[default] Worker group FAILED. 1/3 attempts left; will restart worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Stopping worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous'ing worker group
INFO:torch.distributed.elastic.agent.server.api:[default] Rendezvous complete for workers. Result:
restart_count=3
master_addr=127.0.0.1
master_port=3333
group_rank=0
group_world_size=1
local_ranks=[0, 1]
role_ranks=[0, 1]
global_ranks=[0, 1]
role_world_sizes=[2, 2]
global_world_sizes=[2, 2]
INFO:torch.distributed.elastic.agent.server.api:[default] Starting worker group
INFO:torch.distributed.elastic.multiprocessing:Setting worker0 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_3/0/error.json
INFO:torch.distributed.elastic.multiprocessing:Setting worker1 reply file to: /tmp/torchelastic_2x3bsv88/none_l9qwazu3/attempt_3/1/error.json
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
Traceback (most recent call last):
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 26, in
import pointnet2._ext as _ext
ImportError: /home/sd/.local/lib/python3.7/site-packages/pointnet2-0.0.0-py3.7-linux-x86_64.egg/pointnet2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNSt15__exception_ptr13exception_ptr9_M_addrefEv
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train_dist_mod.py", line 19, in
from main_utils import parse_option, BaseTrainTester
File "/home/sd/Harddisk/sba/BS/EDA-master/main_utils.py", line 27, in
from models import HungarianMatcher, SetCriterion, compute_hungarian_loss
File "/home/sd/Harddisk/sba/BS/EDA-master/models/init.py", line 7, in
from .bdetr import BeaUTyDETR
File "/home/sd/Harddisk/sba/BS/EDA-master/models/bdetr.py", line 18, in
from .backbone_module import Pointnet2Backbone
File "/home/sd/Harddisk/sba/BS/EDA-master/models/backbone_module.py", line 23, in
from pointnet2_modules import PointnetSAModuleVotes, PointnetFPModule
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_modules.py", line 21, in
import pointnet2_utils
File "/home/sd/Harddisk/sba/BS/EDA-master/pointnet2/pointnet2_utils.py", line 30, in
"Could not import _ext module.\n"
ImportError: Could not import _ext module.
Please see the setup instructions in the README: https://github.com/erikwijmans/Pointnet2_PyTorch/blob/master/README.rst
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 1972968) of binary: /home/sd/anaconda3/envs/EDA/bin/python
ERROR:torch.distributed.elastic.agent.server.local_elastic_agent:[default] Worker group failed
INFO:torch.distributed.elastic.agent.server.api:Local worker group finished (FAILED). Waiting 300 seconds for other agents to finish
/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/elastic/utils/store.py:71: FutureWarning: This is an experimental API and will be changed in future.
"This is an experimental API and will be changed in future.", FutureWarning
INFO:torch.distributed.elastic.agent.server.api:Done waiting for other agents. Elapsed: 0.0007307529449462891 seconds
{"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 0, "group_rank": 0, "worker_id": "1972968", "role": "default", "hostname": "Continent", "state": "FAILED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": "{"message": ""}", "metadata": "{"group_world_size": 1, "entry_point": "python", "local_rank": [0], "role_rank": [0], "role_world_size": [2]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.FAILED", "source": "WORKER", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": 1, "group_rank": 0, "worker_id": "1972969", "role": "default", "hostname": "Continent", "state": "FAILED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": "{"message": ""}", "metadata": "{"group_world_size": 1, "entry_point": "python", "local_rank": [1], "role_rank": [1], "role_world_size": [2]}", "agent_restarts": 3}}
{"name": "torchelastic.worker.status.SUCCEEDED", "source": "AGENT", "timestamp": 0, "metadata": {"run_id": "none", "global_rank": null, "group_rank": 0, "worker_id": null, "role": "default", "hostname": "Continent", "state": "SUCCEEDED", "total_run_time": 20, "rdzv_backend": "static", "raw_error": null, "metadata": "{"group_world_size": 1, "entry_point": "python"}", "agent_restarts": 3}}
/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py:354: UserWarning:
CHILD PROCESS FAILED WITH NO ERROR_FILE
CHILD PROCESS FAILED WITH NO ERROR_FILE
Child process 1972968 (local_rank 0) FAILED (exitcode 1)
Error msg: Process failed with exitcode 1
Without writing an error file to <N/A>.
While this DOES NOT affect the correctness of your application,
no trace information about the error will be available for inspection.
Consider decorating your top level entrypoint function with
torch.distributed.elastic.multiprocessing.errors.record. Example:
from torch.distributed.elastic.multiprocessing.errors import record
@record
def trainer_main(args):
# do train
warnings.warn(_no_error_file_warning_msg(rank, failure))
Traceback (most recent call last):
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/launch.py", line 173, in
main()
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/launch.py", line 169, in main
run(args)
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/run.py", line 624, in run
)(*cmd_args)
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 116, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 348, in wrapper
return f(*args, **kwargs)
File "/home/sd/anaconda3/envs/EDA/lib/python3.7/site-packages/torch/distributed/launcher/api.py", line 247, in launch_agent
failures=result.failures,
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
=======================================
Root Cause:
[0]:
time: 2023-12-23_12:33:23
rank: 0 (local_rank: 0)
exitcode: 1 (pid: 1972968)
error_file: <N/A>
msg: "Process failed with exitcode 1"
Other Failures:
[1]:
time: 2023-12-23_12:33:23
rank: 1 (local_rank: 1)
exitcode: 1 (pid: 1972969)
error_file: <N/A>
msg: "Process failed with exitcode 1"
Can anybody help me?