Error While Pretraining about mdetr HOT 2 OPEN

Hi @mmaaz60
Thanks for your interest in MDETR.

Could you provide the following information to help debug your error?:

• Exact command line
• Did you change anything to the dataset?
• Have you tried running it on one gpu first?
• Have you tried running on cpu first?

from mdetr.

Hi @mmaaz60
Thanks for your interest in MDETR.

Could you provide the following information to help debug your error?:

Hi @alcinos,

Thank you for your reply. Please find the required information below.

• Exact command line

export CUBLAS_WORKSPACE_CONFIG=:4096:8
python -m torch.distributed.launch --nproc_per_node=4 --use_env main.py --dataset_config configs/pretrain.json --ema --backbone timm_tf_efficientnet_b3_ns --lr_backbone 5e-5 --batch_size 4 --output-dir ./mdetr/pretrain_batch_4

• Did you change anything to the dataset?

No, I didn't change anything in the dataset.

• Have you tried running it on one gpu first?

I have tried running the same command with --nproc_per_node=1 on the same machine and got the same error. However, I tried running a distributed training on PCs equipped with single GPUs and connected with LAN. The training started successfully.

• Have you tried running on cpu first?

No, I didn't try that.

from mdetr.