2023-08-29 12:51:48,581 INFO cfg_file cfgs/scannet_models/spconv_clip_base15_caption_adamw.yaml
2023-08-29 12:51:48,583 INFO batch_size 4
2023-08-29 12:51:48,584 INFO epochs 128
2023-08-29 12:51:48,586 INFO workers 4
...
2023-08-29 12:51:55,038 INFO **********************Start training scannet_models/spconv_clip_base15_caption_adamw(default)**********************
2023-08-29 12:53:11,734 INFO Epoch [1/128][20/1201] LR: 0.004, ETA: 6 days, 18:09:08, Data: 0.08 (0.42), Iter: 1.91 (3.80), Accuracy: 0.56, loss_seg=1.61, binary_loss=0.46, caption_view=0.17, caption_entity=0.15, loss=2.38, n_captions=13.2(13.1)
2023-08-29 12:53:51,830 INFO Epoch [1/128][40/1201] LR: 0.004, ETA: 5 days, 3:52:20, Data: 0.05 (0.25), Iter: 1.97 (2.90), Accuracy: 0.54, loss_seg=1.55, binary_loss=0.29, caption_view=0.16, caption_entity=0.14, loss=2.15, n_captions=13.2(12.8)
2023-08-29 12:54:29,663 INFO Epoch [1/128][60/1201] LR: 0.004, ETA: 4 days, 13:28:58, Data: 0.07 (0.19), Iter: 2.13 (2.56), Accuracy: 0.61, loss_seg=1.42, binary_loss=0.36, caption_view=0.17, caption_entity=0.15, loss=2.11, n_captions=16.2(12.8)
2023-08-29 12:55:09,310 INFO Epoch [1/128][80/1201] LR: 0.004, ETA: 4 days, 7:14:50, Data: 0.09 (0.16), Iter: 2.17 (2.42), Accuracy: 0.67, loss_seg=1.28, binary_loss=0.35, caption_view=0.19, caption_entity=0.17, loss=1.99, n_captions=18.8(12.7)
2023-08-29 12:55:47,967 INFO Epoch [1/128][100/1201] LR: 0.004, ETA: 4 days, 3:05:21, Data: 0.10 (0.14), Iter: 2.03 (2.32), Accuracy: 0.66, loss_seg=1.24, binary_loss=0.17, caption_view=0.16, caption_entity=0.15, loss=1.72, n_captions=13.2(12.7)
2023-08-29 12:56:31,885 INFO Epoch [1/128][120/1201] LR: 0.004, ETA: 4 days, 2:10:41, Data: 0.06 (0.13), Iter: 1.77 (2.30), Accuracy: 0.70, loss_seg=1.20, binary_loss=0.21, caption_view=0.14, caption_entity=0.10, loss=1.65, n_captions=7.5(12.9)
2023-08-29 12:57:11,898 INFO Epoch [1/128][140/1201] LR: 0.004, ETA: 4 days, 0:20:09, Data: 0.08 (0.12), Iter: 1.93 (2.26), Accuracy: 0.63, loss_seg=1.25, binary_loss=0.19, caption_view=0.15, caption_entity=0.14, loss=1.73, n_captions=9.8(12.9)
2023-08-29 12:57:50,126 INFO Epoch [1/128][160/1201] LR: 0.004, ETA: 3 days, 22:28:16, Data: 0.07 (0.11), Iter: 2.11 (2.21), Accuracy: 0.67, loss_seg=1.21, binary_loss=0.19, caption_view=0.16, caption_entity=0.15, loss=1.71, n_captions=12.0(12.8)
2023-08-29 12:58:31,512 INFO Epoch [1/128][180/1201] LR: 0.004, ETA: 3 days, 21:46:22, Data: 0.07 (0.11), Iter: 1.72 (2.20), Accuracy: 0.61, loss_seg=1.41, binary_loss=0.28, caption_view=0.16, caption_entity=0.09, loss=1.93, n_captions=9.0(12.9)
2023-08-29 12:59:12,194 INFO Epoch [1/128][200/1201] LR: 0.004, ETA: 3 days, 21:03:14, Data: 0.07 (0.11), Iter: 2.43 (2.18), Accuracy: 0.54, loss_seg=1.49, binary_loss=0.26, caption_view=0.18, caption_entity=0.17, loss=2.09, n_captions=18.0(13.0)
2023-08-29 12:59:51,518 INFO Epoch [1/128][220/1201] LR: 0.004, ETA: 3 days, 20:12:49, Data: 0.05 (0.10), Iter: 1.69 (2.16), Accuracy: 0.74, loss_seg=0.96, binary_loss=0.26, caption_view=0.14, caption_entity=0.10, loss=1.46, n_captions=7.8(12.9)
2023-08-29 13:00:32,747 INFO Epoch [1/128][240/1201] LR: 0.004, ETA: 3 days, 19:50:16, Data: 0.10 (0.10), Iter: 2.08 (2.15), Accuracy: 0.60, loss_seg=1.40, binary_loss=0.28, caption_view=0.16, caption_entity=0.15, loss=2.01, n_captions=12.5(12.9)
2023-08-29 13:01:13,779 INFO Epoch [1/128][260/1201] LR: 0.004, ETA: 3 days, 19:29:12, Data: 0.06 (0.10), Iter: 2.38 (2.15), Accuracy: 0.62, loss_seg=1.29, binary_loss=0.23, caption_view=0.18, caption_entity=0.16, loss=1.86, n_captions=19.0(12.9)
2023-08-29 13:01:52,565 INFO Epoch [1/128][280/1201] LR: 0.004, ETA: 3 days, 18:51:06, Data: 0.06 (0.09), Iter: 1.83 (2.13), Accuracy: 0.66, loss_seg=1.18, binary_loss=0.27, caption_view=0.15, caption_entity=0.12, loss=1.72, n_captions=9.8(12.9)
Traceback (most recent call last):
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 322, in reduce_storage
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files
2023-08-29 13:02:32,142 INFO Epoch [1/128][300/1201] LR: 0.004, ETA: 3 days, 18:24:18, Data: 0.08 (0.09), Iter: 1.62 (2.12), Accuracy: 0.52, loss_seg=1.61, binary_loss=0.20, caption_view=0.16, caption_entity=0.13, loss=2.10, n_captions=12.0(12.9)
2023-08-29 14:35:17,045 INFO **********************Start logging**********************
2023-08-29 14:35:17,045 INFO CUDA_VISIBLE_DEVICES=ALL
2023-08-29 14:35:17,046 INFO cfg_file cfgs/scannet_models/spconv_clip_base15_caption_adamw.yaml
2023-08-29 14:35:17,058 INFO batch_size 1
2023-08-29 14:35:17,060 INFO epochs 128
2023-08-29 14:35:17,062 INFO workers 1
...
2023-08-29 14:35:24,926 INFO **********************Start training scannet_models/spconv_clip_base15_caption_adamw(default)**********************
2023-08-29 14:35:43,175 INFO Epoch [1/128][20/4804] LR: 0.004, ETA: 6 days, 9:57:34, Data: 0.01 (0.03), Iter: 0.80 (0.90), Accuracy: 0.57, loss_seg=1.62, binary_loss=0.61, caption_view=0.08, caption_entity=0.06, loss=2.37, n_captions=11.0(11.8)
2023-08-29 14:35:57,832 INFO Epoch [1/128][40/4804] LR: 0.004, ETA: 5 days, 19:34:41, Data: 0.02 (0.03), Iter: 0.67 (0.82), Accuracy: 0.77, loss_seg=1.19, binary_loss=0.52, caption_view=0.04, caption_entity=0.00, loss=1.75, n_captions=3.0(12.0)
2023-08-29 14:36:12,468 INFO Epoch [1/128][60/4804] LR: 0.004, ETA: 5 days, 14:40:01, Data: 0.03 (0.03), Iter: 0.80 (0.79), Accuracy: 0.59, loss_seg=1.53, binary_loss=0.60, caption_view=0.11, caption_entity=0.07, loss=2.32, n_captions=15.0(12.6)
2023-08-29 14:36:25,845 INFO Epoch [1/128][80/4804] LR: 0.004, ETA: 5 days, 9:36:36, Data: 0.01 (0.02), Iter: 0.28 (0.76), Accuracy: 0.43, loss_seg=2.38, binary_loss=0.31, caption_view=0.10, caption_entity=0.07, loss=2.86, n_captions=13.0(12.4)
2023-08-29 14:36:39,422 INFO Epoch [1/128][100/4804] LR: 0.004, ETA: 5 days, 6:51:36, Data: 0.02 (0.02), Iter: 0.46 (0.74), Accuracy: 0.43, loss_seg=1.95, binary_loss=0.26, caption_view=0.09, caption_entity=0.07, loss=2.37, n_captions=10.0(12.9)
2023-08-29 14:36:54,429 INFO Epoch [1/128][120/4804] LR: 0.004, ETA: 5 days, 7:03:13, Data: 0.02 (0.02), Iter: 0.70 (0.74), Accuracy: 0.78, loss_seg=0.90, binary_loss=0.33, caption_view=0.08, caption_entity=0.04, loss=1.34, n_captions=9.0(12.9)
2023-08-29 14:37:07,454 INFO Epoch [1/128][140/4804] LR: 0.004, ETA: 5 days, 4:47:05, Data: 0.02 (0.03), Iter: 0.79 (0.73), Accuracy: 0.64, loss_seg=1.29, binary_loss=0.16, caption_view=0.06, caption_entity=0.03, loss=1.55, n_captions=7.0(12.8)
2023-08-29 14:37:21,984 INFO Epoch [1/128][160/4804] LR: 0.004, ETA: 5 days, 4:40:53, Data: 0.03 (0.03), Iter: 0.91 (0.73), Accuracy: 0.70, loss_seg=1.09, binary_loss=0.18, caption_view=0.11, caption_entity=0.08, loss=1.47, n_captions=15.0(13.1)
2023-08-29 14:37:35,895 INFO Epoch [1/128][180/4804] LR: 0.004, ETA: 5 days, 4:02:25, Data: 0.01 (0.02), Iter: 0.45 (0.73), Accuracy: 0.67, loss_seg=1.09, binary_loss=0.24, caption_view=0.06, caption_entity=0.00, loss=1.39, n_captions=4.0(13.1)
2023-08-29 14:37:49,432 INFO Epoch [1/128][200/4804] LR: 0.004, ETA: 5 days, 3:11:09, Data: 0.03 (0.02), Iter: 0.70 (0.72), Accuracy: 0.78, loss_seg=0.98, binary_loss=0.14, caption_view=0.09, caption_entity=0.06, loss=1.27, n_captions=10.0(13.1)
2023-08-29 14:38:02,928 INFO Epoch [1/128][220/4804] LR: 0.004, ETA: 5 days, 2:27:42, Data: 0.01 (0.02), Iter: 0.55 (0.72), Accuracy: 0.83, loss_seg=0.75, binary_loss=0.26, caption_view=0.09, caption_entity=0.02, loss=1.12, n_captions=7.0(13.3)
2023-08-29 14:38:18,365 INFO Epoch [1/128][240/4804] LR: 0.004, ETA: 5 days, 3:14:10, Data: 0.03 (0.02), Iter: 0.45 (0.72), Accuracy: 0.52, loss_seg=2.06, binary_loss=0.30, caption_view=0.11, caption_entity=0.12, loss=2.59, n_captions=24.0(13.4)
2023-08-29 14:38:31,951 INFO Epoch [1/128][260/4804] LR: 0.004, ETA: 5 days, 2:40:36, Data: 0.02 (0.02), Iter: 0.39 (0.72), Accuracy: 0.78, loss_seg=0.89, binary_loss=0.15, caption_view=0.07, caption_entity=0.03, loss=1.15, n_captions=6.0(13.2)
2023-08-29 14:38:45,580 INFO Epoch [1/128][280/4804] LR: 0.004, ETA: 5 days, 2:13:03, Data: 0.02 (0.02), Iter: 0.49 (0.72), Accuracy: 0.44, loss_seg=2.29, binary_loss=0.32, caption_view=0.11, caption_entity=0.05, loss=2.77, n_captions=12.0(13.1)
2023-08-29 14:38:59,079 INFO Epoch [1/128][300/4804] LR: 0.004, ETA: 5 days, 1:44:38, Data: 0.02 (0.02), Iter: 0.70 (0.71), Accuracy: 0.69, loss_seg=1.06, binary_loss=0.17, caption_view=0.09, caption_entity=0.07, loss=1.39, n_captions=10.0(13.0)
2023-08-29 14:39:13,142 INFO Epoch [1/128][320/4804] LR: 0.004, ETA: 5 days, 1:37:37, Data: 0.02 (0.02), Iter: 0.95 (0.71), Accuracy: 0.72, loss_seg=1.03, binary_loss=0.86, caption_view=0.13, caption_entity=0.13, loss=2.15, n_captions=28.0(13.1)
Traceback (most recent call last):
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 321, in reduce_storage
RuntimeError: unable to open shared memory object </torch_21979_3890372270> in read-write mode
2023-08-29 16:39:54,976 INFO Epoch [1/128][920/2402] LR: 0.004, ETA: 4 days, 2:55:56, Data: 0.05 (0.04), Iter: 0.87 (1.16), Accuracy: 0.77, loss_seg=0.80, binary_loss=0.28, caption_view=0.13, caption_entity=0.11, loss=1.32, n_captions=13.0(13.1)
Traceback (most recent call last):
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/queues.py", line 236, in _feed
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/reduction.py", line 51, in dumps
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 322, in reduce_storage
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/reduction.py", line 194, in DupFd
File "/home/usrs/wangjuan/anaconda3/envs/PLA/lib/python3.7/multiprocessing/resource_sharer.py", line 48, in __init__
OSError: [Errno 24] Too many open files