Validation data load error. Should num_worker be 0? about snake HOT 7 CLOSED

No, it looks like not a problem of dataloader worker.
Maybe there is some problem in your evaluation data.
Use python run.py --type evaluate --config_file your_config_file_path to reproduce this bug.
It occurs at File "/ldap_shared/home/s_zyt/inseg/code/snake/lib/datasets/collate_batch.py", line 7

from snake.

Yes, actually I use the code reproduced this bug.

python run.py --type evaluate --cfg_file configs/sbd_snake.yaml model custom_model train.dataset CocoTrain test.dataset CocoVal

But I have no idea why would this happend, the dataset I use can be trained on other model like the mmdet version mask rcnn.

from snake.

I think maybe the image passed into function default_collate must have the same shape.
My problem is their shapes are not the same.

>>> [b['inp'].shape for b in batch]
[(3, 544, 800), (3, 480, 608), (3, 544, 800), (3, 544, 800), (3, 544, 800), (3, 544, 800), (3, 544, 800), (3, 448, 608), (3, 544, 800), (3, 544, 800), (3, 544, 800), (3, 544, 800), (3, 544, 800), (3, 448, 608), ...]
>>> default_collate(a)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 544 and 480 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

How should I fix this problem?

from snake.

Set batch_size = 1 in test.
Our code does not support batch evaluation.

from snake.

I fixed this problem and the network training is on going.

But I think the training process is very slow and the GPU utilization rate is very low. I have two 2080ti GPU card, but the GPU-Util is 0% in most of the time.

Is that because of the CirConv not being fully optimized by the modern dl framework pytorch?
Or other problem.

I wander how long it will take for your training on coco or sbd dataset?
My dataset is far smaller than them but it also take almost two days.
training batchsize=16.

from snake.

Set a bigger num_workers.
The data loading of your training is very slow.

from snake.

The training on sbd takes about 8 hours.

from snake.