Training Loss is Puzzling about mvsnet_pytorch HOT 12 CLOSED

Hello @xy-guo . Thanks for your amazing work. I have re-trained the network using the same cofiguration, but the training loss don't converge.
I don't think the training loss is very plausible. Do you have this problem??
Regards.

from mvsnet_pytorch.

hi, did you continue training? What does the final loss curve look like? Also does the depth in the image tab seem improving? My training loss also fluctuates a lot.

from mvsnet_pytorch.

Hello @xy-guo . Thanks for your amazing work. I have re-trained the network using the same cofiguration, but the training loss don't converge.
I don't think the training loss is very plausible. Do you have this problem??
Regards.

Have you ever try to use larger batchsize?

from mvsnet_pytorch.

I use 8 gpus to train the model. It seems your training loss is converging, try using larger smoothing weight in tensorboard.

from mvsnet_pytorch.

Do you have the final metrics of abs_depth_error, thres2mm_error, thres4mm_error, thres8mm_error? I want to compare with my results, thank you.

from mvsnet_pytorch.

@xy-guo

from mvsnet_pytorch.

Sorry I have lost the final metrics information. @kwea123

from mvsnet_pytorch.

@kwea123 Have you try bigger batchsize? What's the difference with the training loss of batchsize=1?

from mvsnet_pytorch.

The model consumes too much memory so I cannot try bigger batchsize (batchsize=1 requires ~8GB). You will need multiple gpus.

from mvsnet_pytorch.

@QTODD Hi, have you solve the problem that the training loss is not plausible?

from mvsnet_pytorch.

@xy-guo Do you use 8 GPUS to test?

from mvsnet_pytorch.

I tried, even 8GPUs with batchsize 4 results still in fluctuating losses. I think it's insolvable then; although it doesn't worsen the model performance..

from mvsnet_pytorch.