NAN loss after one epoch about fiery HOT 4 OPEN

Hey both of you, and sorry for the late answer. It is a known issue, and it seems like training the whole network from scratch leads to instability. The fix is to load pre-trained weight from a 1-timestep model (FIERY Static) first because training the whole future prediction model as discussed here: #8

from fiery.

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

I also tried the same setup and have the same issue. I tried to dig a little bit deeper and found out that running means and running variances of batch normalization are getting first to -infinity and then to nan. This also causes the loss to go to nan. The loss seems to get back from the nan-values, however, the model will still output -infinities for the segmentation.

Best regards,
Markus

from fiery.

Hey,

Thanks for sharing the Fiery model and article publicly, it was a great read.

After I train the model for epoch I got loss as negative value.Do you know the reason?

Best regards,
Cara

from fiery.

Hey Cara, the negative loss value is due to the adaptive weighting of the losses using uncertainty. There is an additional loss term that prevents the weight from growing too large (see page 5 https://arxiv.org/pdf/1705.07115.pdf)

from fiery.