Nan question in SupConLoss about ts-tcc HOT 4 CLOSED

Hi,
thanks for reporting this. This issue is more related to the CA-TCC work, where we use the SupConLoss.

To summarize what I understood:
you mean that the process of subtracting the maximum value from each row in the anchor_Dot_Contrast tensor can lead to some extremely negative values in the logits tensor, especially when there's a large disparity between the maximum value and other values in a particular row.
As a consequence, the exponential operation (torch.exp(logits)) and (torch.log(exp_logits.sum(1, keepdim=True))), can result in numerical instability, leading to values of NaN for the loss.

I think you are right, and this can be a valid point. We can address it in the implementation in several ways. For example, we can do clamping to the values of logits before performing the exponential operation. This can be to a predefined range.
Another way is to add a small constant before taking the logarithm to prevent the logarithm of a value close to zero.

It would be very beneficial if you can post the whole error message so that I can check and debug it properly.

from ts-tcc.

Yes, that's what I mean. During the training process, the loss becomes Nan, resulting in all weights becoming Nan, which will not lead to python errors.

But I can provide information on debugging SupConLoss:

• When entering 'forward':
  
• Before subtracting the maximum value:
  
• After subtracting the maximum value:
  
• exp_logits = torch.exp(logits) * logits_mask After exponential operation, it will become zero.
  
• When calculating log_prob, some calculation results become Nan due to -torch. log Nan appeared in the final calculation of the loss:
  

My current solution is to add 'eps=1e-15' when calculating the row sum of 'exp_logits' to avoid zero. But will this lead to situations where losses cannot converge?

from ts-tcc.

If the eps I set is too small, this error message will still appear:

from ts-tcc.

Thank you for your help! Perhaps I can use some methods to alleviate this problem myself.

from ts-tcc.