SimGCL: InfoNCE Loss Calculation Batch vs. Entire Embeddings about sslrec HOT 2 CLOSED

Aha, I see.
Thanks for your reply :)

from sslrec.

Hi 👋!

Thank you for your interest and good question! Indeed, using either the batch embeddings or the entire embeddings to form the negative pairs in the InfoNCE loss is okay here.

However, based on the theory of neural mutual information estimation, having a larger number of negative pairs contributes to a more stable estimation of mutual information during training. Therefore, in our implementation of SSLRec, we utilize the entire embeddings to form negative pairs with anchor views not only in SimGCL but also in methods like SGL and HCCF, ensuring a fair implementation.

I hope the above answers can help with your confusion :)

Best regards,
Xubin Ren

from sslrec.