Form of the h function for infinite dataset about repdistiller HOT 2 CLOSED

Hi, @brotherofken ,

Thanks for your interest. For the eq. 19 in the paper, h will automatically work if you set N and M as the number of negatives to pair each positive and the number of the dataset size, respectively. h approaches to 1 at the beginning, bu will be adjusted very quickly as the training proceeds. This is how NCE works.

Angular similarity might also work, however loses the spirit of posterior probability in NCE.

from repdistiller.

Thanks for the quick response!

I read the paper carefully and found that I missed that the temperature in (19) has to be quite low (0.02-0.3 in your experiments) to compensate for the small value of the N/M ratio.
That became clear now. Thanks!

from repdistiller.