Batch formation potentially causing false negatives about vsepp HOT 1 CLOSED

This is a simplification that in practice does not hurt the training at the scale of MS-COCO.

Particularly, if you look at the loss over the whole mini-batch, the incorrect terms cancel out. In your example, the loss for (i_5, c_19) says I want image i_5 to be closer to c_5 than to c_19 and (i_19, c_5) which is in fact (i_5, c_5) says I want i_5 to be closer to c_19 than to c_5. The gradients from these terms are theoretically exactly the opposite of each other and would cancel out. This is true for both portions of the loss.

Such a simplification would hurt though if the probability of sampling such opposing pairs is high. In that case, the gradient from one mini-batch is accumulated over only a few effective examples and hence the variance of the estimate of the gradient would be high.

Just as a simple test, I tried using a mask to filter out such opposing terms but it did not help.

Besides, this simplification has been typically done in previous work. Take a look at these for example:
https://github.com/ryankiros/visual-semantic-embedding/blob/master/homogeneous_data.py
https://github.com/ryankiros/visual-semantic-embedding/blob/master/datasets.py

from vsepp.