questions on dataset construction about vsepp HOT 3 CLOSED

About start and end tokens, I guess in the current implementation they are not really useful. They are useful if the implementation of sentence encoder can't handle variable lengths in a mini-batch or if it is not parallelized over a mini-batch. But it's not the case in this with the pytorch tools used.

It is true that the validation set can have effectively 1000 images in it. But the small validation set is only used for early stopping. So maybe as a result we don't choose the best model for test set.

from vsepp.

@fartashf Thanks a lot for your reply. I have one more question and I would really appreciate a reply. I have a deeper network for embedding the visual and text features, based on your code (just added more layers). Consider it to be 3 layers for each modality. I am using a ReLU activation function in between all these layers, except for the last one which will be the input to the similarity function. I found out that the results are extremely horrible (0.1, 0.5, 0.3) for R@1, R@5, R@10. But it works well when I remove the ReLU activation between the layers. Any idea why non-linearity harms image-text retrieval?

from vsepp.

Maybe consider these changes:

• Try the sum loss first. Max loss is not always good at the beginning of the optimization.
• Make sure initialization is done correctly. Weights are not too small or too large.
• Make sure fine-tuning is not set.
• Try adding layers to only one modality. See if the problem is with training both together.
• Try adding normalization before each linear layer.

from vsepp.