Concerns and Clarifications Regarding MCQ to Generation Task Conversion about flipped-vqa HOT 3 CLOSED

First, what is the difference between vqa_placeholder_mask and vqa_label?
I think you can use vqa_label as it is, but simply changing the answer part of the input text (e.g., (A) -> playing soccer).

Also, if I understand correctly, mean pooling is applied to embeddings across the sequence length of the answer.
However, generally, the loss is computed for each single token (embedding) of the answer sequence.
You may refer to qav_loss to understand how this works.
Then, you may not need to similarity calculation process.

Finally, if you convert the MCQ setting to the generation task which still includes multiple options, I conjecture that including the options in the input text is more appropriate.
However, in my own preliminary experiment, the MCQ setting shows slightly better performance than the generation task, which might get better through hyperparamter tuning, when handling multiple options.

from flipped-vqa.

Thank you for the reply,

First, what is the difference between vqa_placeholder_mask and vqa_label? I think you can use vqa_label as it is, but simply changing the answer part of the input text (e.g., (A) -> playing soccer).

Yes, you are right. The end result of this operation is still similar to what you mentioned.

Also, if I understand correctly, mean pooling is applied to embeddings across the sequence length of the answer. However, generally, the loss is computed for each single token (embedding) of the answer sequence. You may refer to qav_loss to understand how this works. Then, you may not need to similarity calculation process.

I haven't removed the original loss computation. However, I still want to add the similarity computation to provide more meaningful analysis to my work.

Finally, if you convert the MCQ setting to the generation task which still includes multiple options, I conjecture that including the options in the input text is more appropriate. However, in my own preliminary experiment, the MCQ setting shows slightly better performance than the generation task, which might get better through hyperparamter tuning, when handling multiple options.

I agree that providing the options would allow the model to perform better, but I would like to see how the model performs in a pure generation task without the options been given at all.

I am very concerned about the issue that I'm encountering that makes the model not giving meaningful answers.

I would appreciate it if you provide the inference script as well.

Thanks!

from flipped-vqa.

You may simply remove train_one_epoch() in train.py and add --resume ./your/own/checkpoint.pth to your running command for the inference.

from flipped-vqa.