kaleidophon / greedy-gerbil Goto Github PK

The data set class now also reads in the image features and support the pytorch Dataset class. However, I discovered a logical error in how I create the one-hot vectors for all the data sets. I will have to correct this one tomorrow before I can check off that box in the milestone issue.

The training of the RNN model (on branch "rnn") currently suffers does not support batches due to varying question lengths.
Current, the workaround is to load a batch and then iterate inside the batch. Not as efficient, but the padding strategies I tried weren't successful.

The strategy was to basically take the maximum question length in the current batch and pad all questions that are shorter with -1.
Unfortunately, this clashes with the embedding layer since there's no word index -1.
Padding after embedding is also not easily possible since it would require casting the question variable to something mutable and then recasting it as a Variable and the optimizer really doesn't like this.

Pytorch does seem to have a padding routine somewhere, but so far I haven't gotten it to run (and they also don't use it in their seq2seq example).

Anyway, it's running now, but quite slow.

• Get BoW model working
  • Finish data preparation in regard to image features
  • Adjust model to input data
  • Training
  • Evaluation
    • Collect most difficult/easiest questions
    • Collect most difficult/easiest images
• Brainstorm other models
• Report
  • Initialize report structure

https://www.overleaf.com/12314077tppzydqjnggh#/46825448/