Inference on custom raw video about rmn HOT 4 CLOSED

First, you need to extract the video features and then feed these features to the model.

from rmn.

About extracting video features, I have extracted i3d features for my video.
Which layer of InceptionResnetV2 should be used for extracting frame features?
Also for BUTD, I referred the issue and it says the script is modified, so is the new script sufficient for usage?

from rmn.

1. IRV2 extractor:

class AppearanceEncoder_inceptionresnetv2(nn.Module):
    def __init__(self):
        super(AppearanceEncoder_inceptionresnetv2, self).__init__()
        IRV2 = InceptionResNetV2(num_classes=1001)
        # print('IRV2:\n', IRV2)
        IRV2.load_state_dict(torch.load(opt.IRV2_checkpoint))
        modules = list(IRV2.children())[:-1]  # delete the last fc layer.
        self.IRV2 = nn.Sequential(*modules)
        # print('IRV2:\n', self.IRV2)

    def forward(self, images):
        """Extract feature vectors from input images."""
        with torch.no_grad():
            features = self.IRV2(images)
        features = features.reshape(features.size(0), -1)
        # print(features.size())
        return features

Image preprocessing:

    image -= np.array([0.5, 0.5, 0.5])
    image /= np.array([0.5, 0.5, 0.5])

2.About BUTD, just feed your image to the pre-trained BUTD model and you will get 36 region features. What I modifed is preprossing MSRVTT and MSVD, and saving the extracted features to h5 file, the core code is remained.

from rmn.

Hi tgc

Great work! Thanks for sharing
Would like to ask if you could share your codes on feature extraction using i3d, irv2 and butd? Im not sure how to extract these video features.

Thank you

from rmn.