How to use Pre-trained Models? about vl-bert HOT 5 CLOSED

Yes, you can fine-tune VL-BERT for you task by simply adding a classification head on top of the output feature of first token [CLS] and fine-tuning it together with VL-BERT.
For how to load data and conduct fine-tuning, you can follow our code for downstream tasks (e.g., VQA, RefCOCO+, etc.).

from vl-bert.

@jackroos Thanks Jack, I do have a pre-trained Faster RCNN setup. This is based on the same caffe model you have in the instructions. I generate 100 boxes with nms=0.5 and confidence_threshold = 0.2,
Q1. Would it be possible for you to post 1 image and it's generated boxes, features here so I can verify if my setup is runnable.

Q2. Also I believe I need to use the same FRCNN as you did in case I want to use pretrained models. changing the FRCNN to SSD or any other detector will not work without retrained.

from vl-bert.

@jackroos Would this work even if I have no precomputed features/boxes for the images. It peeked inside vqa/data//datasets/vqa.py and it seems we need to have precomputed stuff.

@webYFDT Did you make it work?

from vl-bert.

@faizanahemad We need precomputed boxes in our VL-BERT. You can use pre-trained Faster RCNN to compute the boxes, following our instructions on preparing Conceptual Captions dataset.

from vl-bert.

@faizanahemad Sorry for the late reply, for examples of the image and generated boxes, maybe you can refer to the caffe bottom-up-attention repo, there are pre-computed boxes and features for coco images. And for your second question, the answer is yes, you need to use the same detector to extract visual features while using the FRCNN in VL-BERT.

from vl-bert.