How can I use natural language queries to retrieve the source image？ about densecap HOT 3 OPEN

We don't have code for that in this repo, but it's relatively simple.

First use the extractFeatures method to get boxes and features for the database images:

https://github.com/jcjohnson/densecap/blob/master/densecap/DenseCapModel.lua#L285

Next run the LanguageModel (https://github.com/jcjohnson/densecap/blob/master/densecap/DenseCapModel.lua#L285) and LanguageModelCriterion (https://github.com/jcjohnson/densecap/blob/master/densecap/DenseCapModel.lua#L120) forward using the features and the query to compute the log-likelihood for the query on the extracted boxes.

Finally use these log-likelihoods to sort all boxes on all images.

from densecap.

@jcjohnson can open-world object detection be done in a similar way?
Also, how do you use the boxes and features from extractFeatures in self.nets.language_model:forward() (the function expects image vectors and gt labels) and self.crits.lm_crit:forward()?

from densecap.

Has anyone got a working demo of this by chance?

from densecap.