In the class EncoderSimilarity(nn.Module) of model.py,
we define the SGR and SAF modules separately:

if module_name == 'SGR':
    self.SGR_module = nn.ModuleList([GraphReasoning(sim_dim) for i in range(sgr_step)])
elif module_name == 'SAF'.
    self.SAF_module = AttentionFiltration(sim_dim)
else.
    raise ValueError('Invalid input of opt.module_name in opts.py')

In other words, you can only train either SGR or SAF respectively, and average these two similarities offline to get the results of independent learning in Table5.

For joint learning, you only need to define both SGR and SAF simultaneously as follows:

self.SGR_module = nn.ModuleList([GraphReasoning(sim_dim) for i in range(sgr_step)])
self.SAF_module = AttentionFiltration(sim_dim)

Activating both modules in the forward process and averaging their similarities online to get the final result, which is then fed into the Rank Loss function.
We experimentally found that there is no gain in this joint learning, as shown in Table 5.

Thanks for your excellent work, I am sincerely appreciative.
In the paper, I saw you train SGR and SAF model seperately, but I want to know how can I get the result of SGRAF? I didn't find how to get the result of SGRAF in your Github. Is it to add the similarity obtained by the SGR and SAF models on the test set? I'm looking forward to your reply, thank you again from the bottom of my heart

Hello thankyou so much for reviewing this issue and reply

When I run train.py ,the result's Loss is nan,what's the problem?

code evaluation.py seems to test on mscoco1k
how to test on 5k?

Hi, Could you please me how to understand the Similarity Pyramid(and Pyramid Spatial Window, different Pyramid Levels, etc.) which used in obtaining image feature that your paper memtioned???
In your released code, it was only region features extracted by Faster-RCNN(Bottom-up Attention)just as the Pioneers' work? I'm confused about that.
Thank you in Advance! :)

I am very interested in your work (SGRAF). I now encounter a visualization problem, that is, how to visualize the results of image retrieval text and the results of text retrieval image (as shown in the figure below). If it's convenient for you, please provide the code of visualization results. Thank you very much

!

Hello author, I'm reading your paper. In the paper, the implementation of formulas 1, 3, 4, 5 is given in github where the code is located? As for the learnable parameter W in formula 1, I could not find the corresponding position in the code. Looking forward to your answer, thank you

I found based on redme and found different dataset formats and then started evaluation first but directly,

请问作者有没有尝试过在flickr8k这样的小数据集上做验证，flickr30k对我的实验来说时间有点久

i have set the virtual environment of my pycharm as same as torch1.2.0 and python2.7. But i can't run through after my train when i want to evaluate the model. it shows indexerror with indices for array. Is something wrong with my environment or my data or some other probability? I tried to solve it with my friends but we found that the code is totally correct. But it still can't successfully eval on my computer. This problem had been haunted me for few days. Thank u.

Attempted to run evaluation.py using provided MS Coco models.

cpu (ie non-gpu) version of Python 3.6 branch

In evaluation.py, line 103, appears to be attempting to insert a record into array img_embs

Specific line is

img_embs[ids] = img_emb.data.cpu().numpy().copy()

This line throws an Error:

IndexError: too many indices for array

Using provided code (evaluation.py) and MS Coco models, ids appears to be a tuple
which prints as a list of integers

img_emb.data is a Tensor object, so the assignment to a numpy array img_embs appears to be an
attempted conversion of a Tensor to a numpy array, however, the actual intent of the assignment
and a work-around for the Error is unclear

Its documented code and the results from the associated paper are good, but unfortunately
the provided models are not working, and do not allow the paper results to be duplicated

Please publish an update to the code which works with provided MS Coco models

I am out of my depth in attempting to update this code.

Hello, I have a problem and want to ask for help. I tried to run your code, but I found that the loss of the model does not decrease and the evaluation index R1,R5,R10 does not increase and the index medr, meanr is very large

Hello, I am very interested in your work, can you share the code for visual query
Many thanks!

你好，作者，我最近看到您这篇论文实验细节中，用到了Flickr 30k数据集，还提取了其区域特征，我想请问一下在哪里能下载论文实施细节提到的Flickr 30k数据集区域特征，麻烦看到了，回复一下，急用，谢谢！

I downloaded your code and ran the code according to the default hyperparameters, but found that the loss did not decrease. Do I need to do some other operations before running this code?

Dear Professor Hiawen Diao,
I am sorry for troubling you. Your help so far has been extraordinary. I am sincerely appreciative.
I have been able to replicate your results. My supervisor said that is good. He asked if i can train my own model.

I have trouble with your data, and it may be my fault. If you could please talk about my data question. When you train a machine learning model, my understanding is that you need to include relevant documents in the training data. For example, include 4 of 5 coco captions for training, and hold one back for validation. If my understanding is correct, then I don't see where this happens in the code.

This is the last piece that my supervisor has asked for. Any clarification you can provide would be very helpful.

Thank you

Kent

何时会公布代码？