zcyang / imageqa-san Goto Github PK

Hello, I would like to implement this network on Torch but I'm struggling a bit so I thought to train at first with the implementation that you guys are uploaded here. Because I don't have much experience with theano and also I have a limited time I would really apprecated if you guys uploaded the pre-processing scripts for both image feature extraction and dataset splits or at least the trainval_feat.h5 and train/val1/val2.pkl files.

On my system, if I defined a float variable, it's default type is float64. When I run "san_att_conv_twolayer.py", there shows error: 'An update must have the same type as the original shared variable (shared_var=conv_unigram_b, shared_var.type=TensorType(float32, vector), update_val=Elem
wise{sub,no_inplace}.0, update_val.type=TensorType(float64, vector)).' What can I do for this error?

I started the training as mentioned in the readme file. This was the last section of the error message i got. Please help

File "/usr/lib/python2.7/subprocess.py", line 711, in init
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1235, in _execute_child
self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

Could you update the project with the code on how to get the features from the Caffe VGGNet-19 model? In preprocess_data.py, it reads:

train_feat_file = 'train2014_fixed_fc7_feat.pkl'
val_feat_file = 'val2014_fixed_fc7_feat.pkl'

but there's no explanation on how to get these features. Could you elaborate? I guess they were obtained by the Caffe model referenced in feature_extract.py, but the commented section is incomplete.

In san_att_lstm_twolayer.py, we can see that the learning rate is 0.05 initially.
In function get_lr(), learning rate will dacay like this

options['lr'] * (options['gamma'] ** power)

However options['gamma'] is equals to 1, which means that the learning rate will not decay.

options['gamma'] = 1

This is the part I'm wondering.

The issue is same as the TITLE!

I'm trying to reproduce the results from DAQAUR-REDUCED dataset as mentioned in paper but it seems the model can't get an accuracy above 40% . The paper states an accuracy of 46.2% with SAN(2, LSTM) model. It would be nice if you could upload the pre-processed files from this dataset also and the options you used for training.

Thank you,
Vasilis Lioutas

Hi, I have downloaded the vqa data (the link has been provided) and run the training. But the pkl file deserialized are some arrays, so I don't known how to distinguish the type of yes/no, number and other. How can I get the result of specific type of question?

Thank you,
WeimingZhang

@zcyang
Hi,
The paper mentioned that it evaluated the best model, SAN(2, CNN), on the standard test server and report the results. How to evaluate are very very important to me! It is possible to provide the evaluated script or specific detail for evaluating?
Thanks a lot!

hi zcyang

there is only training script.

is it possible to provide simple test script?

thanks :)

when I train the model , the output are as following:
2016-07-09 16:22:17,087 - INFO - san_att_conv_twolayer - start training
2016-07-09 16:23:25,368 - INFO - data_provision_att_vqa - finished loading data
2016-07-09 16:23:31,685 - INFO - san_att_conv_twolayer - finished building model
Traceback (most recent call last):
File "san_att_conv_twolayer.py", line 265, in
train(options)
File "san_att_conv_twolayer.py", line 164, in train
= eval(options['optimization'])(shared_params, grad_buf, options)
File "/home/rongerhu/iQA/imageqa-san-master/src/optimization_weight.py", line 33, in sgd
name = 'f_param_update')
File "/home/rongerhu/anaconda2/lib/python2.7/site-packages/theano/compile/function.py", line 320, in function
output_keys=output_keys)
File "/home/rongerhu/anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 442, in pfunc
no_default_updates=no_default_updates)
File "/home/rongerhu/anaconda2/lib/python2.7/site-packages/theano/compile/pfunc.py", line 207, in rebuild_collect_shared
raise TypeError(err_msg, err_sug)
TypeError: ('An update must have the same type as the original shared variable (shared_var=conv_unigram_b, shared_var.type=TensorType(float32, vector), update_val=Elemwise{sub,no_inplace}.0, update_val.type=TensorType(float64, vector)).', 'If the difference is related to the broadcast pattern, you can call the tensor.unbroadcast(var, axis_to_unbroadcast[, ...]) function to remove broadcastable dimensions.')

Thank you for your contribution.
When I need to download your data, it failed in the google drive link, could you please update the file?

I implemented this network in Keras, but got very bad results. After looking for the differences with your Theano codes, I found that you set the learning rate of embedding layer as a very big value 80.
options['w_emb_lr'] = numpy.float32(80)
I tried to comment this line and the results of your Theano version dropped. It turns out this option is very important. Could you please explain the reason for this option? I will be grateful if you could give a suggestion of a paper related to this issue!
Thank you very much!

rzai@rzai00:/prj/imageqa-san/src$ CUDA_VISIBLE_DEVICES=1 PYTHONPATH=/prj/imageqa-san/src python san_att_conv_twolayer.py
Using gpu device 0: GeForce GTX 1080 (CNMeM is disabled, cuDNN 5105)
2016-12-20 11:29:17,803 - INFO - san_att_conv_twolayer - OrderedDict([('data_path', '../data_vqa'), ('feature_file', 'trainval_feat.h5'), ('expt_folder', '../expt'), ('model_name', 'imageqa'), ('train_split', 'trainval1'), ('val_split', 'val2'), ('shuffle', True), ('reverse', False), ('sample_answer', True), ('num_region', 196), ('region_dim', 512), ('n_words', 13746), ('n_output', 1000), ('combined_num_mlp', 1), ('combined_mlp_drop_0', True), ('combined_mlp_act_0', 'linear'), ('sent_drop', False), ('use_tanh', False), ('use_unigram_conv', True), ('use_bigram_conv', True), ('use_trigram_conv', True), ('use_attention_drop', False), ('use_before_attention_drop', False), ('n_emb', 500), ('n_dim', 500), ('n_image_feat', 512), ('n_common_feat', 500), ('num_filter_unigram', 256), ('num_filter_bigram', 512), ('num_filter_trigram', 512), ('n_attention', 512), ('init_type', 'uniform'), ('range', 0.01), ('std', 0.01), ('init_lstm_svd', False), ('optimization', 'sgd'), ('batch_size', 100), ('lr', 0.1), ('w_emb_lr', 80.0), ('momentum', 0.89999998), ('gamma', 1), ('step', 10), ('step_start', 100), ('max_epochs', 50), ('weight_decay', 0.0005), ('decay_rate', 0.99900001), ('drop_ratio', 0.5), ('smooth', 9.9999999e-09), ('grad_clip', 0.1), ('disp_interval', 10), ('eval_interval', 1000), ('save_interval', 500)])
2016-12-20 11:29:17,803 - INFO - san_att_conv_twolayer - start training
2016-12-20 11:37:40,671 - INFO - data_provision_att_vqa - finished loading data
Traceback (most recent call last):
File "san_att_conv_twolayer.py", line 265, in
train(options)
File "san_att_conv_twolayer.py", line 116, in train
= build_model(shared_params, options)
ValueError: need more than 7 values to unpack
rzai@rzai00:~/prj/imageqa-san/src$