lisaanne / eccv2016 Goto Github PK

Hi,

Thank you very much for your code. I would like to generate explanations using your models without retraining. May I please get required prototxt files for that? Or May I please know how can I generate those files?

Thank you in advance!

Hello and thank you for your work!
I am trying to reproduce the results from GVE but the URL:
https://people.eecs.berkeley.edu/~lisa_anne/generating_visual_explanations/data/ is now password protected.
Is there any other way to access the data files of CUB?
Thank you in advance

When I run the make_class_embedding.sh, the got a './make_class_embedding.sh: 行 22: 1950' error! Is it my dataset error?

using https://github.com/LisaAnne/ECCV2016/blob/master/utils/preprocess_captions.py
I can generate

descriptions_bird.test.fg.json using test.txt
descriptions_bird.val.fg.json using val.txt
descriptions_bird.train_noCub.fg.json using train_noCub.txt
test.txt, val.txt train_noCub.txt are provided at https://drive.google.com/drive/folders/1nU9ATTLHAM6_jz-K6hoVlDzNrFcOXtyH (data.zip)

however, there is not any train.txt? what is train_noCub.txt?
could you please help me in this regard?

Hi Lisa,

Thanx a lot for your code. I'm running into a problem while using your code. For computing the class embeddings, one needs to run the description model. The steps in your README direct one to use the make_class_embeddings.sh file, which runs the following code :

python utils/extract_train_val.py --model_name $model
--LM_net $word_net
--dataset_name $dataset_name
--split_name $split_name
--vocab $vocab
--image_net $image_net
--size_input_feature $size_input_feature
--prev_word_restriction
--precomputed_h5 data/CUB_feature_010517.p

There is no such utils/extract_train_val.py file in the repo. Moreover, this file isn't generated by the build_models.sh code either. So I'm confused, and am unable to use the code. Would be glad if you can clear this doubt.

Regards,
Saurabh

Hi Lisa,

We ran the script eval_explanation_label.sh and got the json file created in below path.
generated_sentences/birds_fg_test/gve_models/explanation-label_1006_pred/beam1/generation_result.json

We wanted to know if this is final output for explanation labels for the birds data set you have considered. And we need to extract the labels from this json to display them in a more readable format. Or is there any additional step which we are missing to generate explanation labels? Please advice.

Rgds,
Chandra.

Hello Lisa,

Thank you for sharing your code!
I was interested in understanding how you compute the class relevance scores (Similarity and Rank).
At the end of the README, you mention that in order to compute these scores one should run the "analyze_cider_scores.py" script.
However, I do not seem to find it in the repo...could you please point me to it?
(I have also looked for the "class_meteor_similarity_metric.py" and can't find it either.)

Thank you,
Isabel

We have downloaded your ECCV2016 code from your git repository and were able to successfully install the code. We were able to run the demo of coco-caption-eval in data/coco/coco-caption-eval successfully. Post that we executed the below steps from examples/ECCV2016 folder and were successful.

Getting Started

Please clone my git repo. My code for generating visual explanations is in /examples/ECCV2016. You will need to use my version of caffe.
Download data using the "download_data.sh" script. This will also preprocess the CUB sentences. All my ECCV 2016 models will be put in "gve_models"

But we got stuck on the building the models using below instructions.

Building the models

All the models are generated using NetSpec. Please build them by running "build_nets.sh". "build_nets.sh" will also generate bash scripts you can use to train models.

We are getting errors, details of which are given below. We are debugging from our end by understanding your implementation. But due to time constraint we wanted to check if you can through some light on these errors which can help us to proceed further.

./build_nets.sh
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 218, in
float(args.lstm_drop))
File "build_nets.py", line 130, in caption_classifier
model_train.caption_classifier(save_file_train, accuracy=False, loss=True, embed_drop=embed_drop, lstm_drop=lstm_drop)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 160, in caption_classifier
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 203, in
build_sentence_generation_model('definition', True, False)
File "build_nets.py", line 102, in build_sentence_generation_model
model_train.make_sentence_generation_net(save_file_train, accuracy=False, loss=True)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 116, in make_sentence_generation_net
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 205, in
build_sentence_generation_model('description', False, True)
File "build_nets.py", line 102, in build_sentence_generation_model
model_train.make_sentence_generation_net(save_file_train, accuracy=False, loss=True)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 116, in make_sentence_generation_net
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 207, in
build_sentence_generation_model('explanation-label', True, True)
File "build_nets.py", line 102, in build_sentence_generation_model
model_train.make_sentence_generation_net(save_file_train, accuracy=False, loss=True)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 116, in make_sentence_generation_net
self.write_net(save_file)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 211, in
sentence_generation_reinforce('explanation-dis', orig_proto='prototxt/description_train.prototxt', classify_model=args.classify_model, classify_weights=args.classify_weights, weights=args.weights, class_conditional=False, lw=80) #Loss weight parameter chosen by parameter search
File "build_nets.py", line 169, in sentence_generation_reinforce
model_train.lrcn_reinforce(save_name=save_file_train, RL_loss=RL_loss, lw=lw)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/reinforce.py", line 218, in lrcn_reinforce
self.write_net(save_name)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 213, in
sentence_generation_reinforce('explanation', orig_proto='prototxt/explanation-label_train.prototxt', classify_model=args.classify_model, classify_weights=args.classify_weights, weights=args.weights, class_conditional=True, lw=110) #Loss weight parameter chocen by parameter search
File "build_nets.py", line 169, in sentence_generation_reinforce
model_train.lrcn_reinforce(save_name=save_file_train, RL_loss=RL_loss, lw=lw)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/reinforce.py", line 218, in lrcn_reinforce
self.write_net(save_name)
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 74, in assign_proto
setattr(proto, name, val)
AttributeError: Assignment not allowed (no field "layer" in protocol message object).
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
.........EXECUTING CHILD...........
['/sbin/ldconfig', '-p']
In main and parsing arguments
Traceback (most recent call last):
File "build_nets.py", line 209, in
build_sentence_generation_deploy()
File "build_nets.py", line 72, in build_sentence_generation_deploy
model_train.make_sentence_generation_deploy()
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/lrcn.py", line 59, in make_sentence_generation_deploy
self.write_net('prototxt/deploy.prototxt')
File "/home/chandra/LisaAnne/ECCV2016-ECCV2016/examples/ECCV2016/build_net/caffe_net.py", line 32, in write_net
write_proto = self.n.to_proto()
File "../../python/caffe/net_spec.py", line 178, in to_proto
top._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 92, in _to_proto
return self.fn._to_proto(layers, names, autonames)
File "../../python/caffe/net_spec.py", line 153, in _to_proto
assign_proto(layer, k, v)
File "../../python/caffe/net_spec.py", line 64, in assign_proto
proto_item = getattr(proto, name).add()
AttributeError: 'LayerParameter' object has no attribute 'shape'

Hi there,

for me compilation fails with the following error message:

CXX/LD -o .build_release/tools/convert_imageset.bin
.build_release/lib/libcaffe.so: undefined reference tocaffe::SampleLayer::Forward_gpu(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&)
.b uild_release/lib/libcaffe.so: undefined reference to caffe::SampleLayer::Forward_gpu(std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&, std::vector<caffe::Blob, std::allocator<caffe::Blob> > const&)
collect2: error: ld returned 1 exit status
Makefile:564: recipe for target '.build_release/tools/convert_imageset.bin' failed
make: *** [.build_release/tools/convert_imageset.bin] Error 1

I was able to circumvent the error by declaring the mentioned function in ´sample_layer.cpp´, but that does not seem to be a real solution. Is this a known issue? I am just using the CPU for now, but would eventually like to run the model on a GPU aswell.