the readme.md says the GPU acceleration is optional and this is the only part that I find requires cutorch
here is the informaiton

/Users/Puddinord/torch/install/bin/luajit: /Users/Puddinord/torch/install/share/lua/5.1/trepl/init.lua:384: module 'cutorch' not found:No LuaRocks module found for cutorch
no field package.preload['cutorch']
no file '/Users/Puddinord/.luarocks/share/lua/5.1/cutorch.lua'
no file '/Users/Puddinord/.luarocks/share/lua/5.1/cutorch/init.lua'
no file '/Users/Puddinord/torch/install/share/lua/5.1/cutorch.lua'
no file '/Users/Puddinord/torch/install/share/lua/5.1/cutorch/init.lua'
no file './cutorch.lua'
no file '/Users/Puddinord/torch/install/share/luajit-2.1.0-beta1/cutorch.lua'
no file '/usr/local/share/lua/5.1/cutorch.lua'
no file '/usr/local/share/lua/5.1/cutorch/init.lua'
no file '/Users/Puddinord/.luarocks/lib/lua/5.1/cutorch.so'
no file '/Users/Puddinord/torch/install/lib/lua/5.1/cutorch.so'
no file '/Users/Puddinord/torch/install/lib/cutorch.dylib'
no file './cutorch.so'
no file '/usr/local/lib/lua/5.1/cutorch.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'error'
/Users/Puddinord/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
./densecap/utils.lua:26: in function 'setup_gpus'
run_model.lua:145: in main chunk
[C]: in function 'dofile'
...nord/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x01041b9d00

then I try to install cutorch, but there is an error telling me ' Specify CUDA_TOOLKIT_ROOT_DIR'

Installing https://raw.githubusercontent.com/torch/rocks/master/cutorch-scm-1.rockspec...
Using https://raw.githubusercontent.com/torch/rocks/master/cutorch-scm-1.rockspec... switching to 'build' mode
Cloning into 'cutorch'...
remote: Counting objects: 116, done.
remote: Compressing objects: 100% (110/110), done.
remote: Total 116 (delta 11), reused 45 (delta 4), pack-reused 0
Receiving objects: 100% (116/116), 165.37 KiB | 68.00 KiB/s, done.
Resolving deltas: 100% (11/11), done.
Checking connectivity... done.
cmake -E make_directory build && cd build && cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_PREFIX_PATH="/Users/Puddinord/torch/install/bin/.." -DCMAKE_INSTALL_PREFIX="/Users/Puddinord/torch/install/lib/luarocks/rocks/cutorch/scm-1" && make -j$(getconf _NPROCESSORS_ONLN) install

-- The C compiler identification is AppleClang 7.3.0.7030031
-- The CXX compiler identification is AppleClang 7.3.0.7030031
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Torch7 in /Users/Puddinord/torch/install
CMake Error at /usr/local/Cellar/cmake/3.6.0_1/share/cmake/Modules/FindCUDA.cmake:619 (message):
Specify CUDA_TOOLKIT_ROOT_DIR
Call Stack (most recent call first):
CMakeLists.txt:7 (FIND_PACKAGE)

-- Configuring incomplete, errors occurred!
See also "/tmp/luarocks_cutorch-scm-1-2360/cutorch/build/CMakeFiles/CMakeOutput.log".

Error: Build error: Failed building.

operation System

OSX El Captain release 10.11.5

Installed Rocks

argcheck
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

cwrap
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

dok
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

env
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

fftw3
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

gnuplot
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

graph
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

graphicsmagick
1.scm-0 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

image
1.1.alpha-0 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

lua-cjson
2.1devel-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

luaffi
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

luafilesystem
1.6.3-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

nn
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

nngraph
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

nnx
0.1-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

optim
1.0.5-0 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

paths
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

penlight
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

qtlua
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

qttorch
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

signal
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

stnbhwd
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

sundown
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

sys
1.1-0 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

threads
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

torch
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

torch-rnn
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

trepl
scm-1 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

xlua
1.0-0 (installed) - /Users/Puddinord/torch/install/lib/luarocks/rocks

Hi,

I think that's one of my questions for general object detection method, how to train an end-to-end model if we don't have the ground truth bounding box? in this case, we cannot do box regression or compute region score (?). any ideas?

If there are any existing work on this, please let me know. thanks!

When trainning the model with "train.lua", there are some problrm:

Processed image 122.jpg (4 / 1000) of split 1, detected 191 regions
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-2734/cutorch/lib/THC/generated/../generic/THCTensorSort.cu line=153 error=7 : too many resources requested for launch
/home/.../torch/install/bin/luajit: /home/.../torch/install/share/lua/5.1/nn/Container.lua:67:
In 1 module of nn.Sequential:

And I use the Nvidia K80

Does the current version of DenseCap support text-query based region search as presented in the paper?

Thanks!

th train.lua
.....
Processed image 2395971.jpg (997 / 1000) of split 1, detected 100 regions
Processed image 2395960.jpg (998 / 1000) of split 1, detected 107 regions
Processed image 2395955.jpg (999 / 1000) of split 1, detected 93 regions
Processed image 2395949.jpg (1000 / 1000) of split 1, detected 109 regions
Loss stats:
{
mid_box_reg_loss : 0.0010088438019855
captioning_loss : 27.884303262711
end_box_reg_loss : 0.0020599792839494
end_objectness_loss : 0.042175837257504
total_loss : 28.011510998734
mid_objectness_loss : 0.081963075679541
}
Average loss: 28.011510998734
Traceback (most recent call last):
File "eval/meteor_bridge.py", line 69, in
score = m._score(r['candidate'], r['references'])
File "eval/meteor_bridge.py", line 39, in _score
self.meteor_p.stdin.write(w)
IOError: [Errno 32] Broken pipe
/home/rzai/torch/install/bin/luajit: ./densecap/utils.lua:86: attempt to index local 'file' (a nil value)
stack traceback:
./densecap/utils.lua:86: in function 'read_json'
./eval/eval_utils.lua:100: in function 'score_captions'
./eval/eval_utils.lua:231: in function 'evaluate'
./eval/eval_utils.lua:83: in function 'eval_split'
train.lua:153: in main chunk
[C]: in function 'dofile'
...rzai/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
rzai@rzai00:/prj/densecap$
rzai@rzai00:/prj/densecap$ ll eval
total 19088
drwxrwxr-x 2 rzai rzai 4096 10月 27 11:29 ./
drwxrwxr-x 14 rzai rzai 4096 10月 27 13:42 ../
-rw-rw-r-- 1 rzai rzai 10674 10月 24 17:18 eval_utils.lua
-rw-rw-r-- 1 rzai rzai 38 10月 24 17:18 .gitignore
-rw-rw-r-- 1 rzai rzai 13190274 10月 27 14:31 input.json
-rw-r--r-- 1 rzai rzai 6318623 10月 27 03:09 meteor-1.5.jar
-rw-rw-r-- 1 rzai rzai 2695 10月 24 17:18 meteor_bridge.py
-rw-rw-r-- 1 rzai rzai 2851 10月 24 17:18 README.md
rzai@rzai00:~/prj/densecap$

Hi,

Noticed you have python calling java, and then communicating with lua via json files.

Just so you know, pytorch lets you run lua classes directly from python :-)

pytorch is here: https://github.com/hughperkins/pytorch

Example usage here:

• https://github.com/hughperkins/cifar.pytorch
• https://github.com/hughperkins/pytorch-residual-networks

Hi,
I tried executing tests in the test folder and most of them fail assertions, is there any plans to fix them?

Hi @jcjohnson,

I followed your instruction to run the client/server webcam demo. I was able to generate my self-signed signature on my server. When I point my client browser to the following address with SERVER_URL replaced by my server ip-address, i.e.:
https://cs.stanford.edu/people/jcjohns/densecap/demo/web-client.html?server_url=SERVER_URL

I see the following:
but none of the buttons work! It seems they are disabled

Even if I point my client browser directly to my server ip-address, it does not show me anything to let the browser manually know that the server is trusted. I tried with chrome, firefox and ie but neither of them worked.

I appreciate your help!

The link https://cs.stanford.edu/people/rak248/VG_100K/images.zip is failed.

In localization layer line 146 and line147, this should be _called_forward_size and _called_backward_size,
and line391 similarly.

In model.lua line 12 and line 13, this should be
model.nets.localization_layer.opt.mid_objectness_weight = opt.mid_objectness_weight
model.nets.localization_layer.opt.mid_box_reg_weight = opt.mid_box_reg_weight

This is a very minor issue. I just wanted to point out that the README says:

To run the pretrained model on the provided elephant.jpg image...

But elephant.jpg isn't actually included anywhere in the repo.

Thanks!

rzai@rzai00:/prj/densecap$ th train.lua
/home/rzai/torch/install/bin/luajit: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: module 'torch-rnn' not found:No LuaRocks module found for torch-rnn
no field package.preload['torch-rnn']
no file '/home/rzai/.luarocks/share/lua/5.1/torch-rnn.lua'
no file '/home/rzai/.luarocks/share/lua/5.1/torch-rnn/init.lua'
no file '/home/rzai/torch/install/share/lua/5.1/torch-rnn.lua'
no file '/home/rzai/torch/install/share/lua/5.1/torch-rnn/init.lua'
no file './torch-rnn.lua'
no file '/home/rzai/torch/install/share/luajit-2.1.0-beta1/torch-rnn.lua'
no file '/usr/local/share/lua/5.1/torch-rnn.lua'
no file '/usr/local/share/lua/5.1/torch-rnn/init.lua'
no file '/home/rzai/.luarocks/lib/lua/5.1/torch-rnn.so'
no file '/home/rzai/torch/install/lib/lua/5.1/torch-rnn.so'
no file '/home/rzai/torch/install/lib/torch-rnn.so'
no file './torch-rnn.so'
no file '/usr/local/lib/lua/5.1/torch-rnn.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'error'
/home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
train.lua:17: in main chunk
[C]: in function 'dofile'
...rzai/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
rzai@rzai00:/prj/densecap$

$ python preprocess.py
There are 108077 images total
After filtering for splits there are 87396 images
adding images to hdf5.... (this might take a while)
Exception in thread Thread-2:
Traceback (most recent call last):
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 754, in run
self.__target(_self.__args, *_self.__kwargs)
File "preprocess.py", line 246, in worker
img = imread(filename)
File "/home/sankuai/.local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 154, in imread
im = Image.open(name)
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/site-packages/PIL/Image.py", line 2280, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'data/visual-genome/images/53.jpg'
Exception in thread Thread-5:
Traceback (most recent call last):
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 754, in run
self.__target(_self.__args, *_self.__kwargs)
File "preprocess.py", line 246, in worker
img = imread(filename)
File "/home/sankuai/.local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 154, in imread
im = Image.open(name)
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/site-packages/PIL/Image.py", line 2280, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'data/visual-genome/images/59.jpg'

Exception in thread Thread-3:
Traceback (most recent call last):
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 754, in run
self.__target(_self.__args, *_self.__kwargs)
File "preprocess.py", line 246, in worker
img = imread(filename)
File "/home/sankuai/.local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 154, in imread
im = Image.open(name)
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/site-packages/PIL/Image.py", line 2280, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'data/visual-genome/images/54.jpg'

Exception in thread Thread-1:
Traceback (most recent call last):
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 754, in run
self.__target(_self.__args, *_self.__kwargs)
File "preprocess.py", line 246, in worker
img = imread(filename)
File "/home/sankuai/.local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 154, in imread
im = Image.open(name)
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/site-packages/PIL/Image.py", line 2280, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'data/visual-genome/images/51.jpg'

Exception in thread Thread-4:
Traceback (most recent call last):
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/threading.py", line 754, in run
self.__target(_self.__args, *_self.__kwargs)
File "preprocess.py", line 246, in worker
img = imread(filename)
File "/home/sankuai/.local/lib/python2.7/site-packages/scipy/misc/pilutil.py", line 154, in imread
im = Image.open(name)
File "/opt/meituan/lishengxi/local_torch/lib/python2.7/site-packages/PIL/Image.py", line 2280, in open
fp = builtins.open(filename, "rb")
IOError: [Errno 2] No such file or directory: 'data/visual-genome/images/55.jpg'

hi, @jcjohnson why run the preprocess.py script produce above errors.
The 53.jpg 59.jpg 54.jpg 55.jpg 51.jpg are not in path data/visual-genome/images

Using your existing code, would I be able to train models on GPU without installing Cudnn ?

hi, as is known to us, it is difficult to debug program written in lua. So guys, can you recommend a good IDE to debug LUA for me?

Hi, I have some error after (https://github.com/jcjohnson/densecap/issues):
libloadcaffe.so: undefined symbol: _ZN6google8protobuf8internal23empty_string_once_init_E

$ th train.lua -data_h5 output.HDF5 -data_json output.json -gpu 2
{
  finetune_cnn_after : -1
  weight_decay : 1e-06
  optim_beta2 : 0.999
  val_images_use : 1000
  timing : false
  optim_epsilon : 1e-08
  test_num_proposals : 1000
  input_encoding_size : 512
  losses_log_every : 10
  id : ""
  gpu : 2
  drop_prob : 0.5
  data_json : "output.json"
  clip_final_boxes : 1
  sampler_batch_size : 256
  rnn_size : 512
  test_rpn_nms_thresh : 0.7
  progress_dump_every : 100
  optim_beta1 : 0.9
  end_objectness_weight : 0.1
  seed : 123
  test_final_nms_thresh : 0.3
  eval_first_iteration : 0
  captioning_weight : 1
  checkpoint_start_from : ""
  sampler_high_thresh : 0.7
  mid_objectness_weight : 0.1
  data_h5 : "output.HDF5"
  train_remove_outbounds_boxes : 1
  sampler_low_thresh : 0.3
  debug_max_train_images : -1
  learning_rate : 1e-05
  rpn_hidden_dim : 512
  max_iters : -1
  checkpoint_path : "checkpoint.t7"
  save_checkpoint_every : 10000
  end_box_reg_weight : 0.1
  proposal_regions_h5 : ""
  backend : "cudnn"
  mid_box_reg_weight : 0.05
  box_reg_decay : 5e-05
}
DataLoader loading json file:   output.json 
DataLoader loading h5 file:     output.HDF5 
reading box_to_img  
reading boxes   
reading image_heights   
reading image_widths    
reading img_to_first_box    
reading img_to_last_box 
reading labels  
reading lengths 
reading original_heights    
reading original_widths 
reading split   
assigned 77396/5000/5000 images to train/val/test.  
initialized DataLoader: 
#images: 87396, #regions: 4062806, sequence max length: 15  
initializing a DenseCap model from scratch...   
/home/PPeiMi/torch/install/bin/luajit: /home/PPeiMi/torch/install/share/lua/5.1/trepl/init.lua:384: /home/PPeiMi/torch/install/share/lua/5.1/trepl/init.lua:384: /home/PPeiMi/torch/install/share/lua/5.1/loadcaffe/ffi.lua:10: /home/PPeiMi/torch/install/lib/lua/5.1/libloadcaffe.so: undefined symbol: _ZN6google8protobuf8internal23empty_string_once_init_E
stack traceback:
    [C]: in function 'error'
    /home/PPeiMi/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
    ./densecap/DenseCapModel.lua:21: in function '__init'
    /home/PPeiMi/.luarocks/share/lua/5.1/torch/init.lua:91: in function </home/PPeiMi/.luarocks/share/lua/5.1/torch/init.lua:87>
    [C]: in function 'DenseCapModel'
    ./models.lua:7: in function 'setup'
    train.lua:48: in main chunk
    [C]: in function 'dofile'
    ...eiMi/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
    [C]: at 0x00405840

I can't find how to solve this...

First, thank you :) This research is a big inspiration, and it's great to learn and explore with the code itself. After realizing I needed a beefier GPU, I got it running on AWS with a p2.xlarge (1xK80). Here are a few things I noticed.

After running pip install -r requirements.txt I got an error like:

Command /home/ubuntu/densecap/webcam/.env/bin/python -c "import setuptools, tokenize;__file__='/home/ubuntu/densecap/webcam/.env/build/Pillow/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-e4ld0f-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/ubuntu/densecap/webcam/.env/include/site/python2.7 failed with error code 1 in /home/ubuntu/densecap/webcam/.env/build/Pillow

After searching, it looks like this is a Pillow issue. You need to run sudo apt-get install python-dev before Pillow can be installed.

Then I noticed that you create the virtual env, install the requirements, then activate it. I think the second and third steps should be reversed, otherwise you are installing the requirements outside the virtualenv (and the Python code fails to run).

Then, I got the error module 'socket' not found:No LuaRocks module found for socket when running th webcam/daemon.lua. This was fixed by installing luarocks install luasocket.

Then, on AWS (and I imagined some other servers) ports are available based on a whitelist. I had to open port 5000 for inbound TCP traffic.

When you wrote: https://cs.stanford.edu/people/jcjohns/densecap/demo/web-client.html?server_url=SERVER_URL I thought SERVER_URL meant the domain of my server. Then I realized it meant https://ec2-my-ip-address.compute-1.amazonaws.com.:5000/, the same URL where I accepted the self-signed certificate, and it worked!

Finally, I also have issue #49 despite CUDNN being correctly installed and used for other things, I can't use it here for some reason. So I have to disable it with -use_cudnn 0 when running the daemon.lua or the other demo.

@jcjohnson
Hi,
I ran your nice code and met following error in training.
(We downloaded the visual-genome dataset(i.e. raw images and region descriptions) and then generate both h5, json files using the code, preprocess.py)

iter 11235: mid_box_reg_loss: 0.001, captioning_loss: 22.173, end_objectness_loss: 0.041, mid_objectness_loss: 0.106, end_box_reg_loss: 0.002, [total: 44.644]
iter 11240: mid_box_reg_loss: 0.001, captioning_loss: 28.484, end_objectness_loss: 0.045, mid_objectness_loss: 0.050, end_box_reg_loss: 0.001, [total: 57.159]
/home/digits/torch/install/bin/luajit: bad argument #2 to '?' (out of bounds at /home/digits/torch/pkg/torch/lib/TH/generic/THStorage.c:178)
stack traceback:
[C]: at 0x7f24927ea8e0
[C]: in function 'multinomial'
./densecap/modules/BoxSampler.lua:141: in function 'forward'
./densecap/modules/BoxSamplerHelper.lua:78: in function 'forward'
./densecap/LocalizationLayer.lua:426: in function 'f'
./densecap/LocalizationLayer.lua:228: in function 'timeit'
./densecap/LocalizationLayer.lua:425: in function 'updateOutput'
/home/digits/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
./densecap/DenseCapModel.lua:262: in function 'forward'
./densecap/DenseCapModel.lua:393: in function 'forward_backward'
train.lua:98: in function 'lossFun'
train.lua:135: in main chunk
[C]: in function 'dofile'
...gits/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

Please give me advices to resolve the problem.

total_loss is often larger than the sum of all the loses, for example here is the output of a training run:

Loss stats:    
{
  mid_box_reg_loss : 0.0019056373813539
  captioning_loss : 5.0455059438944
  end_box_reg_loss : 0.002998169460654
  end_objectness_loss : 0.038921515512466
  total_loss : 10.398667634428
  mid_objectness_loss : 0.11150163569537
}
Average loss:     10.398667634428    

Note that 10.3... is not the sum of the other losses in the losses table. I think there is a bug here: https://github.com/jcjohnson/densecap/blob/master/densecap/DenseCapModel.lua#L455-L458

total_loss is added to the losses table, the table is then iterated over, and all values are added to the total_loss entry. At some point in that for loop it will look like this: total_loss = total_loss + total_loss.

no field package.preload['libcustn']
no file '/home/zhangcheng/.luarocks/share/lua/5.1/libcustn.lua'
no file '/home/zhangcheng/.luarocks/share/lua/5.1/libcustn/init.lua'
no file '/home/zhangcheng/torch/install/share/lua/5.1/libcustn.lua'
no file '/home/zhangcheng/torch/install/share/lua/5.1/libcustn/init.lua'
no file './libcustn.lua'
no file '/home/zhangcheng/torch/install/share/luajit-2.1.0-beta1/libcustn.lua'
no file '/usr/local/share/lua/5.1/libcustn.lua'
no file '/usr/local/share/lua/5.1/libcustn/init.lua'
no file '/home/zhangcheng/.luarocks/lib/lua/5.1/libcustn.so'
no file '/home/zhangcheng/torch/install/lib/lua/5.1/libcustn.so'
no file '/home/zhangcheng/torch/install/lib/libcustn.so'
no file './libcustn.so'
no file '/usr/local/lib/lua/5.1/libcustn.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'error'
/home/zhangcheng/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
run_model.lua:5: in main chunk
[C]: in function 'dofile'
...heng/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

And I tried to use luarocks install libcustn. It dosen't work .

Error: No results matching query were found.

As I'm following the Installation guide, and going to train with GPU after install cuDNN. But there are some bugs:

[/home/PPeiMi/densecap-master] 10:06
$ ./../torch/install/bin/th train.lua -data_h5 output.HDF5 -data_json output.json -gpu 2
{
  finetune_cnn_after : -1
  weight_decay : 1e-06
  optim_beta2 : 0.999
  val_images_use : 1000
  timing : false
  optim_epsilon : 1e-08
  test_num_proposals : 1000
  input_encoding_size : 512
  losses_log_every : 10
  id : ""
  gpu : 2
  drop_prob : 0.5
  data_json : "output.json"
  clip_final_boxes : 1
  sampler_batch_size : 256
  rnn_size : 512
  test_rpn_nms_thresh : 0.7
  progress_dump_every : 100
  optim_beta1 : 0.9
  end_objectness_weight : 0.1
  seed : 123
  test_final_nms_thresh : 0.3
  eval_first_iteration : 0
  captioning_weight : 1
  checkpoint_start_from : ""
  sampler_high_thresh : 0.7
  mid_objectness_weight : 0.1
  data_h5 : "output.HDF5"
  train_remove_outbounds_boxes : 1
  sampler_low_thresh : 0.3
  debug_max_train_images : -1
  learning_rate : 1e-05
  rpn_hidden_dim : 512
  max_iters : -1
  checkpoint_path : "checkpoint.t7"
  save_checkpoint_every : 10000
  end_box_reg_weight : 0.1
  proposal_regions_h5 : ""
  backend : "cudnn"
  mid_box_reg_weight : 0.05
  box_reg_decay : 5e-05
}
nil 
/home/PPeiMi/torch/install/bin/luajit: /home/PPeiMi/torch/install/share/lua/5.1/trepl/init.lua:384: /home/PPeiMi/torch/install/share/lua/5.1/trepl/init.lua:384: /home/PPeiMi/.luarocks/share/lua/5.1/cudnn/ffi.lua:1598: 'libcudnn (R5) not found in library path.
Please install CuDNN from https://developer.nvidia.com/cuDNN
Then make sure files named as libcudnn.so.5 or libcudnn.5.dylib are placed in your library load path (for example /usr/local/lib , or manually add a path to LD_LIBRARY_PATH)

stack traceback:
    [C]: in function 'error'
    /home/PPeiMi/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
    train.lua:35: in main chunk
    [C]: in function 'dofile'
    ...eiMi/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
    [C]: at 0x00405840

It says the version of cuDNN needs to be version 5. In the CUDA website, the cuDNN v5 according to CUDA 7.5, however, my CUDA version is only 7.0. Do I have to update my CUDA, or is there any method that can fix this problem?

Hi, @jcjohnson
I followed your instruction to run the client/server webcam demo. When I am running python webcam/server.py, the command was stuck there and it returned

loading checkpoint from data/models/densecap/densecap-pretrained-vgg16.t7
done loading checkpoint
Warning: cudnn.convert does not work with nngraph yet. Ignoring nn.gModule

I know the warning can be ignored safely, but I don't know the reason why the loading process was stuck(it can't run the next command)after it said done loading checkpoint.

And then I tried to run python webcam/server.py, I also have the issue

Traceback (most recent call last):
File "webcam/server.py", line 7, in
from flask import Flask, request
ImportError: No module named flask
zhangcheng@zhangcheng-To-be-filled-by-O-E-M:~/densecap$ sudo apt-get flask
[sudo] password for zhangcheng:
E: Invalid operation flask

I appreciate your help!

run_model.lua fails for me when I run in GPU mode:

$ th run_model.lua -input_image imgs/elephant.jpg

/Users/snowm/torch/install/bin/luajit: /Users/snowm/torch/install/share/lua/5.1/cudnn/convert.lua:26: cudnn.convert does not work with nngraph yet
stack traceback:
        [C]: in function 'error'
        /Users/snowm/torch/install/share/lua/5.1/cudnn/convert.lua:26: in function 'callback'
        ...rs/snowm/torch/install/share/lua/5.1/nngraph/gmodule.lua:197: in function 'replace'
        /Users/snowm/torch/install/share/lua/5.1/nn/Module.lua:391: in function 'convert'
        ./densecap/DenseCapModel.lua:205: in function 'convert'
        run_model.lua:148: in main chunk
        [C]: in function 'dofile'
        ...nowm/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x010a1c0d10

CPU mode works perfectly:

$ th run_model.lua -input_image imgs/elephant.jpg -gpu -1
1/1 processing image imgs/elephant.jpg

When I try to run on new image, it throws this warning. And then I found the generated captions are all like that:
"a large brown and white and white and white and white and white."repeated many times the same caption.

How to fix this problem? Thank you.

I tried to preprocess genome dataset with preprocess.py.But it takes too long time. It has taken more than 12 hours before preprocessing half of the dataset. My god!!!!

It seems that there is something wrong with the hdf5.

There are 108077 images total
After filtering for splits there are 87398 images
Traceback (most recent call last):
File "preprocess.py", line 470, in
main(args)
File "preprocess.py", line 385, in main
f = h5py.File(args.h5_output, 'w')
File "/usr/lib/python2.7/dist-packages/h5py/_hl/files.py", line 207, in init
fid = make_fid(name, mode, userblock_size, fapl)
File "/usr/lib/python2.7/dist-packages/h5py/_hl/files.py", line 85, in make_fid
fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
File "h5f.pyx", line 90, in h5py.h5f.create (h5py/h5f.c:1984)
IOError: unable to create file (File accessibilty: Unable to open file)

How can I fix it? Thanks!

In your paper,your dense captioning model can support image retrieval using natural language queries, and can localize these queries in retrieved images. How can I do the retrieval work?

My mac os x system has 16G RAM and 4G GPU memory. It seems still not enough to run the model.

$ th run_model.lua

THCudaCheck FAIL file=/Users/eryunliu/torch/extra/cutorch/lib/THC/generic/THCStorage.cu line=41 error=2 : out of memory
/Users/eryunliu/torch/install/bin/luajit: /Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:11: cuda runtime error (2) : out of memory at /Users/eryunliu/torch/extra/cutorch/lib/THC/generic/THCStorage.cu:41
stack traceback:
[C]: in function 'resize'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:11: in function 'torch_Storage_type'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:57: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type'
/Users/eryunliu/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/eryunliu/torch/install/share/lua/5.1/nn/Module.lua:126: in function 'type'
./densecap/DenseCapModel.lua:199: in function 'convert'
run_model.lua:148: in main chunk
[C]: in function 'dofile'
...nliu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x010b13bd50

I read the codes, and have this question:
the updateGradInput() in file 'LocalizationLayer.lua' has such lines (line 549~550):

local grad_cnn_features = self.gradInput
grad_cnn_features:resizeAs(cnn_features):zero()

and the lines near the bottom of the same function read：

local din = self.nets.rpn:backward(cnn_features, grad_rpn_out)
grad_cnn_features:add(din)

return self.gradInput

I wonder why the self.gradInput is not updated with grad_cnn_features?
It seems there should be a line something like “self.gradInput = grad_cnn_features”, is it right?

Since it use torch.CudaTensor, so it require GPU, right?
Is there any way I can train it only using CPU?

BTW, here is my command line for training
th train.lua -learning_rate 0.003 -data_json data/training_data/training_json -data_h5 data/training_data/training_h5 -gpu -1 -checkpoint_path data/training/cp.v1 -id densecapv1 -backend 'nn'

And here is the error I got.
/usr/local/google/home/wangyu/torch/install/bin/luajit: ...gle/home/wangyu/.luarocks/share/lua/5.1/torch/Tensor.lua:238: attempt to index a nil value
stack traceback:
...gle/home/wangyu/.luarocks/share/lua/5.1/torch/Tensor.lua:238: in function 'type'
.../google/home/wangyu/.luarocks/share/lua/5.1/nn/utils.lua:52: in function 'recursiveType'
...google/home/wangyu/.luarocks/share/lua/5.1/nn/Module.lua:126: in function 'type'
.../google/home/wangyu/.luarocks/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
...google/home/wangyu/.luarocks/share/lua/5.1/nn/Module.lua:126: in function 'type'
train.lua:48: in main chunk
[C]: in function 'dofile'
...ngyu/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

Hi, I'm quite confused by the METEOR result in paper. Is the METEOR result in table 1 computed by all captions, or the true possitive in detection, or other groups? Much appreciate if anyone could kindly explain it.

Hi,

I'm using your library (run_model.lua) that I modified for computing a 2fps stream of images buffered in memory with the default configurations:

-image_size, 480
-rpn_nms_thresh, 0.7
-final_nms_thresh, 0.3
-num_proposals, 1000
-gpu, 1
-use_cudnn, 1

But I get a big charge of cpu in one core (100% cpu usage). I also have seen that this is produced by model:forward_test(img_caffe:type(dtype))

Is it normal that cpu usage?

Luarocks packages:

argcheck scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

csvigo scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

cudnn scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

cunn scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

cunnx scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

cutorch scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

cv scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

cwrap scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

dok scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

dpnn scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

env scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

fftw3 scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

fn 0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

gnuplot scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

graph scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

graphicsmagick 1.scm-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

hdf5 20-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

image 1.1.alpha-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

loadcaffe 1.0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

logroll 0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

lua-cjson 2.1.0-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

luaffi scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

luafilesystem 1.6.3-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

luasocket 3.0rc1-2 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

moses 1.4.0-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

nn scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

nngraph scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

nnx 0.1-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

optim 1.0.5-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

paths scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

penlight scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

pprint 0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

qtlua scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

qttorch scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

signal scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

stnbhwd scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

struct 1.4-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

sundown scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

sys 1.1-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

threads scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

torch scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

torch-rnn scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

totem 0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

trepl scm-1 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

util 0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

xlua 1.0-0 (installed) - /home/karpathy-dev/torch/install/lib/luarocks/rocks

Cuda version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Wed_May__4_21:01:56_CDT_2016
Cuda compilation tools, release 8.0, V8.0.26

CPU info:

Placa: Z97X-Gaming 3
CPU: Intel(R) Core(TM) i5-4690K CPU @ 3.50GHz
RAM (16GB): G. Skill F3-1600C9S-8GXM (2x8GB) In same channel
GPU: NVidia GTX1080 Founders Edition x2 in SLI
HD: Seagate Desktop HDD ST2000DM001 2TB 64MB Cache SATA 6.0Gb/s 3.5"
Fuente: NOX Hummer 850W Bronze+

And another question, why cannot call method 'scatter'. Thanks a lot.

xhu@linux:~/Projects/torch/examples/mydensecap$ th evaluate_model.lua
Loaded model
Using dtype "torch.CudaTensor"
DataLoader loading json file: data/VG-regions-dicts.json
DataLoader loading h5 file: data/VG-regions.h5
reading box_to_img
reading boxes
reading image_heights
reading image_widths
reading img_to_first_box
reading img_to_last_box
reading labels
reading lengths
reading original_heights
reading original_widths
reading split
assigned 77398/5000/5000 images to train/val/test.
initialized DataLoader:

images: 87398, #regions: 4158176, sequence max length: 15

using split 1
Processed image 59.jpg (1 / 5000) of split 1, detected 103 regions
Processed image 67.jpg (2 / 5000) of split 1, detected 92 regions
Processed image 78.jpg (3 / 5000) of split 1, detected 83 regions
/home/xhu/Projects/torch/install/bin/luajit: ./densecap/LanguageModel.lua:98: attempt to concatenate a nil value
stack traceback:
./densecap/LanguageModel.lua:98: in function 'decodeSequence'
./eval/eval_utils.lua:63: in function 'eval_split'
evaluate_model.lua:63: in main chunk
[C]: in function 'dofile'
...ects/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00405ea0

when i run evaluate_model.lua , I get an error like this. I have no idea about this.

I'm on a mac 15" with nvidia, cuda, cudnn installed and getting this:

qlua webcam/single_machine_demo.lua
loading checkpoint from data/models/densecap/densecap-pretrained-vgg16.t7
done loading checkpoint
THCudaCheck FAIL file=/tmp/luarocks_cutorch-scm-1-8066/cutorch/lib/THC/generic/THCStorage.cu line=65 error=2 : out of memory
qlua: /Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:11: cuda runtime error (2) : out of memory at /tmp/luarocks_cutorch-scm-1-8066/cutorch/lib/THC/generic/THCStorage.cu:65
stack traceback:
[C]: at 0x010c1f0870
[C]: in function 'resize'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:11: in function 'torch_Storage_type'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:57: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/Module.lua:152: in function 'type'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/Module.lua:152: in function 'type'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/Module.lua:152: in function 'type'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:41: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/Module.lua:152: in function 'type'
/Users/yosit/torch/install/share/lua/5.1/nn/utils.lua:45: in function 'recursiveType'
/Users/yosit/torch/install/share/lua/5.1/nn/Module.lua:152: in function 'type'
./densecap/DenseCapModel.lua:199: in function 'convert'
webcam/single_machine_demo.lua:185: in function 'main'
webcam/single_machine_demo.lua:256: in main chunk

any idea

Hi!
I tried to rewrite run_model.lua script to work in batch mode, but get the error

./densecap/DenseCapModel.lua:244: assertion failed!
stack traceback:
[C]: in function 'assert'
./densecap/DenseCapModel.lua:244: in function 'forward'
./densecap/DenseCapModel.lua:321: in function 'forward_test'

Also, I see this comment https://github.com/jcjohnson/densecap/blob/master/densecap/DenseCapModel.lua#L215

So whether there is an opportunity to work in batch mode?
Thanks

I followed the instructions and tried to start running the model on a new image(elephant.jpg), but I got an "Unknow object" error as follows:

hitfishking@ubuntu:~/densecap$ th run_model.lua -gpu -1 -input_image ~/densecap/imgs/elephant.jpg
/home/hitfishking/torch/install/bin/luajit: ...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:375: unknown object
stack traceback:
[C]: in function 'error'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:375: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:353: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:353: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
/home/hitfishking/torch/install/share/lua/5.1/nn/Module.lua:158: in function 'read'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:351: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:369: in function 'readObject'
...e/hitfishking/torch/install/share/lua/5.1/torch/File.lua:409: in function 'load'
run_model.lua:146: in main chunk
[C]: in function 'dofile'
...king/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x0804cbc0

OS: unbuntu Desktop 15.05, 32bit.
Anything wrong with my system configuration? Anyone can help?

rzai@rzai00:/prj/densecap$ th train.lua
/home/rzai/torch/install/bin/luajit: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: /home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: module 'stn' not found:No LuaRocks module found for stn
no field package.preload['stn']
no file '/home/rzai/.luarocks/share/lua/5.1/stn.lua'
no file '/home/rzai/.luarocks/share/lua/5.1/stn/init.lua'
no file '/home/rzai/torch/install/share/lua/5.1/stn.lua'
no file '/home/rzai/torch/install/share/lua/5.1/stn/init.lua'
no file './stn.lua'
no file '/home/rzai/torch/install/share/luajit-2.1.0-beta1/stn.lua'
no file '/usr/local/share/lua/5.1/stn.lua'
no file '/usr/local/share/lua/5.1/stn/init.lua'
no file '/home/rzai/.luarocks/lib/lua/5.1/stn.so'
no file '/home/rzai/torch/install/lib/lua/5.1/stn.so'
no file '/home/rzai/torch/install/lib/stn.so'
no file './stn.so'
no file '/usr/local/lib/lua/5.1/stn.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
[C]: in function 'error'
/home/rzai/torch/install/share/lua/5.1/trepl/init.lua:384: in function 'require'
train.lua:17: in main chunk
[C]: in function 'dofile'
...rzai/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670
rzai@rzai00:/prj/densecap$ luarocks install stn

Error: No results matching query were found.

Hi,

Is there an easy way to test the backward pass ("head of cat") -> bounding box in a single image? Is there any demo script for it?

Thanks

Hi There,
I recently tried to used denseCap. I used the example run before I do anything else, and I got the following error
./densecap/DenseCapModel.lua:205: attempt to call field 'convert' (a nil value) stack traceback: ./densecap/DenseCapModel.lua:205: in function 'convert' run_model.lua:148: in main chunk [C]: in function 'dofile' ...garj/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670
however When I run the example on CPU mode it works just fine (very slowly as expected). So I thought my dependencies are out of date, and I updated all of them. Post update I started seeing this error
ffi.lua:1286: These bindings are for version 4005 or above, while the loaded CuDNN is version: 4004 Are you using an older version of CuDNN? stack traceback: [C]: in function 'error' /work/sagarj/torch/install/share/lua/5.1/trepl/init.lua:383: in function 'require' ./densecap/utils.lua:31: in function 'setup_gpus' run_model.lua:145: in main chunk [C]: in function 'dofile' ...garj/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk [C]: at 0x00406670
I read on some other repos, and they think that this is because of the new cudnn luarock. Do you happen to know what is going on there?
Thanks a lot for the great work.

Hi, I am running your densecap code. But in the DenseCapModel.lua L120, it attempts to call field 'TemporalCrossEntropyCriterion'. But there is no definition about TemporalCrossEntropyCriterion. The errors show that the module TemporalCrossEntropyCriterion is not found. Could you help it?

Apparently, module Image need additional module sys:

module 'sys.fpath' not found:No LuaRocks module found for sys.fpath

It was fixed by running

luarocks install sys

Would be nice to add it to the requirements in README.

Hello ,I am new to this direction and trying the densecap code.But I run the script as fellow:
./../densecap-master$ th run_model.lua -input_image ./imgs/t01f0d6d1b73d08a815.jpg
I meet the problem like this:
Warning: cudnn.convert does not work with nngraph yet. Ignoring nn.gModule1/1 processing image ./imgs/t01f0d6d1b73d08a815.jpg

I tried this:

luarocks install cutorch
luarocks install cunn
luarocks install cudnn

But it had no use.
How can I fix it?
I am new here. Thank you so much.

When I run the "python preprocess.py " , my compute will shut down. Why? Because the data is too big?

In the LanugageModel.lua, I noticed that when view_in, view_out are created, they took size of -1. But they resize into (N * (T + 2), -1) during UpdateOutput, in line 122/123.
Is there any specific reason to do that?

When running the pre-trained model, I receive the following error:

[john204@WE29385 densecap]$ th run_model.lua -input_dir /scratch/projects/ImageClass/data/interest/ -gpu -1                                      
1/51 processing image /scratch/projects/ImageClass/data/interest/00427e3bc67ee452ada7c787f7c4b87194f914b2dbd430aaff2f635a.jpg                    
/home/john204/torch/install/bin/luajit: /home/john204/torch/install/share/lua/5.1/nn/Container.lua:67:                                                            
In 3 module of nn.Sequential:                                                    
In 2 module of nn.Sequential:                                                    
./densecap/modules/BatchBilinearSamplerBHWD.lua:119: bad argument #4 to 'BilinearSamplerBHWD_updateOutput' (torch.FloatTensor expected, got string)               
stack traceback:                                                                 
        [C]: in function 'BilinearSamplerBHWD_updateOutput'                      
        ./densecap/modules/BatchBilinearSamplerBHWD.lua:119: in function <./densecap/modules/BatchBilinearSamplerBHWD.lua:104>                                    
        [C]: in function 'xpcall'                                                
        /home/john204/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'                                                                
        /home/john204/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'                                                                     
        ./densecap/modules/BilinearRoiPooling.lua:88: in function 'forward'      
        ./densecap/LocalizationLayer.lua:348: in function 'f'                    
        ./densecap/LocalizationLayer.lua:228: in function 'timeit'               
        ./densecap/LocalizationLayer.lua:346: in function <./densecap/LocalizationLayer.lua:250>                                                                  
        [C]: in function 'xpcall'                                                
        /home/john204/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'                                                                
        /home/john204/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        ./densecap/DenseCapModel.lua:253: in function 'forward'
        ./densecap/DenseCapModel.lua:299: in function 'forward_test'
        run_model.lua:77: in function 'run_image'
        run_model.lua:164: in main chunk
        [C]: in function 'dofile'
        ...n204/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
        [C]: at 0x00406330

I tried to train with Visual Genome dataset as you described.
I just follow the default parameter of preprocess.py to make data for training.
I faced error at the beginning.
What can I check?

Successfully loaded /disk3/daum/iu/github/densecap/data/models/vgg-16/VGG_ILSVRC_16_layers.caffemodel
conv1_1: 64 3 3 3
conv1_2: 64 64 3 3
conv2_1: 128 64 3 3
conv2_2: 128 128 3 3
conv3_1: 256 128 3 3
conv3_2: 256 256 3 3
conv3_3: 256 256 3 3
conv4_1: 512 256 3 3
conv4_2: 512 512 3 3
conv4_3: 512 512 3 3
conv5_1: 512 512 3 3
conv5_2: 512 512 3 3
conv5_3: 512 512 3 3
fc6: 1 1 25088 4096
fc7: 1 1 4096 4096
fc8: 1 1 4096 1000
total number of parameters in net: 135734875
total number of parameters in CNN: 14454528
/home/andrew/torch/install/bin/luajit: /home/andrew/torch/install/share/lua/5.1/nn/Container.lua:67:
In 3 module of nn.Sequential:
./densecap/modules/BoxSampler.lua:104: attempt to call method 'scatter' (a nil value)
stack traceback:
./densecap/modules/BoxSampler.lua:104: in function 'forward'
./densecap/modules/BoxSamplerHelper.lua:78: in function 'forward'
./densecap/LocalizationLayer.lua:426: in function 'f'
./densecap/LocalizationLayer.lua:228: in function 'timeit'
./densecap/LocalizationLayer.lua:425: in function <./densecap/LocalizationLayer.lua:383>
[C]: in function 'xpcall'
/home/andrew/torch/install/share/lua/5.1/nn/Container.lua:63: in function 'rethrowErrors'
/home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
./densecap/DenseCapModel.lua:253: in function 'forward'
./densecap/DenseCapModel.lua:406: in function 'forward_backward'
train.lua:90: in function 'lossFun'
train.lua:120: in main chunk
[C]: in function 'dofile'
...drew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

WARNING: If you see a stack trace below, it doesn't point to the place where this error occured. Please use only the one above.
stack traceback:
[C]: in function 'error'
/home/andrew/torch/install/share/lua/5.1/nn/Container.lua:67: in function 'rethrowErrors'
/home/andrew/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
./densecap/DenseCapModel.lua:253: in function 'forward'
./densecap/DenseCapModel.lua:406: in function 'forward_backward'
train.lua:90: in function 'lossFun'
train.lua:120: in main chunk
[C]: in function 'dofile'
...drew/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:145: in main chunk
[C]: at 0x00406670

I get the following error when trying to run the pre-trained model:

not found:No LuaRocks module found for loadcaffe

I have a idea and want to complete it based on code of dense caption.I have read the paper and the code.I find it very complex and hard to understand.Who can help me?