endernewton / iter-reason Goto Github PK

View Code? Open in Web Editor NEW

265.0 265.0 37.0 336 KB

Code for Iterative Reasoning Paper (CVPR 2018)

License: MIT License

Makefile 0.05% Python 95.42% Shell 4.53%

classification context recognition tensorflow

iter-reason's People

Contributors

Stargazers

Watchers

iter-reason's Issues

Release the graph-based module ?

@endernewton Hi, xinlei. I like your paper very much, but I have some confusion about the graph based reasoning? Could you release the graph-based global reasoning code?

Does this version include the graph-based module?

Hello, I want to know the implementation details of the graph-based module in your paper, but I can't find the corresponding codes in this version. Does this version include that module? Thanks so much for your help!

Does the attention:a_0 come from the all-zero initialized memory S_0 ?

Hi!
In iter-reason/lib/nets/attend_memory.py row:154

At iteration 0, you use all-zero initialized memory S_0 to predict the confidence of f_0, while i think it's more reasonable to use roi features come from conv_base which is consistent with your paper.

I'm not sure if I understand it correctly.

Undefined name 'part_seg' in lib/datasets/ade.py

$ python2 -m flake8 . --count --select=E9,F82 --show-source --statistics

./lib/datasets/ade.py:223:19: F821 undefined name 'part_seg'
            seg = part_seg
                  ^
1     F821 undefined name 'part_seg'

How to test my own image?

Hello, I am very interested in your novel method. When I finish the "train_memory.sh" ,how can I test my own image? Just apply "test_memory.sh"? I would like to know details,thanks!

Have you modify it into pytorch？

Have you modify it into pytorch？I think pytorch is better to use.

graph based module

Hi，xinlei
I like your paper very much, but I have some confusion about the graph based reasoning? In these issues, I found that you have given some raw codes, however not to incorporate them to the main code .Did you continue to complete this work later?
Looking forward to your answer.

Reshape error when testing on COCO

Hello, I was trying run this code to but got a reshape error when testing on COCO.
I have finished training and got a model. Then I run ./experiments/scripts/test_memory.sh 0 coco local 32a56 72 and got output like this:

+ set -e
+ export PYTHONUNBUFFERED=True
+ PYTHONUNBUFFERED=True
+ NET_BASE=res101
+ GPU_ID=0
+ DATASET=coco
+ NET=res101_local
+ OIFS=' 	
'
+ IFS=a
+ STEP=32a56
+ STEPSIZE='['
+ for i in '$STEP'
+ STEPSIZE='[320000,'
+ for i in '$STEP'
+ STEPSIZE='[320000,560000,'
+ STEPSIZE='[320000,560000,]'
+ IFS=' 	
'
+ ITERS=720000
+ array=($@)
+ len=5
+ EXTRA_ARGS=
+ EXTRA_ARGS_SLUG=
+ case ${DATASET} in
+ TRAIN_IMDB=coco_2014_train+coco_2014_valminusminival
+ TEST_IMDBS=("coco_2014_minival")
+ declare -a TEST_IMDBS
+ [[ ! -z '' ]]
+ EXTRA_ARGS_SLUG=32a56_72
++ date +%Y-%m-%d_%H-%M-%S
+ LOG=experiments/logs/test_res101_local_coco_2014_train+coco_2014_valminusminival_32a56_72.txt.2018-12-28_11-12-35
+ exec
++ tee -a experiments/logs/test_res101_local_coco_2014_train+coco_2014_valminusminival_32a56_72.txt.2018-12-28_11-12-35
+ echo Logging output to experiments/logs/test_res101_local_coco_2014_train+coco_2014_valminusminival_32a56_72.txt.2018-12-28_11-12-35
Logging output to experiments/logs/test_res101_local_coco_2014_train+coco_2014_valminusminival_32a56_72.txt.2018-12-28_11-12-35
+ set +x
+ for TEST_IMDB in '"${TEST_IMDBS[@]}"'
+ CUDA_VISIBLE_DEVICES=0
+ python ./tools/test_memory.py --imdb coco_2014_minival --model output/res101_local/coco_2014_train+coco_2014_valminusminival/32a56_72/res101_local_iter_720000.ckpt --cfg experiments/cfgs/res101_local.yml --tag 32a56_72 --net res101_local --visualize --set
Called with args:
Namespace(cfg_file='experiments/cfgs/res101_local.yml', imdb_name='coco_2014_minival', model='output/res101_local/coco_2014_train+coco_2014_valminusminival/32a56_72/res101_local_iter_720000.ckpt', net='res101_local', set_cfgs=[], tag='32a56_72', visualize=True)
Using config:
{'BOTTLE_SCALE': 16.0,
 'CLASSES': None,
 'DATA_DIR': '/home/zty/iter-reason/data',
 'EPS': 1e-14,
 'EXP_DIR': 'res101_local',
 'MEM': {'BETA': 0.5,
         'C': 512,
         'CONV': 3,
         'CROP_SIZE': 7,
         'CT_CONV': 3,
         'CT_FCONV': 3,
         'CT_L': 3,
         'C_STD': 0.01,
         'FC_C': 4096,
         'FC_L': 2,
         'FM_R': 1.0,
         'FP_R': 1.0,
         'INIT_H': 20,
         'INIT_W': 20,
         'IN_CONV': 3,
         'IN_L': 2,
         'ITER': 3,
         'STD': 0.01,
         'U_STD': 0.01,
         'VG_R': 1.0,
         'WEIGHT': 1.0,
         'WEIGHT_FINAL': 1.0},
 'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
               'FIXED_LAYERS': 5,
               'REGU_DEPTH': False,
               'WEIGHT_DECAY': 4e-05},
 'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
 'POOLING_SIZE': 7,
 'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
 'RNG_SEED': 3,
 'ROOT_DIR': '/home/zty/iter-reason',
 'TEST': {'MAX_SIZE': 1000, 'SCALES': [600]},
 'TRAIN': {'BATCH_SIZE': 128,
           'BBOX_THRESH': 1.0,
           'BIAS_DECAY': False,
           'DISPLAY': 20,
           'DOUBLE_BIAS': False,
           'GAMMA': 0.1,
           'IMS_PER_BATCH': 1,
           'MAX_SIZE': 1000,
           'MOMENTUM': 0.9,
           'RATE': 0.0005,
           'SCALES': [600],
           'SNAPSHOT_ITERS': 10000,
           'SNAPSHOT_KEPT': 2,
           'SNAPSHOT_PREFIX': 'res101_local',
           'STEPSIZE': [30000],
           'SUMMARY_INTERVAL': 180,
           'SUMMARY_ITERS': 500,
           'USE_FLIPPED': True,
           'WEIGHT_DECAY': 0.0001}}
loading annotations into memory...
Done (t=1.07s)
creating index...
index created!
coco_2014_minival gt roidb loaded from /home/zty/iter-reason/data/cache/coco_2014_minival_gt_roidb.pkl
2018-12-28 11:12:40.903894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with properties: 
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:03:00.0
totalMemory: 10.73GiB freeMemory: 10.52GiB
2018-12-28 11:12:40.903961: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu devices: 0
2018-12-28 11:12:41.438366: I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-12-28 11:12:41.438445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0 
2018-12-28 11:12:41.438464: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N 
2018-12-28 11:12:41.438919: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 10152 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:03:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/zty/iter-reason/tools/../lib/nets/base_memory.py:97: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
ITERATION: 00
ITERATION: 01
ITERATION: 02
WARNING:tensorflow:From /home/zty/iter-reason/tools/../lib/nets/attend_memory.py:104: calling softmax (from tensorflow.python.ops.nn_ops) with dim is deprecated and will be removed in a future version.
Instructions for updating:
dim is deprecated, use axis instead
Loading model check point from output/res101_local/coco_2014_train+coco_2014_valminusminival/32a56_72/res101_local_iter_720000.ckpt
Loaded.
score: 1/5000 10.799s
/home/zty/iter-reason/output/res101_local/coco_2014_minival/32a56_72/res101_local_iter_720000/images/COCO_val2014_000000532481.jpg
score: 2/5000 5.542s
score: 3/5000 3.844s
score: 4/5000 2.918s
score: 5/5000 2.351s
score: 6/5000 1.984s
score: 7/5000 1.761s
score: 8/5000 1.562s
score: 9/5000 1.431s
score: 10/5000 1.303s
score: 11/5000 1.192s
score: 12/5000 1.123s
score: 13/5000 1.065s
score: 14/5000 1.017s
score: 15/5000 0.955s
score: 16/5000 0.900s
score: 17/5000 0.869s
score: 18/5000 0.845s
score: 19/5000 0.808s
score: 20/5000 0.772s
score: 21/5000 0.739s
/home/zty/iter-reason/output/res101_local/coco_2014_minival/32a56_72/res101_local_iter_720000/images/COCO_val2014_000000409630.jpg
score: 22/5000 0.722s
score: 23/5000 0.698s
score: 24/5000 0.676s
score: 25/5000 0.652s
Traceback (most recent call last):
  File "./tools/test_memory.py", line 125, in <module>
    test_net(sess, net, imdb, imdb.roidb, filename, args.visualize, iter_test=iter_test)
  File "/home/zty/iter-reason/tools/../lib/model/test.py", line 36, in test_net
    test_net_base(sess, net, imdb, roidb, weights_filename, visualize)
  File "/home/zty/iter-reason/tools/../lib/model/test.py", line 53, in test_net_base
    all_scores[i], blobs = im_detect(sess, imdb, net, [roidb[i]])
  File "/home/zty/iter-reason/tools/../lib/model/test.py", line 21, in im_detect
    _, scores = net.test_image(sess, blobs)
  File "/home/zty/iter-reason/tools/../lib/nets/attend_memory.py", line 274, in test_image
    feed_dict=self._parse_dict(blobs))
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1328, in _do_run
    run_metadata)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1348, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero
	 [[node SMN_1/fc7/flatten/flatten/Reshape (defined at /home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1623)  = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](SMN_1/mem_ct_pool5/crops, SMN_1/fc7/flatten/flatten/Reshape/shape)]]
	 [[{{node aggregate/attend_cls_score/_1113}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2344_aggregate/attend_cls_score", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op u'SMN_1/fc7/flatten/flatten/Reshape', defined at:
  File "./tools/test_memory.py", line 109, in <module>
    net.create_architecture("TEST", imdb.num_classes, tag='default')
  File "/home/zty/iter-reason/tools/../lib/nets/base_memory.py", line 568, in create_architecture
    rois = self._build_memory(training, testing)
  File "/home/zty/iter-reason/tools/../lib/nets/attend_memory.py", line 178, in _build_memory
    rois, batch_ids, iter)
  File "/home/zty/iter-reason/tools/../lib/nets/attend_memory.py", line 152, in _build_pred
    mem_fc7 = self._fc_iter(mem_ct_pool5, is_training, "fc7", iter)
  File "/home/zty/iter-reason/tools/../lib/nets/base_memory.py", line 173, in _fc_iter
    mem_fc7 = slim.flatten(mem_pool5, scope='flatten')
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/framework/python/ops/arg_scope.py", line 182, in func_with_args
    return func(*args, **current_args)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py", line 1623, in flatten
    outputs = core_layers.flatten(inputs)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/layers/core.py", line 311, in flatten
    return layer.apply(inputs)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 817, in apply
    return self.__call__(inputs, *args, **kwargs)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/layers/base.py", line 374, in __call__
    outputs = super(Layer, self).__call__(inputs, *args, **kwargs)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/keras/engine/base_layer.py", line 757, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/keras/layers/core.py", line 551, in call
    outputs = array_ops.reshape(inputs, (array_ops.shape(inputs)[0], -1))
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 6482, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
    return func(*args, **kwargs)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
    op_def=op_def)
  File "/home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1770, in __init__
    self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Reshape cannot infer the missing input size for an empty tensor unless all specified input sizes are non-zero
	 [[node SMN_1/fc7/flatten/flatten/Reshape (defined at /home/zty/anaconda2/lib/python2.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1623)  = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](SMN_1/mem_ct_pool5/crops, SMN_1/fc7/flatten/flatten/Reshape/shape)]]
	 [[{{node aggregate/attend_cls_score/_1113}} = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2344_aggregate/attend_cls_score", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

I'm using python 2.7, CUDA 9.0, and TensorFlow 1.12.0.
Can anybody help this problem? Thanks!

By the way, does anybody know how to use Visual Genome dataset? I don't know where to find the file "synsets.txt" and how to set parameters.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.