[CVPR2020] Multi-task Collaborative Network for Joint Referring Expression Comprehension and Segmentation, CVPR2020 (oral)
Home Page: https://arxiv.org/abs/2003.08813
License: MIT License
Hi,
Your job is so great!
I perform the experiments by your introductions and achieve 62.8 accuracy for RES. When I perform the ablation study of segmentation, I directly set the
Line 519 in bee86df
I want to ask if it is correct to set the loss functions of loss_rec and loss_cem directly to zero?
Hello, thanks for your wonderful work.
In your paper, you claim that you use the UNC partition for refcocog dataset.
However, in your DATA_PREP only has the umd split
And then if I use the default umd split.
The total expression for train + val + test = 95010, which is much fewer than the original expressions (104560)
Can you help to figure out what's wrong with it? Thanks very much!
Hi Gen Luo,
It's a great job and I enjoy reading your paper very much!
I really did not understand the meaning of the formula at first, until I read some interpretations(artcle) in the website based on the gain and phase margin.
I want to know how did you think of constructing such a CEM loss function？
Hi,
Thank you for the great work! I am wondering how can we train on custom dataset, and what data format do I need to prepare for running the code?
Thank you very much for helping in advance.
When I try to exec the data_process.py, there is a fetal error “Process finished with exit code -1073740940 (0xC0000374)” occured on the sentence "mask.decode(rle)" in getmask of refer.py. But no detail exception info or tip info. So, I hope that you can provide the version of scikit-image,numpy,pycocotools.
Hi, thank you for your great work! I am confused about the RES results of Referit in your paper while comparing with others. For example, results in "Referring Image Segmentation via Cross-Modal Progressive Comprehension." Their results were so high. How do you think about it? thx.
ssh://[email protected]:22/home/supermicro-2/anaconda3/envs/MCN/bin/python -u /home/supermicro-2/ldz_project/MCN/train.py
Using TensorFlow backend.
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
WARNING:tensorflow:From /home/supermicro-2/ldz_project/MCN/train.py:19: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
train len 42404
val len 3811
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:102: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
2021-03-04 15:24:38.356658: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2021-03-04 15:24:38.363774: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2021-03-04 15:24:40.556049: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f5b08afce0 executing computations on platform CUDA. Devices:
2021-03-04 15:24:40.556136: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556159: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (1): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556181: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (2): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556214: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (3): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556243: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (4): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556271: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (5): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556298: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (6): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.556326: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (7): GeForce RTX 2080 Ti, Compute Capability 7.5
2021-03-04 15:24:40.589834: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3099990000 Hz
2021-03-04 15:24:40.593161: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55f5b07fd5b0 executing computations on platform Host. Devices:
2021-03-04 15:24:40.593235: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): <undefined>, <undefined>
2021-03-04 15:24:40.596605: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:04:00.0
2021-03-04 15:24:40.599523: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 1 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:05:00.0
2021-03-04 15:24:40.602416: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 2 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:08:00.0
2021-03-04 15:24:40.605263: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 3 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:09:00.0
2021-03-04 15:24:40.607020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 4 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:84:00.0
2021-03-04 15:24:40.608469: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 5 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:85:00.0
2021-03-04 15:24:40.609915: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 6 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:88:00.0
2021-03-04 15:24:40.611352: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 7 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.545
pciBusID: 0000:89:00.0
2021-03-04 15:24:40.611606: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory
2021-03-04 15:24:40.611729: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory
2021-03-04 15:24:40.611847: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory
2021-03-04 15:24:40.611962: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory
2021-03-04 15:24:40.612077: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory
2021-03-04 15:24:40.612190: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Could not dlopen library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory
2021-03-04 15:24:40.617722: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-03-04 15:24:40.617754: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1663] Cannot dlopen some GPU libraries. Skipping registering GPU devices...
2021-03-04 15:24:40.617840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-03-04 15:24:40.617859: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0 1 2 3 4 5 6 7
2021-03-04 15:24:40.617870: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N N N N N N N N
2021-03-04 15:24:40.617879: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 1: N N N N N N N N
2021-03-04 15:24:40.617888: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 2: N N N N N N N N
2021-03-04 15:24:40.617897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 3: N N N N N N N N
2021-03-04 15:24:40.617905: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 4: N N N N N N N N
2021-03-04 15:24:40.617914: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 5: N N N N N N N N
2021-03-04 15:24:40.617922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 6: N N N N N N N N
2021-03-04 15:24:40.617930: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 7: N N N N N N N N
2021-03-04 15:24:40.659361: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=--tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=--xla_hlo_profile.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
Instructions for updating:
Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead.
WARNING:tensorflow:From /home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py:3980: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.
Traceback (most recent call last):
File "/home/supermicro-2/ldz_project/MCN/train.py", line 134, in <module>
learner=Learner()
File "/home/supermicro-2/ldz_project/MCN/train.py", line 52, in __init__
self.yolo_model, self.yolo_body = self.create_model(yolo_weights_path=config['pretrained_weights'],freeze_body=config['free_body'])
File "/home/supermicro-2/ldz_project/MCN/train.py", line 89, in create_model
model_body.load_weights(yolo_weights_path, by_name=True, skip_mismatch=True)
File "/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/keras/engine/network.py", line 1157, in load_weights
with h5py.File(filepath, mode='r') as f:
File "/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/h5py/_hl/files.py", line 427, in __init__
swmr=swmr)
File "/home/supermicro-2/anaconda3/envs/MCN/lib/python3.6/site-packages/h5py/_hl/files.py", line 190, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 96, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name = './data/weights/yolov3_450000.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
Process finished with exit code 1
Where can I get yolov3_450000.h5 ?
Thank you.
Dear sir, thank for your work. I'd like to use bert instead of blstm when text embedding. error occurs as follow:
File "train.py", line 135, in
learner=Learner()
File "train.py", line 52, in init
self.yolo_model, self.yolo_body = self.create_model(yolo_weights_path=config['pretrained_weights'],freeze_body=config['free_body'])
File "train.py", line 87, in create_model
model_body = yolo_body(image_input, q_input, num_anchors,config) ###### place
File "/datadrive/soat_nlp/MCN/MCN-master/model/mcn_model.py", line 170, in yolo_body
return Model([inputs,q_input], [y,E,co_enery])
File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/network.py", line 93, in init
self._init_graph_network(*args, **kwargs)
File "/usr/local/lib/python3.5/dist-packages/keras/engine/network.py", line 147, in _init_graph_network
if len(set(self.inputs)) != len(self.inputs):
TypeError: unhashable type: 'list'
Hello, I just read your paper and thanks for sharing the source code of this great work!
I want to reproduce your work, but the pretrained weights download link is not working.
I think this one-drive link is kind of unstable.. download button shows up, but it doesn't respond anything.
Can you check the link, please?
Thank you.
Hello.It seemed like the link is invalid in my computer,does anyone download the weight successfully?
Hi, thank you for your work! I want to know how you get your anchors for the datasets? Could you share the code or anything else? I did not find any detailed explainations in your paper. Only a little information is shown in model/mcn_model.py line 451:
anchor_mask = [[6,7,8], [3,4,5], [0,1,2]] if num_layers==3 else [[0,1,2]] ##due to deleting 2 scales change [[6,7,8], [3,4,5], [0,1,2]] to [[0,1,2]]But the data in your data/yolo_anchors.txt is
137,256, 248,272, 386,271for [[0,1,2]], deleting [[6,7,8], [3,4,5] scales. I want to get the whole anchors for the datasets.
Hearing from you, thanks~
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.