Hi, I have a little question about the contour sampling. In the Section 3.2 / Contour deformation of your paper, you mentioned that

We first uniformly sample the octagon contour N points along its edges and let it start from the top extreme points x^ex_1.

However, I found it hard to achieve "uniformly sampling" while trying to implement deep snake. If I start from one extreme point and sample 1 point for every perimeter / N distance around the octagon, the other 7 vertices (or 11?) of the octagon may not be in the sampled point set at all. And if I sample (N - 12) / 4 points uniformly along each of the 4 slopes, it doesn't look reasonable because their lengths are different.

Could you explain more about your sampling strategy? Thank you!

In case wanna reproduce paper results, when will code release?

运行命令：python run.py --type evaluate --cfg_file configs/sbd_snake.yaml test.dataset SbdVal
出现段错误（核心已转储），请问大佬有遇到过吗？或者您有什么好的建议？

I`m interested in this project. And tried to deploy it, but visualisation failing every time in the same point.

This error could be healed by downgrading tqdm to 4.19 instead ~4.25 or upgrading to 4.40+. But there are strange another error in same place

Do you know anything about this?

I use nvidia tesla, cuda-10, i`ve compiled apex and _ext from install.md instruction.
And pytorch is convinient to cuda version: https://download.pytorch.org/whl/cu100/torch-1.0.0-cp36-cp36m-linux_x86_64.whl

vignette: “Figure 7 shows some qualitative results on Kins. ” in page 7 should be
“Figure 6 (row 3) shows some qualitative results on Kins. ”

I am confused
how to understand amodal and inmodal or modal in accurate chinese?

i follow the readme, and i run python train_net.py, but i get this error:

and i don't find train in lib, am i miss something?

运行测试指令出现问题，好像是说模型损坏？

作者您好，非常感谢能开源这么优秀的代码。我在训练自己的数据。网络输入尺寸在代码里固定为896*384，我想知道这个值是比较特殊的值吗?我如果改成更大或者更小一点，对训练结果有精度影响吗?

At first glance, you have chosen a traditional strategy known as "detect and segment" widely used in previous methods. I was wondering why didn't you choose FPN to acquire multiple-level information or follow the style of FCOS?

Hi. Thank you very much for your contribution.
I have a problem while running the testing code below:
python run.py --type evaluate --cfg_file configs/city_rcnn_snake.yaml
Below is the error prompted:
ImportError: /home/tunghandsome/Tung/snake/snake/lib/csrc/dcn_v2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E
Does anyone know how to fix this error?

Traceback (most recent call last):
  File "run.py", line 99, in <module>
    globals()['run_'+args.type]()
  File "run.py", line 51, in run_evaluate
    network = make_network(cfg).cuda()
  File "/home/tunghandsome/Tung/snake/snake/lib/networks/make_network.py", line 23, in make_network
    return imp.load_source(module, path).get_network(cfg)
  File "/home/tunghandsome/snake/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "lib/networks/rcnn_snake/__init__.py", line 1, in <module>
    from .ct_rcnn_snake import get_network as get_rcnn
  File "lib/networks/rcnn_snake/ct_rcnn_snake.py", line 2, in <module>
    from .dla import DLASeg
  File "lib/networks/rcnn_snake/dla.py", line 16, in <module>
    from lib.networks.dcn_v2 import DCN
  File "/home/tunghandsome/Tung/snake/snake/lib/networks/dcn_v2.py", line 13, in <module>
    from lib.csrc.dcn_v2 import _ext as _backend
ImportError: /home/tunghandsome/Tung/snake/snake/lib/csrc/dcn_v2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

Thanks for your contribution !
How to change the code about the number of dataset.
And could you tell me the design mode. I 'm missing in same code.

When opening the source code

Hello, I have converted my own data set to the coco data set format and modified the data path, but during the training process, the loss is changing, but all ap, ar values are almost 0 ,the result of visualization is equivalent to the original image, without any detection frame and segmentation effect. I think the data set is not really trained, have I missed any parameter adjustment?

Excellent and inspiring work it is!
But I wander how to train and apply the algorithm on
my own instance segmentation dataset ?

I want to use this algorithm for cells and nucleus instance segmentation. The instance is annotated
with .png format or the widely used .json.

I have download and carefully reviewed your code and project structure but it is too complicated and there is little comment about how the code works or the return result of the function.

Can you give some instructions or tutorials like the popular used repo: https://github.com/matterport/Mask_RCNN

I think it will be a nice work and inspre more wonderful ideas. Thank for your contribution to the community again.

It seems the pre-trained detection model for CityScapes is not provided?
Sincerely thank you!

The backbone is some 'CirConv's in your paper, that is

Then what is the f_N mean? Is it the whole feature and the (f_N)i means the feature of a pixel, which is a vertice?
Could you explain more about your circular convolution implementation, thanks!!

@pengsida
Hi there,

it would be very interesting testing the benefits of SNAKE in text detection.
Have you ever considered that.

笔者你好，今天将代码在Kins上运行，出现如下错误，不知道问题出在哪

index created!
/home/mil/ruan/.pyenv/versions/anaconda3-2019.07/envs/snake1/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
warnings.warn('Was asked to gather along dimension 0, but all '
Traceback (most recent call last):
File "train_net.py", line 54, in
main()
File "train_net.py", line 50, in main
train(cfg, network)
File "train_net.py", line 25, in train
trainer.train(epoch, train_loader, optimizer, recorder)
File "/home/mil/ruan/home/mil/ruan/snake/snake-master/lib/train/trainers/trainer.py", line 43, in train
loss.backward()
File "/home/mil/ruan/.pyenv/versions/anaconda3-2019.07/envs/snake1/lib/python3.7/site-packages/torch/tensor.py", line 102, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/mil/ruan/.pyenv/versions/anaconda3-2019.07/envs/snake1/lib/python3.7/site-packages/torch/autograd/init.py", line 90, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

I constantly get this error in almost every 4-5 epoch training.

Traceback (most recent call last):
  File "train_net.py", line 57, in <module>
    main()
  File "train_net.py", line 53, in main
    train(cfg, network)
  File "train_net.py", line 35, in train
    trainer.val(epoch, val_loader, evaluator, recorder)
  File "/ldap_shared/home/s_zyt/inseg/code/snake/lib/train/trainers/trainer.py", line 76, in val
    for batch in tqdm.tqdm(data_loader):
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/tqdm/_tqdm.py", line 979, in __iter__
    for obj in iterable:
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 346, in __next__
    data = self.dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
    return self.collate_fn(data)
  File "/ldap_shared/home/s_zyt/inseg/code/snake/lib/datasets/collate_batch.py", line 7, in snake_collator
    ret = {'inp': default_collate([b['inp'] for b in batch])}
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 65, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 544 and 480 in dimension 2 at /pytorch/aten/src/TH/generic/THTensor.cpp:689

Should the validation dataloader worker number be set as 0?
Previously, I got this error in train dataloader, so I set the num_worker=0, it works.

Configuration as follow:

train:
    optim: 'adam'
    lr: 1e-4
    milestones: (80, 120, 150, 170)
    gamma: 0.5
    batch_size: 16
    dataset: 'CocoTrain'
    num_workers: 0
    epoch: 150
test:
    # dataset: 'SbdMini'
    dataset: 'CocoVal'
    batch_size: 16
    num_workers: 0

My operating system is Ubuntu16.04.
Thank you!

Windows10
Python3.7

cd dcn_v2
python setup.py build_ext --inplace
cd ../extreme_utils
python setup.py build_ext --inplace

when I enter
python setup.py build_ext --inplace
I got diffirent error from each of them

cd dcn_v2:

dcn_v2_cuda.obj : error LNK2001: 無法解析的外部符號 "struct THCState * state" (?state@@3PEAUTHCState@@ea)
build\lib.win-amd64-3.7_ext.cp37-win_amd64.pyd : fatal error LNK1120: 1 個無法解析的外部符號
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\bin\HostX86\x64\link.exe' failed with exit status 1120

cd extreme_utils:

6 errors detected in the compilation of "C:/Users/evans/AppData/Local/Temp/tmpxft_000000e8_00000000-8_utils.cpp4.ii".
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\nvcc.exe' failed with exit status 2

any paper link or code info before it release?

感谢作者优秀的工作！
最近受到deep snake的启发，想要将其应用到医学数据集中，用于血管的boundary detection。 然而由于数据集结构的不同，我的数据集并没有对应的json文件，并且没有对应的bbox的label。我的数据集所拥有的只有原CTimage 与 所对应的血管内外壁 boundary 的mask。 自己想通过修改作者的代码把数据集移植进去但遇到了困难。 我观察到作者似乎是通过tools.convert_sbd 来对数据集进行处理的。
我想首先听听作者的意见，以及如果可以的话能否share一份处理不同数据集的demo呢？
感激不尽！

Thanks for sharing the paper. We would like to do a research on a V100 GPU and 2080TI. Does the code support multi-GPU?

Can you explain what act_hm, ct_hm and wh in yaml ?
Thanks

I have read your code and found there is no offset, but only 'py_pred'.

Why don't you show results on coco dataset?

It's a nice idea, but I can't install this project in Windows10, and every "requirement" are satisfied. Follow the INSTALL.md， “install apex" ” have error " C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\include\crt/host_config.h(133): fatal error C1189: #error: -- unsupported Microsoft Visual Studio version! Only the versions 2012, 2013, 2015 and 2017 are supported!
error: command 'C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\nvcc.exe' failed with exit status 2". I have installed VS2017 and VS2019. Do you know what's the problem? Or how to change to VS2017

Hi Authors!

Thanks for the nice work! Waiting for the code to play with it~

Just a quick question: did you try the model on COCO dataset and whether it can perform well on that much more challenging dataset as well?

您好！
我认真阅读了您的论文，但是有些疑惑
section 3.2 Initial contour proposal.
“the diamond contour is uniformly upsampled to 40 points, and deep snake correspondingly outputs 40 offsets.”

为什么要对菱形轮廓采40个样？ 后面的loss function 不是只用到了4个点吗？能否解答一下，感激不尽！谢谢！

RT, thanks very much.

I have trained your model successfully yesterday, Now I'm working on testing my obtained model on cityscapes. However, No matter how I tried, a CUDA error always exists. I'm not familiar with CUDA programming, so I have to open an issue to trouble you @pengsida Have you ever encounter this kind of problem? thank you in advance.

loading annotations into memory...
Done (t=0.17s)
creating index...
index created!
loading annotations into memory...
Done (t=0.14s)
creating index...
index created!
  0%|                                                             | 0/492 [00:00<?, ?it/s]THCudaCheck FAIL file=/home/fei/snake/lib/csrc/extreme_utils/src/utils.cu line=299 error=9 : invalid configuration argument

Traceback (most recent call last):
  File "run.py", line 89, in <module>
    globals()['run_'+args.type]()
  File "run.py", line 60, in run_evaluate
    output = network(inp)
  File "/home/fei/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "lib/networks/rcnn_snake/ct_rcnn_snake.py", line 42, in forward
    output = self.gcn(output, cnn_feature, batch)
  File "/home/fei/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in __call__
    result = self.forward(*input, **kwargs)
  File "lib/networks/rcnn_snake/evolve.py", line 111, in forward
    init = self.prepare_testing_init(output)
  File "lib/networks/rcnn_snake/evolve.py", line 41, in prepare_testing_init
    i_it_4py = snake_gcn_utils.uniform_upsample(i_it_4py, snake_config.init_poly_num)
  File "/home/fei/snake/lib/utils/snake/snake_gcn_utils.py", line 226, in uniform_upsample
    extreme_utils.calculate_edge_num(edge_num, edge_num_sum, edge_idx_sort, p_num)
RuntimeError: cuda runtime error (9) : invalid configuration argument at /home/fei/snake/lib/csrc/extreme_utils/src/utils.cu:299

Compile the dcn_v2 some error as follow, is there any idea ?

required from here /usr/local/include/c++/6.1.0/tuple:495:244: error: wrong number of template arguments (5, should be 2)

Why was this happend ?

Traceback (most recent call last):
  File "train_net.py", line 54, in <module>
    main()
  File "train_net.py", line 50, in main
    train(cfg, network)
  File "train_net.py", line 25, in train
    trainer.train(epoch, train_loader, optimizer, recorder)
  File "/ldap_shared/home/s_zyt/inseg/code/snake/lib/train/trainers/trainer.py", line 38, in train
    output, loss, loss_stats, image_stats = self.network(batch)
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 152, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 162, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/_utils.py", line 369, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "lib/train/trainers/snake.py", line 24, in forward
    ct_loss = self.ct_crit(net_utils.sigmoid(output['ct_hm']), batch['ct_hm'])
  File "/ldap_shared/home/s_zyt/.pyenv/versions/anaconda3-2019.07/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in __call__
    result = self.forward(*input, **kwargs)
  File "/ldap_shared/home/s_zyt/inseg/code/snake/lib/utils/net_utils.py", line 50, in forward
    return self.neg_loss(out, target)
  File "/ldap_shared/home/s_zyt/inseg/code/snake/lib/utils/net_utils.py", line 29, in _neg_loss
    pos_loss = torch.log(pred) * torch.pow(1 - pred, 2) * pos_inds
RuntimeError: The size of tensor a (20) must match the size of tensor b (80) at non-singleton dimension 1

How to reproduce:

I used the COCO format dataset. And run:

python train_net.py --cfg_file configs/sbd_snake.yaml model custom_model train.dataset CocoTrain test.dataset CocoVal

Good job, but what’s the performance on the coco benchmark?

I only find component detector in training code for cityscapes dataset.

作者你好，请问如何用您的代码和模型，直接分割一张已经得到boundingbox的图片？

感谢作者优秀的工作！
请问，能否写个输入是boxes 和 image时的demo？

是否仅需要修改ct_snake.py的forward函数的
ct, detection = self.decode_detection(output, cnn_feature.size(2), cnn_feature.size(3))
这条语句，由外部输入的boxes来生成ct。
谢谢！

Can I ask from more function or explain?
I have read the paper for many times.
But I still can't understand how to turn contour(blue) into Object shape(green).
Thanks

I insert of speed test code in the function run_visualize:

def run_visualize():
    from lib.networks import make_network
    from lib.datasets import make_data_loader
    from lib.utils.net_utils import load_network
    import tqdm
    import torch
    from lib.visualizers import make_visualizer

    network = make_network(cfg).cuda()
    load_network(network, cfg.model_dir, resume=cfg.resume, epoch=cfg.test.epoch)
    network.eval()

    data_loader = make_data_loader(cfg, is_train=False)
    visualizer = make_visualizer(cfg)

    time_use = 0
    for batch in tqdm.tqdm(data_loader):
        for k in batch:
            if k != 'meta':
                batch[k] = batch[k].cuda()
        with torch.no_grad():
            start = time()
            output = network(batch['inp'], batch)
            end = time()
            time_use += (end - start)
        visualizer.visualize(output, batch)
    print("FPS: ", len(data_loader) / time_use)

And I got FPS:

FPS:  5.974452101070888

It seems the speed is far slower from real time. Is there any problem in my test code.
The size of my evaluation data is mostly (544, 768).

The backbone is some 'CirConv's in your paper, that is

Then is f_N the hstack(or vstack) of N features? If I want to do the forward, should I take the stack of N features as input, or you really do this by the formula of right side?

@pengsida
Iam having problem while compiling the library.
Have you met these errors when compiling ?

After I finished the setting from your INSTALL.md and when I ran the testing code:

python run.py --type evaluate --cfg_file configs/city_rcnn_snake.yaml test.dataset CityscapesVal

I saw these errors:

WARNING: NO MODEL LOADED !!!
load model: data/model/rcnn_snake/long_rcnn/197.pth
0%| | 0/492 [00:00<?, ?it/s]torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File "run.py", line 99, in
globals()'run_'+args.type
File "run.py", line 57, in run_evaluate
for batch in tqdm.tqdm(data_loader):
File "/home/andrew/anaconda3/lib/python3.7/site-packages/tqdm/std.py", line 1119, in iter
for obj in iterable:
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 345, in next
data = self._next_data()
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 856, in _next_data
return self._process_data(data)
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 881, in _process_data
data.reraise()
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/_utils.py", line 394, in reraise
raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/worker.py", line 178, in _worker_loop
data = fetcher.fetch(index)
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/fetch.py", line 47, in fetch
return self.collate_fn(data)
File "/home/andrew/Documents/snake-master/lib/datasets/collate_batch.py", line 94, in rcnn_snake_collator
ret = {'inp': default_collate([b['inp'] for b in batch])}
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 64, in default_collate
return default_collate([torch.as_tensor(b) for b in batch])
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/utils/data/_utils/collate.py", line 53, in default_collate
storage = elem.storage()._new_shared(numel)
File "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/storage.py", line 128, in _new_shared
return cls._new_using_filename(size)
RuntimeError: error executing torch_shm_manager at "/home/andrew/anaconda3/lib/python3.7/site-packages/torch/bin/torch_shm_manager" at /pytorch/torch/lib/libshm/core.cpp:99

torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
torch_shm_managertorch_shm_manager: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
: error while loading shared libraries: libcudart.so.10.1: cannot open shared object file: No such file or directory
0%| | 0/492 [00:01<?, ?it/s]

My environment are as fellows:
Pytorch: 1.4.0
Torchvision: 0.5.0
Ubuntu: 18.04.2
CUDA version(from "torch.version.cuda"): 10.1

And I have already checked the path of your pretrained model, which is $ROOT/data/model/rcnn_snake/long_rcnn/197.pth

So could you please help me to figure out what is going on? I would be very honored if you can help me!

Thanks!

I measured forward time around 130 ms on GPU, is this speed normal?

How to train, test, and visualize on our coco-type dataset?