Snake - Code for "Deep Snake for Real-Time Instance Segmentation" CVPR 2020 oral

ZJU3DV

Last update: Dec 26, 2022

Related tags

Game Development snake

Overview

Good news! Snake algorithms exhibit state-of-the-art performances on COCO dataset: DANCE

Deep Snake for Real-Time Instance Segmentation

Deep Snake for Real-Time Instance Segmentation
Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, Xiaowei Zhou
CVPR 2020 oral

Any questions or discussions are welcomed!

Installation

Please see INSTALL.md.

Testing

Testing on Cityscapes

Download the pretrained model here and put it to $ROOT/data/model/rcnn_snake/long_rcnn/197.pth.

Test:

# use coco evaluator
python run.py --type evaluate --cfg_file configs/city_rcnn_snake.yaml
# use the cityscapes official evaluator
python run.py --type evaluate --cfg_file configs/city_rcnn_snake.yaml test.dataset CityscapesVal

Speed:

python run.py --type network --cfg_file configs/city_rcnn_snake.yaml

Testing on Kitti

Download the pretrained model here and put it to $ROOT/data/model/snake/kins/149.pth.

Test:

python run.py --type evaluate --cfg_file configs/kins_snake.yaml test.dataset KinsVal

Speed:

python run.py --type network --cfg_file configs/kins_snake.yaml test.dataset KinsVal

Testing on Sbd

Download the pretrained model here and put it to $ROOT/data/model/snake/sbd/149.pth.

Test:

python run.py --type evaluate --cfg_file configs/sbd_snake.yaml test.dataset SbdVal

Speed:

python run.py --type network --cfg_file configs/sbd_snake.yaml test.dataset SbdVal

Visualization

Visualization on Cityscapes

Download the pretrained model here and put it to $ROOT/data/model/rcnn_snake/long_rcnn/197.pth.

Visualize:

# Visualize Cityscapes test set
python run.py --type visualize --cfg_file configs/city_rcnn_snake.yaml test.dataset CityscapesTest ct_score 0.3
# Visualize Cityscapes val set
python run.py --type visualize --cfg_file configs/city_rcnn_snake.yaml test.dataset CityscapesVal ct_score 0.3

If setup correctly, the output will look like

Visualization on Kitti

Download the pretrained model here and put it to $ROOT/data/model/snake/kins/149.pth.

Visualize:

python run.py --type visualize --cfg_file configs/kins_snake.yaml test.dataset KinsVal ct_score 0.3

Visualization on Sbd

Download the pretrained model here and put it to $ROOT/data/model/snake/sbd/149.pth.

Visualize:

python run.py --type visualize --cfg_file configs/sbd_snake.yaml test.dataset SbdVal ct_score 0.3

Demo

We support demo for image and image folder using python run.py --type demo --cfg_file /path/to/yaml_file demo_path /path/to/image ct_score 0.3.

For example:

python run.py --type demo --cfg_file configs/sbd_snake.yaml demo_path demo_images ct_score 0.3
# or
python run.py --type demo --cfg_file configs/sbd_snake.yaml demo_path demo_images/2009_000871.jpg ct_score 0.3

If setup correctly, the output will look like

Training

The training parameters can be found in project_structure.md.

Training on Cityscapes

Two-stage training:

Train the detector:

python train_net.py --cfg_file configs/city_ct_rcnn.yaml model rcnn_det

Train the detector and snake together:

python train_net.py --cfg_file configs/city_rcnn_snake.yaml model rcnn_snake det_model rcnn_det

Training on Kins

python train_net.py --cfg_file configs/kins_snake.yaml model kins_snake

Training on Sbd

python train_net.py --cfg_file configs/sbd_snake.yaml model sbd_snake

Tensorboard

We provide tensorboard for seeing the training status:

# for the rcnn_snake task
tensorboard --logdir data/record/rcnn_snake
# for the snake task
tensorboard --logdir data/record/snake

If setup correctly, the output will look like

Citation

If you find this code useful for your research, please use the following BibTeX entry.

@inproceedings{peng2020deep,
  title={Deep Snake for Real-Time Instance Segmentation},
  author={Peng, Sida and Jiang, Wen and Pi, Huaijin and Li, Xiuli and Bao, Hujun and Zhou, Xiaowei},
  booktitle={CVPR},
  year={2020}
}

Comments

Details of the Circular Convolution

The backbone is some 'CirConv's in your paper, that is Then is f_N the hstack(or vstack) of N features? If I want to do the forward, should I take the stack of N features as input, or you really do this by the formula of right side?

opened by LetsGoFir 45
对于output有一些不解

作者您好，我在移植您代码到医疗图像的过程中对于输出的量有一些不解： def evaluate(self, output, batch): detection = output['detection'] score = detection[:, 4].detach().cpu().numpy() label = detection[:, 5].detach().cpu().numpy().astype(int) py = output['py'][-1].detach().cpu().numpy() * snake_config.down_ratio 由于我需要对结果进行IOU或者Hausdorff distance的计算，在这个detection的key中我输出了label与detection同时进行观察。却发现结果和我理解的有些不一致，能否解释一下output中detection key包含了什么？以下是detection的sample： detection: tensor([[17.1544, 18.0598, 28.8456, 29.9402, 0.8509, 1.0000], [17.1544, 18.0598, 28.8456, 29.9402, 0.8111, 0.0000]], device='cuda:0') label: [1 0]

其中的label又是代表什么？

opened by Ruanyuzhe 34
model.load_pretrained_model(data='imagenet', name='dla34', hash='ba72cf86')下载预训练模型报错？

kins_snake Traceback (most recent call last): File "train_net.py", line 54, in main() File "train_net.py", line 46, in main network = make_network(cfg) File "/data/hongjq/object_detect/snake/lib/networks/make_network.py", line 23, in make_network return imp.load_source(module, path).get_network(cfg) File "lib/networks/snake/init.py", line 17, in get_network network = get_model(num_layers, heads, head_conv, snake_config.down_ratio, cfg.det_dir) File "lib/networks/snake/ct_snake.py", line 64, in get_network network = Network(num_layers, heads, head_conv, down_ratio, det_dir) File "lib/networks/snake/ct_snake.py", line 19, in init head_conv=head_conv) File "lib/networks/snake/dla.py", line 432, in init self.base = globals()base_name File "lib/networks/snake/dla.py", line 315, in dla34 model.load_pretrained_model(data='imagenet', name='dla34', hash='ba72cf86') File "lib/networks/snake/dla.py", line 301, in load_pretrained_model model_weights = model_zoo.load_url(model_url) File "/data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/hub.py", line 506, in load_state_dict_from_url return torch.load(cached_file, map_location=map_location) File "/data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/serialization.py", line 529, in load return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args) File "/data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/serialization.py", line 709, in _legacy_load deserialized_objects[key].set_from_file(f, offset, f_should_read_directly) RuntimeError: unexpected EOF, expected 885842 more bytes. The file might be corrupted. terminate called after throwing an instance of 'c10::Error' what(): owning_ptr == NullType::singleton() || owning_ptr->refcount.load() > 0 INTERNAL ASSERT FAILED at /pytorch/c10/util/intrusive_ptr.h:348, please report a bug to PyTorch. intrusive_ptr: Can only intrusive_ptr::reclaim() owning pointers that were created using intrusive_ptr::release(). (reclaim at /pytorch/c10/util/intrusive_ptr.h:348) frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x33 (0x7fe0a0fcc193 in /data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/lib/libc10.so) frame #1: + 0x18cd59f (0x7fdfd591059f in /data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #2: THStorage_free + 0x17 (0x7fdfd60d8ba7 in /data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/lib/libtorch.so) frame #3: + 0x55d4dd (0x7fe0a82c44dd in /data/tools/anaconda3/envs/snake/lib/python3.7/site-packages/torch/lib/libtorch_python.so) frame #26: __libc_start_main + 0xe7 (0x7fe0ac795b97 in /lib/x86_64-linux-gnu/libc.so.6)

Aborted (core dumped) 我已经下载了dla34-ba72cf86.pth放在snake/lib/networks/snake/dla34-ba72cf86.pth下，请问这个模型是干什么的

opened by seawater668 18
how to make coco dataset label not to Too sensitive?

hello! i need help for coco dataset label now i can use coco dataset label to train snake model but it is Too sensitive many image is not have target than model label that

i have search for the way to "coco dataset label there is no target !" like this but i dont found any solution...

@pengsida can you help me ? thank you! 😊

opened by zoearthmoon 17
The problem about DCN

Every time I run to the DCN module, the program stops with no error, just "Process Finished with exit code-1". I tried to debug where it was wrong and found that the program went to stop at _back.dcn_v2_forward in DCN_v2_conv. My CUDA version is 9.0, and pytorch and other machines are strictly installed in accordance with the version in Install. Could you please tell me what the problem is?

opened by u53216 14
extreme_utils build_ext error

hello when i use

cd ../extreme_utils python setup.py build_ext --inplace it will show error

creating C:\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\build\lib.win-amd64-3.7 C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPATH:C:\Python37\lib\site-packages\torch\lib "/LIBPATH:C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\lib/x64" /LIBPATH:C:\Python37\libs /LIBPATH:C:\Python37\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\ATLMFC\lib\x64" "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.26.28801\lib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\NETFXSDK\4.8\lib\um\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" cudart.lib c10.lib c10_cuda.lib torch.lib torch_python.lib _C.lib /EXPORT:PyInit__ext build\temp.win-amd64-3.7\Release\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\utils.obj build\temp.win-amd64-3.7\Release\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\src\nms.obj build\temp.win-amd64-3.7\Release\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\src\utils.obj /OUT:build\lib.win-amd64-3.7\_ext.cp37-win_amd64.pyd /IMPLIB:build\temp.win-amd64-3.7\Release\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\_ext.cp37-win_amd64.lib 正在建立程式庫 build\temp.win-amd64-3.7\Release\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\_ext.cp37-win_amd64.lib 和物件 build\temp.win-amd64-3.7\Release\Users\T34808\Desktop\snake\lib\csrc\extreme_utils\_ext.cp37-win_amd64.exp utils.obj : error LNK2001: 無法解析的外部符號 "public: long * __cdecl at::Tensor::data_ptr<long>(void)const " (??$data_ptr@J@Tensor@at@@QEBAPEAJXZ) build\lib.win-amd64-3.7\_ext.cp37-win_amd64.pyd : fatal error LNK1120: 1 個無法解析的外部符號 error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.26.28801\\bin\\HostX86\\x64\\link.exe' failed with exit status 1120

i am use pytorch 1.4 + cu10.1 + windows please help~ thank you~

opened by zoearthmoon 14

Error when testing

Hi. Thank you very much for your contribution. I have a problem while running the testing code below: python run.py --type evaluate --cfg_file configs/city_rcnn_snake.yaml Below is the error prompted: ImportError: /home/tunghandsome/Tung/snake/snake/lib/csrc/dcn_v2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E Does anyone know how to fix this error?

Traceback (most recent call last):
  File "run.py", line 99, in <module>
    globals()['run_'+args.type]()
  File "run.py", line 51, in run_evaluate
    network = make_network(cfg).cuda()
  File "/home/tunghandsome/Tung/snake/snake/lib/networks/make_network.py", line 23, in make_network
    return imp.load_source(module, path).get_network(cfg)
  File "/home/tunghandsome/snake/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "lib/networks/rcnn_snake/__init__.py", line 1, in <module>
    from .ct_rcnn_snake import get_network as get_rcnn
  File "lib/networks/rcnn_snake/ct_rcnn_snake.py", line 2, in <module>
    from .dla import DLASeg
  File "lib/networks/rcnn_snake/dla.py", line 16, in <module>
    from lib.networks.dcn_v2 import DCN
  File "/home/tunghandsome/Tung/snake/snake/lib/networks/dcn_v2.py", line 13, in <module>
    from lib.csrc.dcn_v2 import _ext as _backend
ImportError: /home/tunghandsome/Tung/snake/snake/lib/csrc/dcn_v2/_ext.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZN6caffe26detail36_typeMetaDataInstance_preallocated_7E

opened by tunghustvn 14

Fine-tuning on a customized dataset
Hi, I am trying to fine-tune a pre-trained model on a customized dataset. My doubts are:

Is it possible to modify the training protocol or, in other words, is it possible to train a pre-trained model keeping the weights of specific layers frozen and others not? If it is and considering the task of object segmentation in a scenario with objects placed on a table, have you investigated which layers of the model is convenient to re-train and which ones is convenient to froze during the training phase?

Could you kindly provide, if possible, the weights of your model (.pth) trained on the COCO dataset if possible?

Finally I am wondering if it could be possible during the training phase to load a pre-trained model but excluding only the last layer and initializing the weights of a new final layer in a randomic way for example?

Thank you for your attention and your work!
opened by tommi95 13
显卡配置问题

请问这个数据可以在RTX 3070显卡上跑吗？为什么我一直都是显示：NO MODEL LOADED!!!,然后就是错误1：RuntimeError：cuDNN error：CUDNN_STATUS_MAPPING_ERROR，错误2：RuntimeError：cannot join current thread，这个是显卡太新造成的吗？是不是只能换个显卡才行？

谢谢您！！

opened by monarchdemon 12
what's sbd_snake mean?

Excuse me , i can't understand what the data is traing for, can i choose a item to train, and get a higher detection? And below cmd i can't understand what's the model "sbd_snake" mean?

my project is going to detect screw and nut and return position meanwhile, can you guide some direction for me?

Apologize for freshman in opencv, thx

opened by Aaron-Ace 10

RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51

(snake) ws@ws-OMEN-Laptop-15-ek0xxx:~/snake$ python run.py --type demo --cfg_file configs/sbd_snake.yaml demo_path demo_images/2009_000871.jpg ct_score 0.3
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error
Traceback (most recent call last):
  File "run.py", line 99, in <module>
    globals()['run_'+args.type]()
  File "run.py", line 95, in run_demo
    demo.demo()
  File "/home/ws/snake/tools/demo.py", line 59, in demo
    network = make_network(cfg).cuda()
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 265, in cuda
    return self._apply(lambda t: t.cuda(device))
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 193, in _apply
    module._apply(fn)
  [Previous line repeated 1 more time]
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 199, in _apply
    param.data = fn(param.data)
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/nn/modules/module.py", line 265, in <lambda>
    return self._apply(lambda t: t.cuda(device))
  File "/home/ws/anaconda3/envs/snake/lib/python3.7/site-packages/torch/cuda/__init__.py", line 163, in _lazy_init
    torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51

opened by wangshuo2020 10

I can not find those files, can you help me ?

Thanks for your great work, but when I run this code "python train_net.py --cfg_file configs/city_ct_rcnn.yaml model rcnn_det". I can not find "make_trainer, make_optimizer, make_lr_scheduler, make_recorder, set_lr_scheduler" from lib.train Can you help me ?
Tranks for your reply.

opened by H-0813 0
get_affine_transform() is buggy

The get_affine_transform() function seems buggy and appear in many github repos.

https://github.com/zju3dv/snake/blob/2c9b0a6788a9fd38e95a22e2ac7aabbb1a9303c5/lib/utils/data_utils.py#L138-L139

It should be src_h and dst_h for the y coordinate.

https://github.com/Jeff-sjtu/HybrIK/issues/118

opened by nnop 0