Understanding Convolution for Semantic Segmentation

Last update: Dec 31, 2022

Overview

TuSimple-DUC

by Panqu Wang, Pengfei Chen, Ye Yuan, Ding Liu, Zehua Huang, Xiaodi Hou, and Garrison Cottrell.

Introduction

This repository is for Understanding Convolution for Semantic Segmentation (WACV 2018), which achieved state-of-the-art result on the CityScapes, PASCAL VOC 2012, and Kitti Road benchmark.

Requirement

We tested our code on:

Ubuntu 16.04, Python 2.7 with

MXNet (0.11.0), numpy(1.13.1), cv2(3.2.0), PIL(4.2.1), and cython(0.25.2)

Usage

Clone the repository:

git clone [email protected]:TuSimple/TuSimple-DUC.git
python setup.py develop --user

Download the pretrained model from Google Drive.

Build MXNet (only tested on the TuSimple version):

git clone --recursive [email protected]:TuSimple/mxnet.git
vim make/config.mk (we should have USE_CUDA = 1, modify USE_CUDA_PATH, and have USE_CUDNN = 1 to enable GPU usage.)
make -j
cd python
python setup.py develop --user

For more MXNet tutorials, please refer to the official documentation.

Training:
```
cd train
python train_model.py ../configs/train/train_cityscapes.cfg
```
The paths/dirs in the .cfg file need to be specified by the user.

Testing

cd test
python predict_full_image.py ../configs/test/test_full_image.cfg

The paths/dirs in the .cfg file need to be specified by the user.

Results:

Modify the result_dir path in the config file to save the label map and visualizations. The expected scores are:

(single scale testing denotes as 'ss' and multiple scale testing denotes as 'ms')
- ResNet101-DUC-HDC on CityScapes testset (mIoU): 79.1(ss) / 80.1(ms)
- ResNet152-DUC on VOC2012 (mIoU): 83.1(ss)

Citation

If you find the repository is useful for your research, please consider citing:

@article{wang2017understanding,
  title={Understanding convolution for semantic segmentation},
  author={Wang, Panqu and Chen, Pengfei and Yuan, Ye and Liu, Ding and Huang, Zehua and Hou, Xiaodi and Cottrell, Garrison},
  journal={arXiv preprint arXiv:1702.08502},
  year={2017}
}

Questions

Please contact [email protected] or [email protected] .

Comments

Training data list for Cityscapes

Hi,

Thanks for the effort for making the training code publicly available. I would like to try training the cityscapes model from scratch by your code. I saw on the paper that you used some data augmentation trick to enlarge the number of training images. Could you provide the augmented data you used and some instruction for generating the data list? Thank you very much!

opened by chienyiwang 12
Train from "ImageNet pretrained weights"

Hi,

I would like to train the network from ImageNet pretrained weights using your network structure and configurations. However, I didn't see the pretrained weights in your provided model folder. Is it possible to provide the ImageNet pretrained weight you used for training from scratch? Thank you so much!

opened by chienyiwang 10
how to set the val.lst

Hi,_ when i use predict_full_image.py , I make my val.lst like this:

/home/xxx/MXnetProjects/TuSimple-DUC/Data/cityscapes/gtFine/test/berlin/berlin_000000_000019_leftImg8bit.png_` But these is always a error, i debug it and found that this is a error when use im = cv.imread(img_path)[:, :, ::-1] in predict_single, for i set the vla.lst in wrong format. can you help me. THANKS!

opened by ZzzjzzZ 4
Where is the ImageNet pretrained classification models?

In the link to the models (https://drive.google.com/drive/folders/0B72xLTlRb0SoREhISlhibFZTRmM), I can only find the segmentation pretrained models. Would you please tell me where can I find the classification model pretrained on ImageNet?

opened by yzou2 4
Failed to load the BatchNorm Block in ResNet_DUC_HDC_CityScapes-symbol.json

Hi, I'm very interested in your work. But when I tried to repeat your experiment, a MXNetError appeared that Cannot find argument 'fix_linear_trans' when loading the ResNet_DUC_HDC_CityScapes-symbol.json. My current environment is Python 2.7, Mxnet version 1.0.0 under Win10. Is this error because of the operating system or the different Mxnet version ?

opened by Serian1992 3
ImageNet Pretrained ResNet-101 Model

I tried to convert the deeplab model in #1, but failed due to error Message type "caffe.ImageDataParameter" has no field named "label_type" So would you please share the ImageNet pretrained ResNet-101 Model please? Thanks

opened by ArsenLuca 3

simple_bind error during Cityscapes testing

Hi,

I'm trying to run the Cityscapes classifier in test mode on an Amazon EC2 instance, but I'm running into this error:

$ python predict_full_image.py ../configs/test/test_full_image.cfg
[06:22:55] src/nnvm/legacy_json_util.cc:190: Loading symbol saved by previous version v0.8.0. Attempting to upgrade...
[06:22:55] src/nnvm/legacy_json_util.cc:198: Symbol successfully upgraded!
/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/module/base_module.py:53: UserWarning: You created Module with Module(..., label_names=['softmax_label']) but input with name 'softmax_label' is not found in symbol.list_arguments(). Did you mean one of:
        data
        seg_loss_label
  warnings.warn(msg)
/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/module/base_module.py:65: UserWarning: Data provided by label_shapes don't match names specified by label_names ([] vs. ['softmax_label'])
  warnings.warn(msg)
[06:22:55] /home/ubuntu/TuSimple-DUC/mxnet/dmlc-core/include/dmlc/./logging.h:308: [06:22:55] src/storage/storage.cc:113: Compile with USE_CUDA=1 to enable GPU usage

Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(+0x12a7b3a) [0x7f95e0047b3a]
[bt] (1) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet11StorageImpl5AllocEmNS_7ContextE+0x57) [0x7f95e0048317]
[bt] (2) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(+0x131869f) [0x7f95e00b869f]
[bt] (3) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec15ReshapeOrCreateERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4nnvm6TShapeEiRKNS_7ContextEPSt13unordered_mapIS6_NS_7NDArrayESt4hashIS6_ESt8equal_toIS6_ESaISt4pairIS7_SH_EEE+0xa4f) [0x7f95e00be58f]
[bt] (4) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor13InitArgumentsERKN4nnvm12IndexedGraphERKSt6vectorINS2_6TShapeESaIS7_EERKS6_IiSaIiEERKS6_INS_7ContextESaISG_EESK_SK_RKS6_INS_9OpReqTypeESaISL_EERKSt13unordered_setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4hashISW_ESt8equal_toISW_ESaISW_EEPKNS_8ExecutorEPSt13unordered_mapISW_NS_7NDArrayESY_S10_SaISt4pairIKSW_S19_EEEPS6_IS19_SaIS19_EES1I_S1I_+0xaa4) [0x7f95e00c1f24]
[bt] (5) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorIS4_SaIS4_EESR_SR_RKSt13unordered_mapISD_NS2_6TShapeESt4hashISD_ESt8equal_toISD_ESaISG_ISH_ST_EEERKSS_ISD_iSV_SX_SaISG_ISH_iEEERKSN_INS_9OpReqTypeESaIS18_EERKSt13unordered_setISD_SV_SX_SaISD_EEPSN_INS_7NDArrayESaIS1I_EES1L_S1L_PSS_ISD_S1I_SV_SX_SaISG_ISH_S1I_EEEPNS_8ExecutorERKSS_INS2_9NodeEntryES1I_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISG_IKS1S_S1I_EEE+0x88b) [0x7f95e00c9edb]
[bt] (6) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet8Executor10SimpleBindEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_St4lessISC_ESaISt4pairIKSC_S3_EEERKSt6vectorIS3_SaIS3_EESQ_SQ_RKSt13unordered_mapISC_NS1_6TShapeESt4hashISC_ESt8equal_toISC_ESaISF_ISG_SS_EEERKSR_ISC_iSU_SW_SaISF_ISG_iEEERKSM_INS_9OpReqTypeESaIS17_EERKSt13unordered_setISC_SU_SW_SaISC_EEPSM_INS_7NDArrayESaIS1H_EES1K_S1K_PSR_ISC_S1H_SU_SW_SaISF_ISG_S1H_EEEPS0_+0x233) [0x7f95e00ca583]
[bt] (7) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(MXExecutorSimpleBind+0x2a67) [0x7f95e00888c7]
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f9603afae40]
[bt] (9) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f9603afa8ab]

Traceback (most recent call last):
  File "predict_full_image.py", line 100, in <module>
    tester = ImageListTester(config)
  File "predict_full_image.py", line 33, in __init__
    self.tester = Tester(self.config)
  File "/home/ubuntu/TuSimple-DUC/tusimple_duc/test/tester.py", line 42, in __init__
    predictor.bind(data_shapes=[('data', data_shape)], for_training=False)
  File "/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/module/module.py", line 417, in bind
    state_names=self._state_names)
  File "/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/module/executor_group.py", line 231, in __init__
    self.bind_exec(data_shapes, label_shapes, shared_group)
  File "/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/module/executor_group.py", line 327, in bind_exec
    shared_group))
  File "/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/module/executor_group.py", line 603, in _bind_ith_exec
    shared_buffer=shared_data_arrays, **input_shapes)
  File "/home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/symbol.py", line 1479, in simple_bind
    raise RuntimeError(error_msg)
RuntimeError: simple_bind error. Arguments:
data: (1, 3, 1024, 2048)
[06:22:55] src/storage/storage.cc:113: Compile with USE_CUDA=1 to enable GPU usage

Stack trace returned 10 entries:
[bt] (0) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(+0x12a7b3a) [0x7f95e0047b3a]
[bt] (1) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet11StorageImpl5AllocEmNS_7ContextE+0x57) [0x7f95e0048317]
[bt] (2) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(+0x131869f) [0x7f95e00b869f]
[bt] (3) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec15ReshapeOrCreateERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4nnvm6TShapeEiRKNS_7ContextEPSt13unordered_mapIS6_NS_7NDArrayESt4hashIS6_ESt8equal_toIS6_ESaISt4pairIS7_SH_EEE+0xa4f) [0x7f95e00be58f]
[bt] (4) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor13InitArgumentsERKN4nnvm12IndexedGraphERKSt6vectorINS2_6TShapeESaIS7_EERKS6_IiSaIiEERKS6_INS_7ContextESaISG_EESK_SK_RKS6_INS_9OpReqTypeESaISL_EERKSt13unordered_setINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEESt4hashISW_ESt8equal_toISW_ESaISW_EEPKNS_8ExecutorEPSt13unordered_mapISW_NS_7NDArrayESY_S10_SaISt4pairIKSW_S19_EEEPS6_IS19_SaIS19_EES1I_S1I_+0xaa4) [0x7f95e00c1f24]
[bt] (5) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet4exec13GraphExecutor4InitEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES4_St4lessISD_ESaISt4pairIKSD_S4_EEERKSt6vectorIS4_SaIS4_EESR_SR_RKSt13unordered_mapISD_NS2_6TShapeESt4hashISD_ESt8equal_toISD_ESaISG_ISH_ST_EEERKSS_ISD_iSV_SX_SaISG_ISH_iEEERKSN_INS_9OpReqTypeESaIS18_EERKSt13unordered_setISD_SV_SX_SaISD_EEPSN_INS_7NDArrayESaIS1I_EES1L_S1L_PSS_ISD_S1I_SV_SX_SaISG_ISH_S1I_EEEPNS_8ExecutorERKSS_INS2_9NodeEntryES1I_NS2_13NodeEntryHashENS2_14NodeEntryEqualESaISG_IKS1S_S1I_EEE+0x88b) [0x7f95e00c9edb]
[bt] (6) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(_ZN5mxnet8Executor10SimpleBindEN4nnvm6SymbolERKNS_7ContextERKSt3mapINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES3_St4lessISC_ESaISt4pairIKSC_S3_EEERKSt6vectorIS3_SaIS3_EESQ_SQ_RKSt13unordered_mapISC_NS1_6TShapeESt4hashISC_ESt8equal_toISC_ESaISF_ISG_SS_EEERKSR_ISC_iSU_SW_SaISF_ISG_iEEERKSM_INS_9OpReqTypeESaIS17_EERKSt13unordered_setISC_SU_SW_SaISC_EEPSM_INS_7NDArrayESaIS1H_EES1K_S1K_PSR_ISC_S1H_SU_SW_SaISF_ISG_S1H_EEEPS0_+0x233) [0x7f95e00ca583]
[bt] (7) /home/ubuntu/TuSimple-DUC/mxnet/python/mxnet/../../lib/libmxnet.so(MXExecutorSimpleBind+0x2a67) [0x7f95e00888c7]
[bt] (8) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call_unix64+0x4c) [0x7f9603afae40]
[bt] (9) /usr/lib/x86_64-linux-gnu/libffi.so.6(ffi_call+0x2eb) [0x7f9603afa8ab]

Any idea what is causing it and how to fix it?

System: Ubuntu 16.04.3 mxnet 0.11.0 numpy 1.13.3 cv2 3.2.0 PIL 1.1.7 cython 0.27.1

Thanks!

opened by atward424 3

Data prep script

I am trying to understand your pipeline so that I can run it on my own data. I see a script in data_prep/get_cityscapes_list.py but I cannot find from your readme how to run it. Can you please add a description of the expected format and/or document the data prep pipeline? Thank you

opened by DSLituiev 2
What is the `test_scales` in multiple scale testing？

I would like to use your open model to reproduce this evaluation:

ResNet101-DUC-HDC on CityScapes testset (mIoU): 80.1(multiple scale testing )

But I do not know what test_scales does you use in test_full_image.cfg, could you tell me? Thank you very much.

opened by acgtyrant 2
How to apply CRFs?

Hi, I feel confused to CRFs. I'm sorry that i can't find the way you apply the CRFs to your ResNet-DUC. Could you please tell me how to do it? Thank you very much!!!!

opened by reborm 2

nice to meet you,can you help me ?there is an error in you code,

when I run your code of train_model , there is an error : config_path = sys.argv[1] => list index out of range , what I mistake?
                           thank you!

opened by zhouyuan888888 1

troubles met in the procedure of testing

Hello,first of allThanks for all the effort you have made for the segmentation. During the testing procedure, I have met some problems. When I build the repository, after executing make command , the terminal gives the tips 'can not find the '....../ps-lite/make/ps.mk'.Is ther any way to solve this issues?thanks for all

opened by bowenroom 0
Evaluate PASCAL VOC 2012 test set

Hi! How do you evaluate the VOC 2012 test set? How did you test the segmentation results of VOC2012 test data set on PASCAL VOC official website? Looking forward to your reply. Thanks!

opened by InstantWindy 0
A question about DUC

If the orginal image is (600,800), After downsample 16X, the feature map is (37,50), How to use DUC to reproduce orginal size? upsample 16X is (592,800)

opened by WangTianYuan 2
hi，I try your DUC method as decoder on my network(similiar to mobile net for high performance),but there are many grid on the predicted alpha

hi，I try your DUC method as decoder on my network(similiar to mobile net for high performance),but there are many grid on the predicted alpha. the loss seems normal. could you please figure it out?

opened by leonardodora 1
Hello,I am interesting in your work,but I have a trouble

I have read your papers(Understanding Convolution for semantic segmentation ),and run your net.Then I make an improvement.But When submit my results to pascal voc server,I always get an evaluation error. I use the command: tar -zcvf results.tgz results. My directory structure ：/results/VOC2012/Segmentation/comp6_test_cls/ The pixel of every image is similar [0,64,128]

Could you give me some point?

opened by mshmoon 0
Reproduce failed.
I write resnet.symbol by myself and use it as pretrained model to train TuSimple-DUC, and keep cfg consistent with the paper except cropped size. See my fork.

However, train_IoU is only 0.563 while I can not validate temporarily #21 , I think the val_IoU should not be too far away:

01-03 02:23:28 Epoch[19] Train-acc_ignore=0.932954 01-03 02:23:28 Epoch[19] Train-IoU=0.563348 01-03 02:23:28 Epoch[19] Train-SoftmaxLoss=0.212817 01-03 02:23:28 Epoch[19] Time cost=10693.555 01-03 02:23:29 Saved checkpoint to "../models/ResNet_DUC_HDC_CityScapes/2017_12_31_15:19:05/ResNet_DUC_HDC_CityScapes-0020.params"

Anyone reproduce successfully?
opened by acgtyrant 2

Owner

TuSimple

The Future of Trucking

GitHub https://arxiv.org/abs/1702.08502

Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

337 Dec 15, 2022

Implementation of CVPR'21: RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction

RfD-Net [Project Page] [Paper] [Video] RfD-Net: Point Scene Understanding by Semantic Instance Reconstruction Yinyu Nie, Ji Hou, Xiaoguang Han, Matthi

162 Jan 6, 2023

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP Abstract: We introduce a method that allows to automatically se

134 Dec 19, 2022

TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

147 Dec 3, 2022

Mae segmentation - Reproduction of semantic segmentation using masked autoencoder (mae)

ADE20k Semantic segmentation with MAE Getting started Install the mmsegmentation

97 Dec 17, 2022

PyTorch implementation of "Conformer: Convolution-augmented Transformer for Speech Recognition" (INTERSPEECH 2020)

PyTorch implementation of Conformer: Convolution-augmented Transformer for Speech Recognition. Transformer models are good at capturing content-based

565 Jan 4, 2023

Official pytorch implementation of paper "Inception Convolution with Efficient Dilation Search" (CVPR 2021 Oral).

IC-Conv This repository is an official implementation of the paper Inception Convolution with Efficient Dilation Search. Getting Started Download Imag

111 Dec 31, 2022

[CVPR 2021] Involution: Inverting the Inherence of Convolution for Visual Recognition, a brand new neural operator

involution Official implementation of a neural operator as described in Involution: Inverting the Inherence of Convolution for Visual Recognition (CVP

1.3k Dec 28, 2022

Code for Mesh Convolution Using a Learned Kernel Basis

Mesh Convolution This repository contains the implementation (in PyTorch) of the paper FULLY CONVOLUTIONAL MESH AUTOENCODER USING EFFICIENT SPATIALLY

35 Jan 3, 2023

Diverse Branch Block: Building a Convolution as an Inception-like Unit

Diverse Branch Block: Building a Convolution as an Inception-like Unit (PyTorch) (CVPR-2021) DBB is a powerful ConvNet building block to replace regul

253 Dec 24, 2022

(CVPR 2021) PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds by Mutian Xu*, Runyu Ding*, Hengshuang Zhao, and Xiaojuan Qi. Int

228 Dec 25, 2022

Implementation of the "PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences" paper.

PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences Introduction Point cloud sequences are irregular and unordered in the spatial dimen

63 Dec 9, 2022

PyTorch reimplementation of the paper Involution: Inverting the Inherence of Convolution for Visual Recognition [CVPR 2021].

Involution: Inverting the Inherence of Convolution for Visual Recognition Unofficial PyTorch reimplementation of the paper Involution: Inverting the I

100 Dec 1, 2022