Hi! Thanks for this repo. I am unable to install the inplace_abn library that is required to train your models. I have tried many ways to install and debug but it still shows the CUDA error. error: command '/usr/bin/nvcc' failed with exit status 1. Is it possible to train the model without using this library by making a few changes in the code?

Hello, I wonder why your Resnet architecture mismatches standard Resnet architectures? It does not allow to use ImageNet pre-trained models.

Yours has two extra convolution layers; and there is no convolution layer with kernels 7x7.

Thanks!

Standard Resnet arch.:

https://github.com/speedinghzl/Pytorch-Deeplab https://github.com/kazuto1011/deeplab-pytorch https://github.com/isht7/pytorch-deeplab-resnet https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py

class ResNet(nn.Module):

    def __init__(self, block, layers, num_classes=1000, zero_init_residual=False):
        super(ResNet, self).__init__()
        self.inplanes = 64
        self.conv1 = nn.Conv2d(3, 64, kernel_size=7, stride=2, padding=3,
                               bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        self.relu = nn.ReLU(inplace=True)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
        self.layer4 = self._make_layer(block, 512, layers[3], stride=2)

...

Your Resnet arch.:

class ResNet(nn.Module):
    def __init__(self, block, layers, num_classes):
        self.inplanes = 128
        super(ResNet, self).__init__()
        self.conv1 = conv3x3(3, 64, stride=2)
        self.bn1 = BatchNorm2d(64)
        self.relu1 = nn.ReLU(inplace=False)
        self.conv2 = conv3x3(64, 64)
        self.bn2 = BatchNorm2d(64)
        self.relu2 = nn.ReLU(inplace=False)
        self.conv3 = conv3x3(64, 128)
        self.bn3 = BatchNorm2d(128)
        self.relu3 = nn.ReLU(inplace=False)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1)

        self.relu = nn.ReLU(inplace=False)
        self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2, padding=1, ceil_mode=True) # change
        self.layer1 = self._make_layer(block, 64, layers[0])
        self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
        self.layer3 = self._make_layer(block, 256, layers[2], stride=1, dilation=2)
        self.layer4 = self._make_layer(block, 512, layers[3], stride=1, dilation=4, multi_grid=(1,1,1))

....

Hi, @speedinghzl . Thanks for the great job. I have a little problem about the activation using in the model. As in the inplaceABN paper, the ReLU activation is not invertible. Do you use the ReLU activation to train the model?

I was trying to run the model without any modifications on google cloud with V100 GPU. I got following error:

RuntimeError: CUDA Error encountered in <function CompiledLib.bn_mean_var_cuda at 0x7fd263866c80> Traceback (most recent call last): File "evaluate.py", line 253, in main() File "evaluate.py", line 198, in main saved_state_dict = torch.load(args.restore_from) File "/opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 356, in load f = open(f, 'rb')

Is it related to training on V100 GPU. Currently I have only one GPU attached.

Hi, Thanks for sharing your code. I am trying to train using CUDA 9.2 and pytorch 0.4.1 on a NVIDIA Tesla v100. I modified the build.sh file including the architecture for the v100, and built the custom module. When I try to run the main training script, I receive this output:

ImportError: /home/fabio/pytorch-segmentation-toolbox/libs/_ext/__ext.so: undefined symbol: __cudaPopCallConfiguration

Do you have any idea on how to solve it?

I tried the code, but the program stopped when the tensor x get into dsn. I locate the Error with GDB and found ：

Program received signal SIGSEGV, Segmentation fault. 0x00002aaafb067270 in unsigned long thrust::system::cuda::detail::bulk_::detail::block_size_with_maximum_potential_occupancy<thrust::system::cuda::detail::bulk_::detail::cuda_launch_config_detail::util::zero_function >(thrust::system::cuda::detail::bulk_::detail::function_attributes_t const&, thrust::system::cuda::detail::bulk_::detail::device_properties_t const&, thrust::system::cuda::detail::bulk_::detail::cuda_launch_config_detail::util::zero_function) [clone .isra.24] () from /home/my/code/pytorch-segmentation-toolbox-master/libs/_ext/__ext.so

I don't understand the error in .so file, how can I fix this problem? thanks !

File "train.py", line 256, in <module> main() File "train.py", line 220, in main preds = model(images) File "/home/dingcheng/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "/home/dingcheng/.local/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 121, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/dingcheng/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "/home/dingcheng/seg/pytorch-segmentation-toolbox-master/networks/pspnet.py", line 139, in forward x = self.relu1(self.bn1(self.conv1(x))) File "/home/dingcheng/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__ result = self.forward(*input, **kwargs) File "/home/dingcheng/seg/pytorch-segmentation-toolbox-master/libs/bn.py", line 184, in forward self.activation, self.slope) File "/home/dingcheng/seg/pytorch-segmentation-toolbox-master/libs/functions.py", line 183, in forward _check(_ext.bn_mean_var_cuda, x, mean, var) File "/home/dingcheng/seg/pytorch-segmentation-toolbox-master/libs/functions.py", line 16, in _check raise RuntimeError("CUDA Error encountered in {}".format(fn)) RuntimeError: CUDA Error encountered in <function CompiledLib.bn_mean_var_cuda at 0x7f46e5355730> Hello, I encountered the above error when running the code. I hope you can help me to answer the following

hi,thanks for you nice work! I want to ask same question: In your implementation,what is your performance in cityscapes validation set and in what condition? I meet some confusion because i use your project only get 77.51 which batchsize is 12 in 4GPU and 77.40 which batchsize is 24 in 8GPU in val.but in the paper the mIoU can reach 78.5%. have you reach this result? thanks!

Thank you for your amazing repo!

However, as mentioned in the deeplabv3 paper, the training output stride is 16 and for inference, the os = 8. Will you support this in future? (Actually I'm not sure how they do this).

Due to GPU limitation,I set the BS=2,STEPS= 160000 and GPU_IDS= 0,(also Titan XP),But I'm confused that the my reimplement results is not so well,mIOU is only 0.687...could you double check are there some details go wrong?As I found the args.ohem was set to false,the set_bn_eval and set_bn_momentum function seems not working...As for evaluate,It seems that args.recurrence working in any way...etc..I'm not sure if these will influence the reimplement results or not,could you give me some advice?

Hi, @speedinghzl

I got RuntimeError: invalid argument 3: divide by zero for running_var.mul_((1 - ctx.momentum)).add_(ctx.momentum * var * n / (n - 1)) in functions.py", line 209, in forward

2975 images are loaded!
Traceback (most recent call last):
  File "train.py", line 251, in <module>
    main()
  File "train.py", line 215, in main
    preds = model(images)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 112, in forward
    return self.module(*inputs[0], **kwargs[0])
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/code8/pytorch-segmentation-toolbox/networks/pspnet.py", line 148, in forward
    x = self.head(x)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/code8/pytorch-segmentation-toolbox/networks/pspnet.py", line 85, in forward
    priors = [F.upsample(input=stage(feats), size=(h, w), mode='bilinear', align_corners=True) for stage in self.stages] + [feats]
  File "/data/code8/pytorch-segmentation-toolbox/networks/pspnet.py", line 85, in <listcomp>
    priors = [F.upsample(input=stage(feats), size=(h, w), mode='bilinear', align_corners=True) for stage in self.stages] + [feats]
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/container.py", line 91, in forward
    input = module(input)
  File "/root/anaconda3/envs/tf1.3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/data/code8/pytorch-segmentation-toolbox/libs/bn.py", line 184, in forward
    self.activation, self.slope)
  File "/data/code8/pytorch-segmentation-toolbox/libs/functions.py", line 209, in forward
    running_var.mul_((1 - ctx.momentum)).add_(ctx.momentum * var * n / (n - 1))
RuntimeError: invalid argument 3: divide by zero at /pytorch/aten/src/THC/generic/THCTensorMathPairwise.cu:88
(tf1.3) root@milton-ThinkCentre-M93p:/data/code8/pytorch-segmentation-toolbox#

Any suggestion to fix it?

THX!

hi, if I want to use 3090 with CUDA 11， how to change the inplace_abn? some parts of the codes are based on CUDA 9, where to change the code of Inplace-ABN? your original implementation:

Install Pytorch-1.1

$ conda install pytorch torchvision cudatoolkit=9.0 -c pytorch

Install Inplace-ABN

$ git clone https://github.com/mapillary/inplace_abn.git $ cd inplace_abn $ python setup.py install

这里多尺度预测说对于大于原图的采用裁剪的方式，请问是否也采用的滑动窗口预测呢？但是代码里没有采用滑动窗口的代码啊 def predict_multiscale(net, image, tile_size, scales, classes, flip_evaluation, recurrence): """ Predict an image by looking at it with different scales. We choose the "predict_whole_img" for the image with less than the original input size, for the input of larger size, we would choose the cropping method to ensure that GPU memory is enough. """ image = image.data N_, C_, H_, W_ = image.shape full_probs = np.zeros((H_, W_, classes))
for scale in scales: scale = float(scale) print("Predicting image scaled by %f" % scale) scale_image = ndimage.zoom(image, (1.0, 1.0, scale, scale), order=1, prefilter=False) scaled_probs = predict_whole(net, scale_image, tile_size, recurrence) if flip_evaluation == True: flip_scaled_probs = predict_whole(net, scale_image[:,:,:,::-1].copy(), tile_size, recurrence) scaled_probs = 0.5 * (scaled_probs + flip_scaled_probs[:,::-1,:]) full_probs += scaled_probs full_probs /= len(scales) return full_probs

evaluate.py in branch pytorch-1.1 line 134 and line 135 you only use the prediction result of the first image, but treat it as the result of the remaining images in the same batch, in line 136. so something goes wrong when evaluate with batch size larger than 1. you should modify it with prediction = padded_prediction[:, 0:img.shape[2], 0:img.shape[3], :] and count_predictions[:, y1:y2, x1:x2] += 1

@speedinghzl hi when i try to extract or viz the cityscapes datasets its like this , how can i extract teh polygon from the semanttic segmentation map

This PR fixes the predict_sliding function of evaluate.py not giving correct values.

Currently instead of adding the full batch of obtained probabilities, it only adds the ones from the first image. This does not throw and error since numpy casts the dimensions on addition, resulting on the predictions for the first image being stored in place of the rest. This results in a very low MIoU whenever the script is run with batch_size > world_size