Segmentation models with pretrained backbones. PyTorch.

i get the error below when i try to use my unet with se_resnext50 models trained weight file for ensembling :

RuntimeError: Error(s) in loading state_dict for Unet: Missing key(s) in state_dict: "decoder.blocks.0.conv1.0.weight", "decoder.blocks.0.conv1.1.weight", "decoder.blocks.0.conv1.1.bias", "decoder.blocks.0.conv1.1.running_mean", "decoder.blocks.0.conv1.1.running_var", "decoder.blocks.0.conv2.0.weight", "decoder.blocks.0.conv2.1.weight", "decoder.blocks.0.conv2.1.bias", "decoder.blocks.0.conv2.1.running_mean", "decoder.blocks.0.conv2.1.running_var", "decoder.blocks.1.conv1.0.weight", "decoder.blocks.1.conv1.1.weight", "decoder.blocks.1.conv1.1.bias", "decoder.blocks.1.conv1.1.running_mean", "decoder.blocks.1.conv1.1.running_var", "decoder.blocks.1.conv2.0.weight", "decoder.blocks.1.conv2.1.weight", "decoder.blocks.1.conv2.1.bias", "decoder.blocks.1.conv2.1.running_mean", "decoder.blocks.1.conv2.1.running_var", "decoder.blocks.2.conv1.0.weight", "decoder.blocks.2.conv1.1.weight", "decoder.blocks.2.conv1.1.bias", "decoder.blocks.2.conv1.1.running_mean", "decoder.blocks.2.conv1.1.running_var", "decoder.blocks.2.conv2.0.weight", "decoder.blocks.2.conv2.1.weight", "decoder.blocks.2.conv2.1.bias", "decoder.blocks.2.conv2.1.running_mean", "decoder.blocks.2.conv2.1.running_var", "decoder.blocks.3.conv1.0.weight", "decoder.blocks.3.conv1.1.weight", "decoder.blocks.3.conv1.1.bias", "decoder.blocks.3.conv1.1.running_mean", "decoder.blocks.3.conv1.1.running_var", "decoder.blocks.3.conv2.0.weight", "decoder.blocks.3.conv2.1.weight", "decoder.blocks.3.conv2.1.bias", "decoder.blocks.3.conv2.1.running_mean", "decoder.blocks.3.conv2.1.running_var", "decoder.blocks.4.conv1.0.weight", "decoder.blocks.4.conv1.1.weight", "decoder.blocks.4.conv1.1.bias", "decoder.blocks.4.conv1.1.running_mean", "decoder.blocks.4.conv1.1.running_var", "decoder.blocks.4.conv2.0.weight", "decoder.blocks.4.conv2.1.weight", "decoder.blocks.4.conv2.1.bias", "decoder.blocks.4.conv2.1.running_mean", "decoder.blocks.4.conv2.1.running_var", "segmentation_head.0.weight", "segmentation_head.0.bias". Unexpected key(s) in state_dict: "decoder.layer1.block.0.block.0.weight", "decoder.layer1.block.0.block.1.weight", "decoder.layer1.block.0.block.1.bias", "decoder.layer1.block.0.block.1.running_mean", "decoder.layer1.block.0.block.1.running_var", "decoder.layer1.block.0.block.1.num_batches_tracked", "decoder.layer1.block.1.block.0.weight", "decoder.layer1.block.1.block.1.weight", "decoder.layer1.block.1.block.1.bias", "decoder.layer1.block.1.block.1.running_mean", "decoder.layer1.block.1.block.1.running_var", "decoder.layer1.block.1.block.1.num_batches_tracked", "decoder.layer2.block.0.block.0.weight", "decoder.layer2.block.0.block.1.weight", "decoder.layer2.block.0.block.1.bias", "decoder.layer2.block.0.block.1.running_mean", "decoder.layer2.block.0.block.1.running_var", "decoder.layer2.block.0.block.1.num_batches_tracked", "decoder.layer2.block.1.block.0.weight", "decoder.layer2.block.1.block.1.weight", "decoder.layer2.block.1.block.1.bias", "decoder.layer2.block.1.block.1.running_mean", "decoder.layer2.block.1.block.1.running_var", "decoder.layer2.block.1.block.1.num_batches_tracked", "decoder.layer3.block.0.block.0.weight", "decoder.layer3.block.0.block.1.weight", "decoder.layer3.block.0.block.1.bias", "decoder.layer3.block.0.block.1.running_mean", "decoder.layer3.block.0.block.1.running_var", "decoder.layer3.block.0.block.1.num_batches_tracked", "decoder.layer3.block.1.block.0.weight", "decoder.layer3.block.1.block.1.weight", "decoder.layer3.block.1.block.1.bias", "decoder.layer3.block.1.block.1.running_mean", "decoder.layer3.block.1.block.1.running_var", "decoder.layer3.block.1.block.1.num_batches_tracked", "decoder.layer4.block.0.block.0.weight", "decoder.layer4.block.0.block.1.weight", "decoder.layer4.block.0.block.1.bias", "decoder.layer4.block.0.block.1.running_mean", "decoder.layer4.block.0.block.1.running_var", "decoder.layer4.block.0.block.1.num_batches_tracked", "decoder.layer4.block.1.block.0.weight", "decoder.layer4.block.1.block.1.weight", "decoder.layer4.block.1.block.1.bias", "decoder.layer4.block.1.block.1.running_mean", "decoder.layer4.block.1.block.1.running_var", "decoder.layer4.block.1.block.1.num_batches_tracked", "decoder.layer5.block.0.block.0.weight", "decoder.layer5.block.0.block.1.weight", "decoder.layer5.block.0.block.1.bias", "decoder.layer5.block.0.block.1.running_mean", "decoder.layer5.block.0.block.1.running_var", "decoder.layer5.block.0.block.1.num_batches_tracked", "decoder.layer5.block.1.block.0.weight", "decoder.layer5.block.1.block.1.weight", "decoder.layer5.block.1.block.1.bias", "decoder.layer5.block.1.block.1.running_mean", "decoder.layer5.block.1.block.1.running_var", "decoder.layer5.block.1.block.1.num_batches_tracked", "decoder.final_conv.weight", "decoder.final_conv.bias".

Encountering this from today: 19-Jun-2021

ImportError Traceback (most recent call last) in () 4 get_ipython().system('pip install -U segmentation-models-pytorch') 5 ----> 6 import segmentation_models_pytorch as smp 7 8

11 frames /usr/local/lib/python3.7/dist-packages/timm/models/layers/helpers.py in () 4 """ 5 from itertools import repeat ----> 6 from torch._six import container_abcs 7 8

ImportError: cannot import name 'container_abcs' from 'torch._six' (/usr/local/lib/python3.7/dist-packages/torch/_six.py)

I saw in the documentation that the metrics for multilabel prediction require (batch, num_class, height, width). But then, I have a multi-class mask of one channel as target where each pixel are labeled by the class.

How do I use this for that scenario?

Also, this seems to be computing it per batch. How do I do it per epoch?

import segmentation_models_pytorch as smp

# lets assume we have multilabel prediction for 3 classes
output = torch.rand([10, 3, 256, 256])
target = torch.rand([10, 3, 256, 256]).round().long()

# first compute statistics for true positives, false positives, false negative and
# true negative "pixels"
tp, fp, fn, tn = smp.metrics.get_stats(output, target, mode='multilabel', threshold=0.5)

# then compute metrics with required reduction (see metric docs)
iou_score = smp.metrics.iou_score(tp, fp, fn, tn, reduction="micro")
f1_score = smp.metrics.f1_score(tp, fp, fn, tn, reduction="micro")
f2_score = smp.metrics.fbeta_score(tp, fp, fn, tn, beta=2, reduction="micro")
accuracy = smp.metrics.accuracy(tp, fp, fn, tn, reduction="macro")
recall = smp.metrics.recall(tp, fp, fn, tn, reduction="micro-imagewise")

Hello,

I added support for Apple's MobileOne encoder.

Paper: Link

There were very few changes I had to make to their official github repo: Link

It works with all decoders and has impressive inference time for images with 256x256:

| Encoder-Decoder| Inference time in vanilla torch| | ------------- | ------------- | | mobileone_s1_pspnet_256 | 0.0313718318939209 | mobileone_s0_pan_256 | 0.03421592712402344 mobileone_s2_pspnet_256 | 0.036206960678100586 mobileone_s3_pspnet_256 | 0.04711484909057617 mobileone_s1_pan_256 | 0.05329489707946777 mobileone_s0_linknet_256 | 0.05789995193481445 mobileone_s0_deeplabv3plus_256 | 0.058853864669799805 mobileone_s0_fpn_256 | 0.07664108276367188 mobileone_s4_pspnet_256 | 0.0768282413482666 mobileone_s1_deeplabv3plus_256 | 0.07886672019958496 mobileone_s2_pan_256 | 0.07946181297302246 mobileone_s3_pan_256 | 0.09101414680480957 mobileone_s1_fpn_256 | 0.09615683555603027 mobileone_s1_linknet_256 | 0.09956574440002441 mobileone_s2_fpn_256 | 0.11291790008544922 mobileone_s0_unet_256 | 0.11676502227783203 mobileone_s2_linknet_256 | 0.12518310546875 mobileone_s3_deeplabv3plus_256 | 0.12642478942871094 mobileone_s2_deeplabv3plus_256 | 0.1289658546447754 mobileone_s3_fpn_256 | 0.1370537281036377 mobileone_s4_pan_256 | 0.14015984535217285 mobileone_s1_unet_256 | 0.15249204635620117 mobileone_s3_linknet_256 | 0.15824413299560547 mobileone_s4_deeplabv3plus_256 | 0.16476082801818848 mobileone_s0_manet_256 | 0.17203474044799805 mobileone_s2_unet_256 | 0.17334604263305664 mobileone_s4_fpn_256 | 0.182358980178833 mobileone_s3_unet_256 | 0.20330286026000977 mobileone_s4_linknet_256 | 0.21462082862854004 mobileone_s0_deeplabv3_256 | 0.22992897033691406 mobileone_s4_unet_256 | 0.24337363243103027 mobileone_s0_unetplusplus_256 | 0.29451799392700195 mobileone_s1_deeplabv3_256 | 0.31217503547668457 mobileone_s1_manet_256 | 0.3140380382537842 mobileone_s1_unetplusplus_256 | 0.5090749263763428 mobileone_s2_deeplabv3_256 | 0.5372707843780518 mobileone_s3_deeplabv3_256 | 0.5489542484283447 mobileone_s2_unetplusplus_256 | 0.5728631019592285 mobileone_s4_deeplabv3_256 | 0.638185977935791 mobileone_s2_manet_256 | 0.6446411609649658 mobileone_s3_manet_256 | 0.6838269233703613 mobileone_s3_unetplusplus_256 | 0.6991360187530518 mobileone_s4_manet_256 | 0.748121976852417 mobileone_s4_unetplusplus_256 | 0.9898359775543213

Hi, love using this library.

I have encountered problem, that my datasets are very imbalanced, they have multiple classes, but classes take less than 2% of the image space, they are mainly small objects, the rest is background and it seems that Unet fails to predict accurately.

Using your segmentation_models for Tensorflow library I was able to use class weights for losses and it increased model prediction accuracy.

Is it possible to use class weights on this library? Might there be any code snippet?

Best Regards, Augustas

I've noticed more and more timm backbones being added here, which is great, but a lot of the effort is currently duplicating some features of timm, ie tracking channel numbers, modifying the networks, etc.

timm has a features_only arg in the model factory that will return a model setup as a backbone to produce pyramid features. It has a .features_info attribute you can query to understand what the channels of each output, the approx reduction factor is, etc.

I've adapted the unet and deeplab impl here in the past to use this successfully, although it was quick hack and train work, nothing to serve as a clean example.

If this was supported, any timm model (vit excluded right now) can be used as a backbone in generic fashion, just by model name string passed to creation fn, possibly a small config mapping of model types to index specificiations (some models have slightly different out_indices alignment to strides if they happen be a stride 64 model, or don't have a stride=2 feature, etc). All tap points are the latest possible point for a given feature map stride. Some, but not all of the timm backbones also support an output_stride= arg that will dilate the blocks appropriately for 8, 16 network strides.

Some references:

• https://rwightman.github.io/pytorch-image-models/feature_extraction/#multi-scale-feature-maps-feature-pyramid
• https://github.com/rwightman/efficientdet-pytorch/blob/92bb66fd0cf91d0e23fe8b10cba97e2f0bb9884f/effdet/efficientdet.py#L554-L569

For most of the models, the featuers are extracted by flattening part of the backbone model via wrapper. A few models where the feature taps are embedded deep within the model use hooks, which causes some issues with torchscript but that will likely be fixed soon in PyTorch.

Thank you very much for any help, your code is so cool! Hi, I am using the segmentation code from the example. I use my own data set to perfectly segment individual categories. But when I tried to split images of multiple categories, I still just split one category. The image I split includes three categories and a background. What I have done is ACTIVATION = 'softmax2d' I changed the category to 4 (including background) The current result is that the output of the training output is 4 masks, but only one class and one background class are included, and the other two classes are empty. Thank you again!

In the master branch, DiceLoss is implemented in such a way that the loss computed is along all class masks while it should be a mean of each diceloss for each class. And so, multiclass segmentation does not work well. This update should correct this problem

Hi,

I'm trying to convert a segmentation model (ENCODER = efficientnet-b2, DECODER = FPN) to ONNX and afterwards to TensorRT (TRT). Converting to ONNX seems to work but I can't get the conversion to TRT right. I tried the 'torch2trt' library but couldn't succeed...

Does anyone has experience with this?

Running: efficientnet-pytorch==0.6.3 onnx==1.7.0 segmentation-models-pytorch==0.1.0 torch==1.6.0 torch2trt==0.1.0 torchvision==0.7.0

Thanks in advance,

Michiel

I used the SMP library to create a UNet model with the following configurations: model = smp.Unet(encoder_name='resnet50', encoder_weights='imagenet', in_channels=3, classes=30)

However, I have also tried with other encoders (including the default resnet34) and the error seems to appear for every encoder that I choose. I am training it on a custom dataset of which the dimensions of the images are: w=320, h=192

My code runs fine until one of the final steps in the decoder block. The error traces back to smp/unet/decoder.py. When I'm running a training epoch, the error occurs in def forward(self, x, skip=None) of decoder.py

def forward(self, x, skip=None):
    x = F.interpolate(x, scale_factor=2, mode="nearest")
    if skip is not None:
            x = torch.cat([x, skip], dim=1)
            x = self.attention1(x)
    x = self.conv1(x)
    x = self.conv2(x)
    x = self.attention2(x)
    return x

For the first steps, everything runs fine and the dimensions of 'x' match with 'skip'. Below you can find a list of the dimensions of both x and skip as I go through the decoder:

STEP 1
x.shape
Out[1]: torch.Size([1, 2048, 14, 20])
skip.shape
Out[2]: torch.Size([1, 1024, 14, 20])
STEP 2
x.shape
Out[3]: torch.Size([1, 256, 28, 40])
skip.shape
Out[4]: torch.Size([1, 512, 28, 40])
STEP 3
x.shape
Out[5]: torch.Size([1, 128, 56, 80])
skip.shape
Out[6]: torch.Size([1, 256, 55, 80])
STEP 4
x.shape
Out[7]: torch.Size([1, 128, 56, 80])
skip.shape
Out[8]: torch.Size([1, 256, 55, 80])
STEP 5
x.shape
Out[9]: torch.Size([1, 3, 192, 320])
skip.shape
Out[10]: torch.Size([1, 256, 55, 80])

Around step 3, a mismatch between the tensors starts occurring which causes the error. This error traceback can be seen in the indented block below. What I find weird about this, is that I have used the exact same codebase with a different dataset that only consisted of 6 classes and in that case there was no issue. I am also unsure where this is happening as I cannot seem to find the root cause.

Traceback

(most recent call last): File "/Users/fc/Desktop/ct/segmentation_code/main.py", line 141, in trainer.train() File "/Users/fc/Desktop/ct/segmentation_code/ops/trainer.py", line 44, in train self.train_logs = self.train_epoch.run(self.trainloader) File "/Users/fc/.local/lib/python3.8/site-packages/segmentation_models_pytorch/utils/train.py", line 47, in run loss, y_pred = self.batch_update(x, y) File "/Users/fc/.local/lib/python3.8/site-packages/segmentation_models_pytorch/utils/train.py", line 87, in batch_update prediction = self.model.forward(x) File "/Users/fc/.local/lib/python3.8/site-packages/segmentation_models_pytorch/base/model.py", line 16, in forward decoder_output = self.decoder(*features) File "/Users/fc/miniconda3/envs/ct/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/Users/fc/.local/lib/python3.8/site-packages/segmentation_models_pytorch/unet/decoder.py", line 119, in forward x = decoder_block(x, skip) File "/Users/fc/miniconda3/envs/ct/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/Users/fc/.local/lib/python3.8/site-packages/segmentation_models_pytorch/unet/decoder.py", line 38, in forward x = torch.cat([x, skip], dim=1) RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 56 but got size 55 for tensor number 1 in the list.

Hi

I am using a Unet model with the encoder set to 'resnet34' and the pretrained weights are imagenet.

When I look at the model I do not see where the upsampling is occuring. The convolutions in the encoder side are occuring (although the downsampling is seemingly occuring after the intended layer e.g downsampling from layer 1 to layer 2 only occurs after layer 2), however I do not see where the upsampling takes place in the decoder side.

There is also the case where I do not see the centre block convolutions occuring.

Can I please be explained where the upsampling occurs?

My model for reference:

resnet34 Unet model.txt

I have a pretrained model checkpoint that I would like to use as the encoder weights for a segmentation model and then train this segmentation model on a new task. It looks likethe only options for encoder_weights argument are strings to certain pretrained weights within the smp library that are listed in the table. Is there a workaround to for example load some other pretrained resnet50 backbone in form of a checkpoint file as the encoder weight to an smp model?

import segmentation_models_pytorch as smp

smp.PAN(encoder_name="mit_b0")

raises the exception:

ValueError: MixVisionTransformer encoder does not support dilated mode

Since the default PAN uses dilation, this config is uncompatible atm?

If we use a configuration of PAN that does not use dilation the error, of course, does not apper:

smp.PAN(encoder_name="mit_b0", encoder_output_stride=32)

I did not test yet though if output strides of 32 still deliver comparable results. My guess would be that the default stride of 16 should encode a lot more of information that might be beneficial for better performence.

Is there any way to get it to work with dilation?

I would compute the metrics individually for each class so I would like to have in output a (1xC) vector where C is the number of classes, I was trying like this but it throws me an error:

output = torch.rand([10, 3, 256, 256])
target = torch.rand([10, 1, 256, 256]).round().long()

# first compute statistics for true positives, false positives, false negative and
# true negative "pixels"
tp, fp, fn, tn = smp.metrics.get_stats(output, target, mode='multi class', num_classes = 3)

# then compute metrics with required reduction (see metric docs)
iou_score = smp.metrics.iou_score(tp, fp, fn, tn, reduction="macro-imagewise")
f1_score = smp.metrics.f1_score(tp, fp, fn, tn, reduction="macro-imagewise")
false_negatives = smp.metrics.false_negative_rate(tp, fp, fn, tn, reduction=None)
recall = smp.metrics.recall(tp, fp, fn, tn, reduction=None)

The error:

ValueError: For ``multiclass`` mode ``target`` should be one of the integer types, got torch.float32.

When defining a smp model in __init__() as:

self.base = smp.Unet(encoder_name='resnet50', pretrained='imagenet', 
                                  in_channels=3, classes=7,
                                  activation='softmax') 

This will throw the following warning upon initialisation:

~/anaconda3/envs/geo/lib/python3.7/site-packages/segmentation_models_pytorch/base/modules.py:116: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument. return self.activation(x)

As the softmax is defined by passing 'softmax' as an arg, I'm not sure where/how to include the dim as the warning suggests? Many thanks

See also this closed (but without stating resolution) issue: Originally posted by @vdplasthijs in https://github.com/qubvel/segmentation_models.pytorch/issues/169#issuecomment-1334066128

I was following the cars example but I keep getting this error. I installed the module with pip in both suggested ways but I keep getting this error even if I checked the presence of utils in the repo. What am I doing wrong?

This is the code:

!pip install git+https://github.com/qubvel/segmentation_models.pytorch
# !pip install -q segmentation-models-pytorch

train_epoch = smp.utils.train.TrainEpoch(
    model, 
    loss=loss, 
    metrics=metrics, 
    optimizer=optimizer,
    device=DEVICE,
    verbose=True,
)