Really interesting work. I have a baseline with softmax loss with the Deeplabv3 and achieve mIOU=76.7 on the Cityscapes.

And I simply replace the cross entropy loss with your proposed loss and train the models with the same learning rate and weight decay, but I only achieve mIOU=64.7.

Could you give me some hint?

I also notice that you also do not train the ENet from scratch and you just finetune the models.

Besides, I also conduct a small experiments to train the models with both the cross entropy loss and your proposed loss, which achieves a good performance: mIOU=78.4.

It would be great if you could share me your advice!

I'm trying to use your tensorflow implementation for a U-net using keras. The problem I face is that I cannot simply plugin the lovasz_softmax loss function U-Net model since the loss function takes the labels as image batches dim(labels) = (batchsize, width, height, 1) and the probas in the one-hot-vector notation - as it should be for such a problem (dim(probas)=(batchsize, width, height, n_classes).

Simply taking argmax of the probas does not work because keras raises an Exception when trying to calculate the gradient of argmax during training. If I understood your paper correctly, this is problem what the use lovasz softmax should avoid.

model.compile(optimizer=SDG() , loss=lovasz_sofmax, metrics=["accuracy"])

Hello, nice to read this paper. I have encountered the problem that I got a very low miou(0.003) from Deeplabv3+ with Lovasz_softmax. It can normally achieve miou=76% using cross entropy loss. Environment: pytorch 1.0 Ubuntu 16.04 batch size: 10 dataset: Pascal VOC 2012 (aug) loaded ImageNet pretrained ResNet-101 weight

And here is the code of Lovasz softmax:

class LovaszSoftmax(nn.Module):
    def __init__(self, per_image=False):
        super(LovaszSoftmax, self).__init__()
        self.lovasz_softmax = lovasz_softmax
        self.per_image = per_image

    def forward(self, pred, label):
        pred = F.softmax(pred, dim=1)
        return self.lovasz_softmax(pred, label, per_image=self.per_image, ignore=255)

Thanks!

Hi there,

Thanks for sharing the code of this fantastic job. Congratulations on your CVPR paper! I have a question about the labels in the ground-truths. The gt labels should be {-1,1} (-1:background, 1:foreground) and for instance {0,1} (0:background, 1:foreground) doesn't work properly, right?

I have an image segmentation task with small batch size (4-8) and some samples that have only the background (negative) class.

I have implemented lovasz softmax as below:

loss2 = lovasz_softmax(probs, labels, classes=[1], per_image=False)

where probs are B, H, W, C, and label is B, H, W with a range of [0, 1]

However, the network does not learn at all -- the output feature maps look random, and tuning the learning rate does not improve the issue.

The same network works fine with dice, Tversky, focal, or BCE loss.

I think it is due to the presence of background-only classes -- I know that classes = 'present' solves this for multi class problems. Is there a way to do the same for a binary lovasz softmax?

Hi, I've tried to run PyTorch implementation in Python 3.6 and got the error:

    226     l = iter(l)
    227     if ignore_nan:
--> 228         l = ifilterfalse(isnan, l)
    229     try:
    230         n = 1

NameError: name 'ifilterfalse' is not defined

I think the function needs to be aliased after import in Python 3 like this:

from itertools import filterfalse as filterfalse

An error occurred in line 177 ( C = probas.size(1) ) when input labels are all ignored, because labels got empty. This PR deal with such problems.

One thing I concerned about is the difference between the lines I added and below lines

        if len(labels) == 0:
            # only void pixels, the gradients should be 0
            return logits.sum() * 0.

Above is a part of the original implementation of lovasz_flat. This doesn't exclude void labels for calculation the loss. Is it excepted behavior?

Hello, I loved your work on Lovasz softmax very much and implemented it on a modified version of Deeplabv3+ in Tensorflow. However, I experienced significant speed drop, the time used per step increased from 0.4s ( using cross entropy) to now almost 3.8s. Is this normal or did I do something wrong? Thankyou!

Hi, Thank you for your work.

I believe there is error here

https://github.com/bermanmaxim/LovaszSoftmax/blob/6309c68a2276ada25ebf04692575bce937460f1a/lovasz_losses.py#L26

should be jaccard[1:p] = jaccard[1:p] - jaccard[:p-1]

Hi Maxim Berman! Great Work. I am using the lovasz_hinge loss and iou_binary as my metric. My labels are binary masks of foreground represented as 1 and background as 0. And I am currently overfitting one example just to see how my model(which is a form of hyper-network) works. But as the Lovasz Hinge loss decreases, the output of iou_binary also decreases. Thanks alot in advance for helping

hi, Maxim

you present a very interesting and solid work! but I met some implementation error while using your lovasz loss in my deeplab v3+. My initial loss is tf.losses.softmax_cross_entropy, and I prepared:

onehot_labels: [batch_size, num_classes] target one-hot-encoded labels. logits: [batch_size, num_classes] logits outputs of the network .

as its input, but it just didn't fit in your loss, could you please give some advice about how to transfer original params into your params like probas, labels? Thank you!

hi, @bermanmaxim jaccard_loss = 1 - IOU why do the jaccard[1:] - jaccard[0:-1]?

def lovasz_grad(gt_sorted):
    """
    Computes gradient of the Lovasz extension w.r.t sorted errors
    See Alg. 1 in paper
    """
    p = len(gt_sorted)
    gts = gt_sorted.sum()
    intersection = gts - gt_sorted.float().cumsum(0)
    union = gts + (1 - gt_sorted).float().cumsum(0)
    jaccard = 1. - intersection / union
    if p > 1: # cover 1-pixel case
        jaccard[1:p] = jaccard[1:p] - jaccard[0:-1]
    return jaccard

pip install lovasz

ERROR: Could not find a version that satisfies the requirement lovasz (from versions: none) ERROR: No matching distribution found for lovasz Note: you may need to restart the kernel to use updated packages.

Please let me know the solution to this.

Dear BermanMaxin, Thanks for your great work,it has helped me a lot! I got a confusion that how to combine lovasz hinge and bce in binary segmentaion task appropriately. As we all know,lovasz hinge expects logits(without sigmoid),but bce need relust after sigmoid.What confuses me is that whether these two different types(with/without sigmoid) losses can get along well. Other combo loss,e.g. bce+lovasz softmax,bce+dice, all need sigmoid so in my mind there is no problem. Could you give me some advice about this,thanks! In addition,if 'per_image=False' can bring a faster convergence when batch size is big. Thanks.

Hi, thanks for your great work. But I have a question here. emm, when I have a multi classes semantic segmentation task, I can convert the label to one-hot format and do sigmoid to the output of the network then apply nn.BCELoss() to the label and outputs. (Certainly, one-hot + no sigmoid outputs + nn.BCEWithLogitsLoss is also ok), when i do the inference, i just do torch.sigmoid to the outputs of the network and set the thershold as 0.5, then i can get the correct results of semantic segmentation.So may I do the same thing to the lovasz_hinge()? one-hot + no sigmoid outputs + lovasz_hinge?Does that work? And the inference process is same as above?

Hi，thanks for your work. I have added the tensorflow version of lovaszsoftmax to my task，I trained the model with Cross entropy loss first, then fine tuned it with Cross entropy loss + lovaszsoftmax loss（the weights is 1:1）, the mIoU improved about 2%. But when tested on videos it seems like that the model without lovaszsoftmax loss performs better, especially on recall. Do you have any idea about this, thank you.

my task is 2-class lane segmentation