U^2-Net - Portrait matting This repository explores possibilities of using the original u^2-net model for portrait matting.

Last update: Nov 25, 2022

Related tags

Overview

U^2-Net - Portrait matting

This repository explores possibilities of using the original u^2-net model for portrait matting. The training code is rewritten and optimised for performance and reliability:

allow mixed precision training
allow channels_last to increase performance on tensor cores (initial observations show a performance increase of roughly 80%)
prefetch batches to the GPU
more efficient tensor conversations
configuration management with hydra
tensorboard logging
augmentations, dataset weights, suppert for aisegment rgb masks, ...
streamlit demo frontend using pymatting for foreground estimations
a lot more

I changed the loss function of the original proposal to allow two different kind of masks as ground truth: rough masks and refined masks where refined masks can be for example alpha mattes synthesized using pymatting (my experiments were quite successful). This however is just a proof of concept and the refined masks itself are quite noisy. The different losses are in fact conflicting a bit to each other however initial results show quite interesting results.

Example & pretrained model

The model is able to pickup details quite precisely (e.g. hair regions) with very little overshoot however it also shows weaknesses of this training approach e.g. the confusion at the left shoulder. The model has only been trained for a couple of epochs on a synthesized dataset and is not ready for production use. However feel free to re-use it for any similar experiments.

The provided checkpoint is compatible with the original u^2-net implementation and code.

OneDrive

Citation

@InProceedings{Qin_2020_PR,
title = {U2-Net: Going Deeper with Nested U-Structure for Salient Object Detection},
author = {Qin, Xuebin and Zhang, Zichen and Huang, Chenyang and Dehghan, Masood and Zaiane, Osmar and Jagersand, Martin},
journal = {Pattern Recognition},
volume = {106},
pages = {107404},
year = {2020}
}

Comments

Hi, how to solve below error to start training? [email protected]

/usr/local/lib/python3.7/dist-packages/hydra/core/utils.py:148: UserWarning: register_resolver() is deprecated. See https://github.com/omry/omegaconf/issues/426 for migration instructions.

OmegaConf.register_resolver(name, f) The current experiment will be tracked as 20220329-134515 Results will be saved in ./experiments/20220329-134515 /usr/local/lib/python3.7/dist-packages/hydra/utils.py:60: UserWarning: OmegaConf.is_none() is deprecated, see https://github.com/omry/omegaconf/issues/547 if OmegaConf.is_none(config): /usr/local/lib/python3.7/dist-packages/hydra/_internal/utils.py:710: UserWarning: OmegaConf.is_none() is deprecated, see https://github.com/omry/omegaconf/issues/547 if OmegaConf.is_none(v): /usr/local/lib/python3.7/dist-packages/hydra/_internal/utils.py:577: UserWarning: OmegaConf.is_none() is deprecated, see https://github.com/omry/omegaconf/issues/547 if OmegaConf.is_dict(x) and not OmegaConf.is_none(x): Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/hydra/utils.py", line 109, in instantiate return target(*args, **final_kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 245, in init raise ValueError('sampler option is mutually exclusive with ' ValueError: sampler option is mutually exclusive with shuffle

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/content/u-2-net-portrait-master/train.py", line 77, in main sampler=sampler) File "/usr/local/lib/python3.7/dist-packages/hydra/utils.py", line 116, in instantiate ).with_traceback(sys.exc_info()[2]) File "/usr/local/lib/python3.7/dist-packages/hydra/utils.py", line 109, in instantiate return target(*args, **final_kwargs) File "/usr/local/lib/python3.7/dist-packages/torch/utils/data/dataloader.py", line 245, in init raise ValueError('sampler option is mutually exclusive with ' ValueError: Error instantiating/calling 'torch.utils.data.dataloader.DataLoader' : sampler option is mutually exclusive with shuffle

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

opened by nissansz 22
Loss does not change from nan

Hi, I am really appreciate of your work

I am enjoying with your trained model. I am trying to train my own dataset, I got stuck with that loss keeps showing me nan

log :

src image is source image of mask. Cannot upload here due to copyright

mask:

env: env.txt

opened by gyupro 2
The model is in the train mode in demo

Hello! Why is the model put into train mode in the demo? https://github.com/dennisbappert/u-2-net-portrait/blob/master/demo.py#L78 The difference between train() and eval() is huge actually.

opened by dfrumkin 2
Real time webcam background matting?

Is it fast enough to replace webcam background in real time? Can I run it in the browser with tensorflow.js? Is there online demo similar to the MODNet demo and notebook?

opened by benbro 2

Create segmentation maps from Supervise.ly Persons dataset

Hi @dennisbappert,

Thanks a lot for open sourcing your implementation!

I want to use JSON information in Supervise.ly dataset to create grayscale segmentation map for the images. Consider the following image (file name: pexels-photo-11392.png).

pexels-photo-11392

Its annotation (file name: pexels-photo-11392.png.json) is given as following in JSON format.

{
    "description": "",
    "tags": [],
    "size": {
        "height": 800,
        "width": 1291
    },
    "objects": [
        {
            "id": 511624464,
            "classId": 1846336,
            "description": "",
            "geometryType": "bitmap",
            "labelerLogin": "[email protected]",
            "createdAt": "2020-09-11T08:57:16.643Z",
            "updatedAt": "2020-09-11T08:57:16.643Z",
            "tags": [],
            "classTitle": "person_bmp",
            "bitmap": {
                "data": "eJwB9AUL+olQTkcNChoKAAAADUlIRFIAAAI0AAAC9QEDAAAA6vHsUQAAAAZQTFRFAAAA////pdmf3QAAAAF0Uk5TAEDm2GYAAAWcSURBVHic7d1LctswDAbgZLrIkkfgUXg08mg6Co/ApRceqVPXbSyJT/BnDNPAuvMNBUIQZEnpx0chtltcS/+sEGb7F65D0dt3eMhqti10OA/MdqEzn49OR6Z/PTor3fl6dDo2TO2chezonePJjtk5gezYnUPf+L1D37C9s6EcR3X2DH3jD04AOeQNY+Z8Hhxq53gXh3qC/XoTh9o4uDlfoxwnzmOoUc4izmPoo+PFyTqB5ph3cYgDB8qx7+IQBxeUc2TEGeIcx0ycQxvsuDmnsU6cXZzGMeLAKs7LO06crLPM4Khxjp/B0eOcAHJIAyI3x0zq2HEOabBDOWeG5JzHOpxDGYBmdc7j4cTOJ8j5chhHLxBnsx7jEK5gcSeAnPaOGHfaO0fcaa9EcWZwUPUjTt5Bne/PcyK3XwyPK4iTdbw4P+IskzruaU7s/ouf0868lUO54UE5dqBDuYF/J4fyAxDK0QOdIE7W8SBnATkEJuaQHjRGHNqLDpjtijkLyHEYh/hiJSjNZyc81zGTOpq742mOEicbp18UuDkLzTnd8czquOc6pydFROfUoFEOkTlNviCH/IGTGeOQv7Q7OOTvgLg5eozjxXkpZ3m2o97D8eJkncDMIffDQQ75+nVwyNfTg0OeW0Y5XpysE5g51BPj6FBPjFEOdYAe5jiao0c5C80xR8fTHAtyjgzxxBj3QRGtgE7lg3thPYAc1Au1lATFXhilJCj6oDqAHEKCoi/cEGaFyHaREqSjTvs1w0ad9gOLO+09Mc4073y0DAkHlnJaDyz+3kX7gcXf+9qaS1qlnMYD00nHNzkm6bQ1IZt2mhaUZppKKFk+jQtKlk/jgpLl07gglXXqa9FmneoFZdPckKF8euoXpEtO5YJKTGX7yFbPPWocVeHULMhUOBULKu167YKKu36LclFrkGOrnHIJVTHlRNdUz59wBUdVOr7gGJBjK52QZ+qqcCvOZnVVuBULSNU6hQIytU6hgKqZfAHVVuFWGKar01woIF3vhJxj651cAVVX4ZYvoIY0ZwtINTi5AjItjoOkJ1dAqsnxScc2OSHFtOzWlikg1eYkC8i2OakCajyspKMbnVQhtjKJQmxoPfdYoo5pdjzmsOKF2LpbKUe1O9GC1iDHYpy21vM3YicYIc1RR4EcTXBiJ6olOLETlcJEHEqaY44COZrkOEyaI42DxJwdWprPTntPjTua5pxOeAtyaMzJIab55BDTfHI00Tk2DgtyiMzRoaZ5lEPdLnbOYQJSzBzNzDHMHPt0JwxxyMwYhzJijnTIbewwaXJz6G1sjKOe7+wmVj2pY5g5lpnTwezuVHoc/810tDHWTkcbG+J0tDGcE74dxcG54B3NzDE9zhXv2B5nxTs9DGOnq40NcLra2MMFDOV0tTF+joO0sYc7OZSjmTmGmWN5OB7t9DFwp6+N4ZyAaWNsnc429n9wmc25ipONdXJHdTrbrI4TJxvLpI7HOno2J0zqXMT5SceII84zHT+pc78OWnHi4Zg53PLjQU5g5tz7Ri8Dc64gZ53U2WZ1HDNnQTq9P0MCHT+pE5g53PLD0un9GXu7nxfivIzjxPkR53b56v05HOeszJy/eRYnGQvI8SAngJzbjYoS51Wcqziv52g+zjqps83quEmdBegYRo5j5nxM6qDy40FOADmXiR0rzss4QZxscNsvjg6AYedcmTkrM2fj5jhmzsLM8YjHFwydII444kQcwOOUiR2HcVbIY6vb3AJxgjjv6gAeE4lTios44vBwlDjpuIojzlyOFicdqzjizOUYccQRh71jxcmEOOLM5UAYcd7W0eKIM5FjEAzDuU4cccQRRxxxxJnP0eKI0+EoccTpcCDvgaDeP1xB34OsoO/IVtB3o//+tLUFOQ9BInP/wX0LmXfqyWqnQK4fvwEMZtcJQnACDQAAAABJRU5ErkJggkOY3sA=",
                "origin": [
                    363,
                    42
                ]
            }
        }
    ]
}

I don't understand how to create segmentation map from this data. Could you please guide me?

Regards Rahul Bhalley

opened by RahulBhalley 0

Regarding intuitions behind certain improvements

Hi, thanks for the work! i was interested in certain improvements you had made, and was wondering if you had an insight about why they help. you mention mixed precision training and channels first help. could you help me understand why these have that effect?

thanks

opened by tumble-weed 2

training

Thank you for sharing your work! I'm trying to figure out how to start training, but am not very familiar with hydra and am running into errors with it. I modified conf/dataset.yaml, but also changed it back to what it was originally and it still gives the same error. Perhaps you might have an idea what the problem is?

File "train.py", line 279, in <module>
   main()
 File "/usr/local/lib/python3.8/dist-packages/hydra/main.py", line 49, in decorated_main
   _run_hydra(
 File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 367, in _run_hydra
   run_and_report(
 File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 214, in run_and_report
   raise ex
 File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 211, in run_and_report
   return func()
 File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/utils.py", line 368, in <lambda>
   lambda: hydra.run(
 File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/hydra.py", line 110, in run
   _ = ret.return_value
 File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 233, in return_value
   raise self._return_value
 File "/usr/local/lib/python3.8/dist-packages/hydra/core/utils.py", line 160, in run_job
   ret.return_value = task_function(task_cfg)
 File "train.py", line 73, in main
   train_collate_fn = dataset.get_collate_fn(mode='train', channels_last=cfg.trainer.channels_last)
 File ".../u-2-net-portrait/lib/data/dataset.py", line 74, in get_collate_fn
   transforms[configuration.identifier] = [instantiate(step) for step in steps]
 File ".../u-2-net-portrait/lib/data/dataset.py", line 74, in <listcomp>
   transforms[configuration.identifier] = [instantiate(step) for step in steps]
 File "/usr/local/lib/python3.8/dist-packages/hydra/_internal/instantiate/_instantiate2.py", line 182, in instantiate
   raise InstantiationException(
hydra.errors.InstantiationException: Top level config has to be OmegaConf DictConfig, plain dict, or a Structured Config class or instance

As a side note, a requirements.txt file would be useful to include in this repo

opened by ucb-pb 3

about train my class segmentation
hi，thanks for your great working，I want to train my single class segmentation based on your project，I have some doubt about training：

the dataset format：the data directory should include image and mask，the mask is binary single channel image，right？？

could I run train.py directly？？what else should I consider and modify？？ thanks best regard
opened by xiongzhu666 2
Training dataset size

Hi! Thanks a lot for the great work here. Would you mind sharing how big was the dataset you used to train your model? Also, which strategies did you use to create the 440x440 images you used for training? Simply resizing original pics + alpha mattes or random cropping to preserve size ratios?

Thanks a lot again and have a great one!

opened by FraPochetti 2

Owner

Dennis Bappert

GitHub

Rethinking Portrait Matting with Privacy Preserving

Rethinking Portrait Matting with Privacy Preserving This is the official repository of the paper Rethinking Portrait Matting with Privacy Preserving.

184 Jan 3, 2023

Script that receives an Image (original) and a set of images to be used as "pixels" in reconstruction of the Original image using the set of images as "pixels"

picinpics Script that receives an Image (original) and a set of images to be used as "pixels" in reconstruction of the Original image using the set of

1 Oct 24, 2021

RGBD-Net - This repository contains a pytorch lightning implementation for the 3DV 2021 RGBD-Net paper.

[3DV 2021] We propose a new cascaded architecture for novel view synthesis, called RGBD-Net, which consists of two core components: a hierarchical depth regression network and a depth-aware generator network.

4 May 26, 2022

Home repository for the Regularized Greedy Forest (RGF) library. It includes original implementation from the paper and multithreaded one written in C++, along with various language-specific wrappers.

Regularized Greedy Forest Regularized Greedy Forest (RGF) is a tree ensemble machine learning method described in this paper. RGF can deliver better r

364 Dec 28, 2022

This repository contains an overview of important follow-up works based on the original Vision Transformer (ViT) by Google.

75 Dec 2, 2022

This repository is all about spending some time the with the original problem posed by Minsky and Papert

This repository is all about spending some time the with the original problem posed by Minsky and Papert. Working through this problem is a great way to begin learning computer vision.

1 Jan 23, 2022

Neural networks applied in recognizing guitar chords using python, AutoML.NET with C# and .NET Core

Chord Recognition Demo application The demo application is written in C# with .NETCore. As of July 9, 2020, the only version available is for windows

24 Oct 22, 2022

Implementation of SETR model, Original paper: Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.

SETR - Pytorch Since the original paper (Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers.) has no official

112 Dec 16, 2022

An original implementation of "Noisy Channel Language Model Prompting for Few-Shot Text Classification"

Channel LM Prompting (and beyond) This includes an original implementation of Sewon Min, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer. "Noisy Cha

92 Jan 7, 2023

PyTorch original implementation of Cross-lingual Language Model Pretraining.

XLM NEW: Added XLM-R model. PyTorch original implementation of Cross-lingual Language Model Pretraining. Includes: Monolingual language model pretrain

2.7k Dec 27, 2022

Arch-Net: Model Distillation for Architecture Agnostic Model Deployment

Arch-Net: Model Distillation for Architecture Agnostic Model Deployment The official implementation of Arch-Net: Model Distillation for Architecture A

22 Jan 5, 2023

《Deep Single Portrait Image Relighting》(ICCV 2019)

Ratio Image Based Rendering for Deep Single-Image Portrait Relighting [Project Page] This is part of the Deep Portrait Relighting project. If you find

62 Dec 21, 2022

dualFace: Two-Stage Drawing Guidance for Freehand Portrait Sketching (CVMJ)

dualFace dualFace: Two-Stage Drawing Guidance for Freehand Portrait Sketching (CVMJ) We provide python implementations for our CVM 2021 paper "dualFac

46 Nov 10, 2022

Official Implementation and Dataset of "PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask and Group-Level Consistency", CVPR 2021

Portrait Photo Retouching with PPR10K Paper | Supplementary Material PPR10K: A Large-Scale Portrait Photo Retouching Dataset with Human-Region Mask an

184 Dec 11, 2022

[TOG 2021] PyTorch implementation for the paper: SofGAN: A Portrait Image Generator with Dynamic Styling.

This repository contains the official PyTorch implementation for the paper: SofGAN: A Portrait Image Generator with Dynamic Styling. We propose a SofGAN image generator to decouple the latent space of portraits into two subspaces: a geometry space and a texture space. Experiments on SofGAN show that our system can generate high quality portrait images with independently controllable geometry and texture attributes.

694 Dec 23, 2022

The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

261 Jan 9, 2023

The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic Neural Rendering"

Website | ArXiv | Get Start | Video PIRenderer The source code of the ICCV2021 paper "PIRenderer: Controllable Portrait Image Generation via Semantic

81 Sep 25, 2021

Official code for paper Exemplar Based 3D Portrait Stylization.

3D-Portrait-Stylization This is the official code for the paper "Exemplar Based 3D Portrait Stylization". You can check the paper on our project websi

60 Dec 7, 2022

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

The Medical Detection Toolkit contains 2D + 3D implementations of prevalent object detectors such as Mask R-CNN, Retina Net, Retina U-Net, as well as a training and inference framework focused on dealing with medical images.

1.2k Jan 4, 2023