Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

Related tags

Deep Learning SCUNet
Overview

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

visitors

[Paper] [Online Demo]

The following results are obtained by our SCUNet with purely synthetic training data! We did not use the paired noisy/clean data by DND and SIDD during training!

Swin-Conv-UNet (SCUNet) denoising network

The architecture of the proposed Swin-Conv-UNet (SCUNet) denoising network. SCUNet exploits the swin-conv (SC) block as the main building block of a UNet backbone. In each SC block, the input is first passed through a 1×1 convolution, and subsequently is split evenly into two feature map groups, each of which is then fed into a swin transformer (SwinT) block and residual 3×3 convolutional (RConv) block, respectively; after that, the outputs of SwinT block and RConv block are concatenated and then passed through a 1×1 convolution to produce the residual of the input. “SConv” and “TConv” denote 2×2 strided convolution with stride 2 and 2×2 transposed convolution with stride 2, respectively.

New data synthesis pipeline for real image denoising

Schematic illustration of the proposed paired training patches synthesis pipeline. For a high quality image, a randomly shuffled degradation sequence is performed to produce a noisy image. Meanwhile, the resizing and reverse-forward tone mapping are performed to produce a corresponding clean image. A paired noisy/clean training patches are then cropped for training deep blind denoising model. Note that, since Poisson noise is signal-dependent, the dashed arrow for “Poisson” means the clean image is used to generate the Poisson noise. To tackle with the color shift issue, the dashed arrow for “Camera Sensor” means the reverse-forward tone mapping is performed on the clean image.

Synthesized noisy/clean patch pairs via our proposed training data synthesis pipeline. The size of the high quality image patch is 544×544. The size of the noisy/clean patches is 128×128.

Web Demo

Try Replicate web demo for SCUNet models here Replicate

Codes

  1. Download SCUNet models
python main_download_pretrained_models.py --models "SCUNet" --model_dir "model_zoo"
  1. Gaussian denoising

    1. grayscale images
    python main_test_scunet_gray_gaussian.py --model_name scunet_gray_25 --noise_level_img 25 --testset_name set12
    1. color images
    python main_test_scunet_color_gaussian.py --model_name scunet_color_25 --noise_level_img 25 --testset_name bsd68
  2. Blind real image denoising

    python main_test_scunet_real_application.py --model_name scunet_color_real_psnr --testset_name real3

Results on Gaussian denoising

Results on real image denoising

@article{zhang2022practical,
title={Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis},
author={Zhang, Kai and Li, Yawei and Liang, Jingyun and Cao, Jiezhang and Zhang, Yulun and Tang, Hao and Timofte, Radu and Van Gool, Luc},
journal={arXiv preprint},
year={2022}
}
Comments
  • Data range is clipped for 8 bit images with noise?

    Data range is clipped for 8 bit images with noise?

    Hi I am using 8-bit greyscale as input, and note the usual /255 operations to put data in the range 0-1. However the operation

            if args.need_degradation:  # degradation process
                np.random.seed(seed=0)  # for reproducibility
                img_L += np.random.normal(0, args.noise_level_img/255., img_L.shape)
    

    can result in values outside this range, as shown in the image below: image

    I note that the values are later clipped by util.single2uint so what is saved is not precisely what is processed. Furthermore I wonder the effect on inferencing - presumably values are also clipped to 0-1 range?

    opened by robmarkcole 2
  • Will there be more models?

    Will there be more models?

    Very impressed with the results! However at times even the lowest noise reduction "15" model can remove too much detail. Will there be a "5" or "10" noise level model in the future?

    Thank you for all the hard work!

    opened by AIisCool 2
  • Demo and pre-trained models on Hugging Face Spaces

    Demo and pre-trained models on Hugging Face Spaces

    Hello! Your demo on Replicate is extremely cool ✨ Would you be open to adding a demo on Hugging Face Spaces as well? We've got docs for doing that, but I'd be happy to help out with it!

    If you'd like, you can also mirror your model weights over on the Hugging Face model hub as well, which I know our users would love 🤗 People have been building all kinds of cool things with the models on the Hub these days, like this demo that combines 3 different models 👀

    scunet_color_real_gan could even be uploaded as a private or gated model, if that's something you're interested in.

    cc: @osanseviero

    opened by NimaBoscarino 1
  • Add Web Demo & Docker environment

    Add Web Demo & Docker environment

    Hey @cszn! 👋

    This pull request makes it possible to run your model inside a Docker environment, which makes it easier for other people to run it. We're using an open source tool called Cog to make this process easier.

    This also means we can make a web page where other people can try out your model! View it here: https://replicate.com/cszn/scunet. The docker file can be found under the tab ‘run model with docker’.

    We have added some examples, but do claim the page so you can own the page, customise the Example gallery as you like, push any future update to the web demo, and we'll feature it on our website and tweet about it too. Moreover, when the page is claimed, it will be automatically linked to the arXiv website as well (under “Demos”)! You can find the 'Claim this model' button on the top of the page ~

    In case you're wondering who I am, I'm from Replicate, where we're trying to make machine learning reproducible. We got frustrated that we couldn't run all the really interesting ML work being done. So, we're going round implementing models we like. 😊

    opened by chenxwh 1
  • Fantastic project, great results, but...

    Fantastic project, great results, but...

      File "C:\Users\Miki\SCUNet\models\network_scunet.py", line 88, in forward
        if self.type!='W': output = torch.roll(output, shifts=(self.window_size//2, self.window_size//2), dims=(1,2))
    RuntimeError: CUDA out of memory. Tried to allocate 486.00 MiB (GPU 0; 12.00 GiB total capacity; 10.76 GiB already allocated; 0 bytes free; 11.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
    

    @cszn Please, add minor changes to code, to support --tile.... in before opened issue, I asked for same question, followed your instructions, uncommented line # number 99th but same problem appears again.... No way to denoise photos larger than 3 Mpx...due cuda oom errors. It is pain doing in graphic editor manualy cropping & after denoise joinning big photo gallery I kindly ask for help... Cause this project has realy great potencial. Onothers like swinir, restormer etc... doesnt do joob like your, in terms on real denoising.... tnx very much in advance

    opened by zelenooki87 1
  • Control sigma for scunet_color_real_psnr?

    Control sigma for scunet_color_real_psnr?

    Results look very good, better than the "scunet_color" at times. But it would be nice to control the noise reduction like the grayscale and color ones.

    Thank you.

    opened by AIisCool 1
  • Training time of SCUNet

    Training time of SCUNet

    you have mention in paper that

    It takes about three days to train a denoising model on four NVIDIA RTX 2080 Ti GPUs.

    however, in my case, it takes about 5 days to train on 4 V100 GPUs for 1200000 iterations and pics are crop into 544*544. It is much longer than 3 days. How can i speed up? thanks.

    opened by ncbnwklrt 0
  • Cannot convert to TensorRT.

    Cannot convert to TensorRT.

    https://github.com/NVIDIA-AI-IOT/torch2trt/issues/807

    TensorRT not support move batch dim.

    https://forums.developer.nvidia.com/t/tensorrt-does-not-support-permute-in-n-batch-dimension/69925

    opened by Ken1256 0
Owner
Kai Zhang
Image Restoration; Inverse Problems
Kai Zhang
The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"

Swin-Unet The codes for the work "Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation"(https://arxiv.org/abs/2105.05537). A validatio

null 869 Jan 7, 2023
Code for Blind Image Decomposition (BID) and Blind Image Decomposition network (BIDeN).

arXiv, porject page, paper Blind Image Decomposition (BID) Blind Image Decomposition is a novel task. The task requires separating a superimposed imag

null 64 Dec 20, 2022
A python software that can help blind people find things like laptops, phones, etc the same way a guide dog guides a blind person in finding his way.

GuidEye A python software that can help blind people find things like laptops, phones, etc the same way a guide dog guides a blind person in finding h

Munal Jain 0 Aug 9, 2022
This is an official implementation of the CVPR2022 paper "Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots".

Blind2Unblind: Self-Supervised Image Denoising with Visible Blind Spots Blind2Unblind Citing Blind2Unblind @inproceedings{wang2022blind2unblind, tit

demonsjin 58 Dec 6, 2022
CoaT: Co-Scale Conv-Attentional Image Transformers

CoaT: Co-Scale Conv-Attentional Image Transformers Introduction This repository contains the official code and pretrained models for CoaT: Co-Scale Co

mlpc-ucsd 191 Dec 3, 2022
Sdf sparse conv - Deep Learning on SDF for Classifying Brain Biomarkers

Deep Learning on SDF for Classifying Brain Biomarkers To reproduce the results f

null 1 Jan 25, 2022
Direct application of DALLE-2 to video synthesis, using factored space-time Unet and Transformers

DALLE2 Video (wip) ** only to be built after DALLE2 image is done and replicated, and the importance of the prior network is validated ** Direct appli

Phil Wang 105 May 15, 2022
BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

Bilateral Denoising Diffusion Models (BDDMs) This is the official PyTorch implementation of the following paper: BDDM: BILATERAL DENOISING DIFFUSION M

null 172 Dec 23, 2022
[NeurIPS 2020] Blind Video Temporal Consistency via Deep Video Prior

pytorch-deep-video-prior (DVP) Official PyTorch implementation for NeurIPS 2020 paper: Blind Video Temporal Consistency via Deep Video Prior TensorFlo

Yazhou XING 90 Oct 19, 2022
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 44 Dec 27, 2022
This is an official implementation for "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" on Object Detection and Instance Segmentation.

Swin Transformer for Object Detection This repo contains the supported code and configuration files to reproduce object detection results of Swin Tran

Swin Transformer 1.4k Dec 30, 2022
Code of PVTv2 is released! PVTv2 largely improves PVTv1 and works better than Swin Transformer with ImageNet-1K pre-training.

Updates (2020/06/21) Code of PVTv2 is released! PVTv2 largely improves PVTv1 and works better than Swin Transformer with ImageNet-1K pre-training. Pyr

null 1.3k Jan 4, 2023
This project aims to explore the deployment of Swin-Transformer based on TensorRT, including the test results of FP16 and INT8.

Swin Transformer This project aims to explore the deployment of SwinTransformer based on TensorRT, including the test results of FP16 and INT8. Introd

maggiez 87 Dec 21, 2022
Unofficial PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution

PyTorch reimplementation of the paper Swin Transformer V2: Scaling Up Capacity and Resolution [arXiv 2021].

Christoph Reich 122 Dec 12, 2022
Implementation of the Swin Transformer in PyTorch.

Swin Transformer - PyTorch Implementation of the Swin Transformer architecture. This paper presents a new vision Transformer, called Swin Transformer,

null 597 Jan 3, 2023
Tensorflow implementation of Swin Transformer model.

Swin Transformer (Tensorflow) Tensorflow reimplementation of Swin Transformer model. Based on Official Pytorch implementation. Requirements tensorflow

null 167 Jan 8, 2023
This is an official implementation for "Video Swin Transformers".

Video Swin Transformer By Ze Liu*, Jia Ning*, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin and Han Hu. This repo is the official implementation of "V

Swin Transformer 981 Jan 3, 2023
Unofficial implementation of "Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" (https://arxiv.org/abs/2103.14030)

Swin-Transformer-Tensorflow A direct translation of the official PyTorch implementation of "Swin Transformer: Hierarchical Vision Transformer using Sh

null 52 Dec 29, 2022
This is an official implementation for "Self-Supervised Learning with Swin Transformers".

Self-Supervised Learning with Vision Transformers By Zhenda Xie*, Yutong Lin*, Zhuliang Yao, Zheng Zhang, Qi Dai, Yue Cao and Han Hu This repo is the

Swin Transformer 529 Jan 2, 2023