Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA results for single-image motion deblurring, image deraining, image denoising (synthetic and real data), and dual-pixel defocus deblurring.

Overview

Restormer: Efficient Transformer for High-Resolution Image Restoration

Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang

Paper: https://arxiv.org/abs/2111.09881

Testing codes and pre-trained models will be released on Nov 22nd!


Abstract: Since convolutional neural networks (CNNs) perform well at learning generalizable image priors from large-scale data, these models have been extensively applied to image restoration and related tasks. Recently, another class of neural architectures, Transformers, have shown significant performance gains on natural language and high-level vision tasks. While the Transformer model mitigates the shortcomings of CNNs (i.e., limited receptive field and inadaptability to input content), its computational complexity grows quadratically with the spatial resolution, therefore making it infeasible to apply to most image restoration tasks involving high-resolution images. In this work, we propose an efficient Transformer model by making several key designs in the building blocks (multi-head attention and feed-forward network) such that it can capture long-range pixel interactions, while still remaining applicable to large images. Our model, named Restoration Transformer (Restormer), achieves state-of-the-art results on several image restoration tasks, including image deraining, single-image motion deblurring, defocus deblurring (single-image and dual-pixel data), and image denoising (Gaussian grayscale/color denoising, and real image denoising).


Network Architecture

Results

Image Deraining comparisons on the Test100, Rain100H, Rain100L, Test1200, and Test2800 testsets.


Single-Image Motion Deblurring results. Our Restormer is trained only on the GoPro dataset and directly applied to the HIDE and RealBlur benchmark datasets.


Defocus Deblurring comparisons on the DPDD testset (containing 37 indoor and 39 outdoor scenes). S: single-image defocus deblurring. D: dual-pixel defocus deblurring.


Gaussian Image Denoising comparisons for two categories of methods. Top super row: learning a single model to handle various noise levels. Bottom super row: training a separate model for each noise level.

Grayscale

Color

Real Image Denoising on SIDD and DND datasets. ∗ denotes methods using additional training data. Our Restormer is trained only on the SIDD images and directly tested on DND.

Citation

If you use Restormer, please consider citing:

@article{Zamir2021Restormer,
    title={Restormer: Efficient Transformer for High-Resolution Image Restoration}, 
    author={Syed Waqas Zamir and Aditya Arora and Salman Khan and Munawar Hayat 
            and Fahad Shahbaz Khan and Ming-Hsuan Yang},
    journal={ArXiv 2111.09881},
    year={2021}
}

Contact

Should you have any question, please contact [email protected]

Issues
  •  denoising training dataset

    denoising training dataset

    well done ! But can you tell me about your denoising-working ,what dataset your used? real training dataset and Gaussian Denoising dataset. Thank you very much!

    opened by 17346604401 3
  • Loss function and supp

    Loss function and supp

    In this paper , I dont find the defination of loss function, could you tell me the loss function of restormer.
    and where can i find the supplement of this paper. thank you

    opened by jiaaihhy 2
  • DeamNet and AINDNet results in Real image denoising

    DeamNet and AINDNet results in Real image denoising

    Hello, I doubt the results showed in real image denoising are wrong. The DeamNet and AINDNet use SIDD benchmark to test. However, in your paper, you utilize the results of the SIDD benchmark to compare the results tested in SIDD validation datasets. MIRNet and MPRNet also can find the same problem. so why not directly compare the results of the Benchmark dataset? Like the DND dataset, the results can be obtained on websites.

    opened by it-hao 2
  • Pretrained model for Defocus Deblurring links to wrong file

    Pretrained model for Defocus Deblurring links to wrong file

    Restormer / Defocus_Deblurring / Pretrained_Models / README.MD contains a link to download the pretrained model for defocus deblurring.

    However, this link appears to lead to the pre-trained model for Deraining. It is the same URL as is found in Restormer / Deraining / Pretrained_Models / README.MD

    I assume this is an error, rather then one being intended to use the same model for both of them. If so, there is currently no pre-trained model available for Defocus Deblurring.

    An updated link would be appreciated.

    opened by Nordskog 2
  • waiting for the code!

    waiting for the code!

    waiting for the code!

    opened by jiaaihhy 1
  • Runtime environment

    Runtime environment

    麻烦问一下这个代码运行的环境是什么,显卡使用情况以及pytorch版本信息

    opened by ConanMusk 1
  • train code

    train code

    hello,deal authors,I want to train a restormer model on myself,but I can't find the training code of restormer ,can you please write a readme document for the traing of restormer model.

    opened by ConanMusk 1
  • Does not correspond to paper MPRNet

    Does not correspond to paper MPRNet

    0V%7W6C)GL~C%I5K10NZ%AM

    Figure 3 in this paper and Figure 5 in the author's previous paper MPRNet use the same test picture, the SAME PSNR, but the visualization results are very different. Whether I can doubt the authenticity and rigor of the results of the paper

    opened by Jeasco 1
  • About  layer normalization code

    About layer normalization code

    There are some discrepancies between the layer normalization that I understand and the implementation method in the code. I hope someone can answer my doubts:

    I understand layer normalization: input X, dimension B C H W, calculate the mean variance of length B for each B, multiply the weight and bias of length B

    Layer normalization in the Restomer code: input X, dimension B C H W, calculate the mean variance of B H*W dimension, multiply the weight and bias of length C

    opened by zzg-tju 1
  • Training code

    Training code

    Your work is so wonderful,could you please provide training scripts?

    opened by swaiwi 1
  • How about the performance on image super-resolution task?

    How about the performance on image super-resolution task?

    Great job! Have you tried working on image super-resolution task and how does it compare to existing work (e.g. SwinIR) ?

    opened by blackcow 0
  • How many GPUS are used when training

    How many GPUS are used when training

    Hi, could tell me the detailed information about the GPUS(GPU type and number) used in the experiment and the loss for motion deblur? I do not find these details in the paper, thanks so much.

    opened by jixiang2016 0
  • colab?

    colab?

    I am pleased with your work; the level of completeness is really professional! Do you guys have any plan to release the code for Google Colab? Unfortunately, I can't run the code on my local machine due to some poor factors.

    opened by osushilover 0
  • some problems

    some problems

    Since no training code is given, I write my own training program to train Restormer. However, at the beginning of the training, I could only set batchsize to 48 due to the limitation of GPUs memory. However, I found that the loss would hardly decrease when the first 10,000 to 20,000 iteration was carried out, which verified that the PSNR remained unchanged at about 26.2. Is the training relatively slow, or what is the problem? And if prob, I would like to know the upward trend of Val PSNR and the downward trend of loss during your training

    opened by jiaaihhy 1
  • GPU

    GPU

    In Gopro deblurring, I want to know How many gpus were used and how long were they trained in [patchsize 128, batchsize 64].

    opened by jiaaihhy 0
Owner
Syed Waqas Zamir
Research Scientist
Syed Waqas Zamir
[CVPR 2021] Official PyTorch Implementation for "Iterative Filter Adaptive Network for Single Image Defocus Deblurring"

IFAN: Iterative Filter Adaptive Network for Single Image Defocus Deblurring Checkout for the demo (GUI/Google Colab)! The GUI version might occasional

Junyong Lee 103 Dec 30, 2021
[ICCV 2021] Official Tensorflow Implementation for "Single Image Defocus Deblurring Using Kernel-Sharing Parallel Atrous Convolutions"

KPAC: Kernel-Sharing Parallel Atrous Convolutional block This repository contains the official Tensorflow implementation of the following paper: Singl

Hyeongseok Son 29 Dec 30, 2021
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 41 Jan 4, 2022
[ICCV 2021] Official Pytorch implementation for Discriminative Region-based Multi-Label Zero-Shot Learning SOTA results on NUS-WIDE and OpenImages

Discriminative Region-based Multi-Label Zero-Shot Learning (ICCV 2021) [arXiv][Project page >> coming soon] Sanath Narayan*, Akshita Gupta*, Salman Kh

Akshita Gupta 41 Jan 4, 2022
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 18 Dec 26, 2021
Selective Wavelet Attention Learning for Single Image Deraining

SWAL Code for Paper "Selective Wavelet Attention Learning for Single Image Deraining" Prerequisites Python 3 PyTorch Models We provide the models trai

Bobo 7 Nov 13, 2021
Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation"

SharinGAN Official repo for the work titled "SharinGAN: Combining Synthetic and Real Data for Unsupervised GeometryEstimation" The official project we

Koutilya PNVR 22 Sep 13, 2021
Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Implementation of Vision Transformer, a simple way to achieve SOTA in vision classification with only a single transformer encoder, in Pytorch

Phil Wang 8.2k Jan 17, 2022
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 24 Nov 24, 2021
Shape Matching of Real 3D Object Data to Synthetic 3D CADs (3DV project @ ETHZ)

Real2CAD-3DV Shape Matching of Real 3D Object Data to Synthetic 3D CADs (3DV project @ ETHZ) Group Member: Yue Pan, Yuanwen Yue, Bingxin Ke, Yujie He

null 19 Nov 21, 2021
[NeurIPS 2020] Official repository for the project "Listening to Sound of Silence for Speech Denoising"

Listening to Sounds of Silence for Speech Denoising Introduction This is the repository of the "Listening to Sounds of Silence for Speech Denoising" p

Henry Xu 36 Jan 13, 2022
Semi-supervised Video Deraining with Dynamical Rain Generator (CVPR, 2021, Pytorch)

S2VD Semi-supervised Video Deraining with Dynamical Rain Generator (CVPR, 2021) Requirements and Dependencies Ubuntu 16.04, cuda 10.0 Python 3.6.10, P

Zongsheng Yue 41 Jan 16, 2022
Offical implementation for "Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation".

Trash or Treasure? An Interactive Dual-Stream Strategy for Single Image Reflection Separation (NeurIPS 2021) by Qiming Hu, Xiaojie Guo. Dependencies P

Qiming Hu 12 Dec 31, 2021
PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning This repository is for EMSRDPN introduced in the foll

null 4 Jan 12, 2022
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 3 Dec 9, 2021
Official implementation of Deep Convolutional Dictionary Learning for Image Denoising.

DCDicL for Image Denoising Hongyi Zheng*, Hongwei Yong*, Lei Zhang, "Deep Convolutional Dictionary Learning for Image Denoising," in CVPR 2021. (* Equ

Z80 57 Jan 7, 2022
Official implementation of SynthTIGER (Synthetic Text Image GEneratoR) ICDAR 2021

?? SynthTIGER: Synthetic Text Image GEneratoR Official implementation of SynthTIGER | Paper | Datasets Moonbin Yim1, Yoonsik Kim1, Han-cheol Cho1, Sun

Clova AI Research 130 Jan 7, 2022