Official implementation of the paper 'Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution'

Related tags

Deep Learning DASR
Overview

DASR

Paper

Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution
Jie Liang, Hui Zeng, and Lei Zhang.
In arxiv preprint.

Abstract

Efficient and effective real-world image super-resolution (Real-ISR) is a challenging task due to the unknown complex degradation of real-world images and the limited computation resources in practical applications. Recent research on Real-ISR has achieved significant progress by modeling the image degradation space; however, these methods largely rely on heavy backbone networks and they are inflexible to handle images of different degradation levels. In this paper, we propose an efficient and effective degradation-adaptive super-resolution (DASR) network, whose parameters are adaptively specified by estimating the degradation of each input image. Specifically, a tiny regression network is employed to predict the degradation parameters of the input image, while several convolutional experts with the same topology are jointly optimized to specify the network parameters via a non-linear mixture of experts. The joint optimization of multiple experts and the degradation-adaptive pipeline significantly extend the model capacity to handle degradations of various levels, while the inference remains efficient since only one adaptively specified network is used for super-resolving the input image. Our extensive experiments demonstrate that the proposed DASR is not only much more effective than existing methods on handling real-world images with different degradation levels but also efficient for easy deployment.

Overall pipeline of the DASR:

illustration

For more details, please refer to our paper.

Getting started

  • Clone this repo.
git clone https://github.com/csjliang/DASR
cd DASR
  • Install dependencies. (Python 3 + NVIDIA GPU + CUDA. Recommend to use Anaconda)
pip install -r requirements.txt
  • Prepare the training and testing dataset by following this instruction.
  • Prepare the pre-trained models by following this instruction.

Training

First, check and adapt the yml file options/train/DASR/train_DASR.yml, then

  • Single GPU:
PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0 python dasr/train.py -opt options/train/DASR/train_DASR.yml --auto_resume
  • Distributed Training:
YTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --nproc_per_node=4 --master_port=4335 dasr/train.py -opt options/train/DASR/train_DASR.yml --launcher pytorch --auto_resume

Training files (logs, models, training states and visualizations) will be saved in the directory ./experiments/{name}

Testing

First, check and adapt the yml file options/test/DASR/test_DASR.yml, then run:

PYTHONPATH="./:${PYTHONPATH}" CUDA_VISIBLE_DEVICES=0 python basicsr/test.py -opt options/test/DASR/test_DASR.yml

Evaluating files (logs and visualizations) will be saved in the directory ./results/{name}

License

This project is released under the Apache 2.0 license.

Citation

@article{jie2022DASR,
  title={Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution},
  author={Liang, Jie and Zeng, Hui and Zhang, Lei},
  journal={arXiv preprint arXiv:2203.14216},
  year={2022}
}

Acknowledgement

This project is built based on the excellent BasicSR project.

Contact

Should you have any questions, please contact me via [email protected].

Issues
  • error when loading pretrained models

    error when loading pretrained models

    I downloaded the pretrained models as you said, and the file name is "net_g.pth" and "net_p.pth", However, when I tried to load "net_g.pth" using train_DASR.yml, it shows an error as below.

    Traceback (most recent call last): File "./dasr/train.py", line 15, in train_pipeline(root_path) File "/nas/workspace/anse/code/pytorch/SR/DASR/basicsr/train.py", line 128, in train_pipeline model = build_model(opt) File "/nas/workspace/anse/code/pytorch/SR/DASR/basicsr/models/init.py", line 27, in build_model model = MODEL_REGISTRY.get(opt['model_type'])(opt) File "/nas/workspace/anse/code/pytorch/SR/DASR/dasr/models/DASR_model.py", line 20, in init super(DASRModel, self).init(opt) File "/nas/workspace/anse/code/pytorch/SR/DASR/basicsr/models/srgan_dynamic_model.py", line 41, in init self.load_network_init_alldynamic(self.net_g, load_path, self.opt['num_networks'], self.opt['path'].get('strict_load_g', True), load_key) File "/nas/workspace/anse/code/pytorch/SR/DASR/basicsr/models/base_model.py", line 372, in load_network_init_alldynamic load_net = load_net[param_key] KeyError: 'params'

    I think that the pretrained model weights (similar as dictionary?) has no key 'params'. 캡처 So, I add the key 'params', and this code shows another error. 캡처 캡처

    Could you tell me what the problem is?

    opened by anse3832 11
  • TypeError: tuple indices must be integers or slices, not str

    TypeError: tuple indices must be integers or slices, not str

    File "C:\DASR\basicsr\test.py", line 45, in <module> test_pipeline(root_path) File "C:\DASR\basicsr\test.py", line 19, in test_pipeline make_exp_dirs(opt) File "C:\Python39\lib\site-packages\basicsr\utils\dist_util.py", line 80, in wrapper return func(*args, **kwargs) File "C:\Python39\lib\site-packages\basicsr\utils\misc.py", line 40, in make_exp_dirs path_opt = opt['path'].copy() TypeError: tuple indices must be integers or slices, not str

    opened by AIisCool 7
  • why the training is not  convergence

    why the training is not convergence

    i use the train_DASR.yml as you offered, just change two place. 1.training samples is DIV2K. 2.pretrain_network_g is none. and it trained from random init. then i found all of the losses are nan. should i trained it using pretrained model?

    opened by Lvhhhh 4
  • Questions about pretrained MSRResNet

    Questions about pretrained MSRResNet

    Thx for sharing codes! I carefully studied your codes but didnt find the pretrained MSRResNet model (Not trained DASR model). Could you provide a link for it? Also very interested in the training yml of MSRResnet, great thx if you could update it!

    Some minor ques:

    1. I applied similar idea about degradation sub-space and predictor in my sr model, but found it really hard to train a good predictor, the avg L1 regression loss stays around 0.25 (which means the predictor only output a random embedding I think) and stops decreasing. Wonder if you meet similar problem.
    2. I find a "cycle_opt" loss in train_DASR yml, but actually unused in training. Any special meaning?

    Thx again for your work.

    opened by orchidmalevolence 2
  • How to train a model to retain more texture details?

    How to train a model to retain more texture details?

    I'm currently trying to train your model, but I found that when encountering leaves, lawns, sand grains, etc., the model recreates these scenes badly, how can I adjust the training loss to make the model support these scenes, or need to add more such scenes dataset?

    opened by kelisiya 1
  • suggestion for fixing the code to use multi GPU

    suggestion for fixing the code to use multi GPU

    When I tried to use multi GPU, the code shows an error (Unfortunately, I didn't save the error massage. It is related to dimension error)

    So, I fixed the code in DASR/dasr/models/DASR_model.py as below, and it works well. ( multiplying self.opt['num_gpu'] ) image image image

    Please check if my correction is adequate. Thanks!

    opened by anse3832 1
  • Pretrained model correspond to which degradation space subset?

    Pretrained model correspond to which degradation space subset?

    Hi and thanks for sharing your interesting research! My question is related to the pretrained model:

    • the pretrained model correspond to which degradation space? S_1, S_2 or S_3?
    • Or does the pretrained model correspond to the training done for the parameters in the train_DASR.yml file? So all three degradation spaces with the given probability:
    degree_list: ['weak_degrade_one_stage', 'standard_degrade_one_stage', 'severe_degrade_two_stage']
    degree_prob: [0.3, 0.3, 0.4]
    

    Would it be possible to share (if you have done it and if is possible) the pretrained model only for the degradation spaces separately, ie one model for weak_degrade_one_stage, one model for standard_degrade_one_stageand one model for severe_degrade_two_stage? Thanks!

    opened by g-moschetti 1
  • degradation params

    degradation params

    I have two questions.

    1. the sinc kernel_size may be negative when the prob larger than final_sinc_prob? https://github.com/csjliang/DASR/blob/ff2e1ec02c767b75d09b5d60f85c5cbd4115d058/dasr/models/DASR_model.py#L106
    2. why are the previous degradation params overwritten , that is to say, the sinc degradation_params[:, 9:10] is overwritten by the second blur prob? https://github.com/csjliang/DASR/blob/ff2e1ec02c767b75d09b5d60f85c5cbd4115d058/dasr/models/DASR_model.py#L161
    opened by jiamingNo1 1
  • pretrained weights for 2X model

    pretrained weights for 2X model

    The shared link https://drive.google.com/drive/folders/18TuFlx5Fp9W9dDHQ-LyNFae5vakpjGq- contains weights for the 4X model. Can I get access to 2X model weights?

    opened by prasannakdev0 0
  • question about

    question about "User-Interactive Super-resolution"

    In your paper, you mentioned User-Interactive Super-resolution, how can I manually increasing and de-creasing the scale of blur kernel or manually increasing and decreasing the level of noise?

    opened by zack1943 0
  • DF2Kmultiscale+OST_sub

    DF2Kmultiscale+OST_sub

    Hi! I have some questions about DF2Kmultiscale+OST_sub dataset. When i ran the script extract_subimages.py. The script stopped with no errors, with the output being 'All processes done. There are many images' resolution of OST smaller than the crop size 480, how do you deal with these small images?

    opened by Lucien1996 0
[CVPR 2021] Unsupervised Degradation Representation Learning for Blind Super-Resolution

DASR Pytorch implementation of "Unsupervised Degradation Representation Learning for Blind Super-Resolution", CVPR 2021 [arXiv] Overview Requirements

Longguang Wang 277 Jun 29, 2022
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 27 Jun 19, 2022
Real-CUGAN - Real Cascade U-Nets for Anime Image Super Resolution

Real Cascade U-Nets for Anime Image Super Resolution 中文 | English ?? Real-CUGAN

tarsin 64 Jun 18, 2022
Official repository for "Restormer: Efficient Transformer for High-Resolution Image Restoration". SOTA for motion deblurring, image deraining, denoising (Gaussian/real data), and defocus deblurring.

Restormer: Efficient Transformer for High-Resolution Image Restoration Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan,

Syed Waqas Zamir 650 Jun 21, 2022
Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

MEGVII Research 70 Jun 23, 2022
Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

LDL Paper | Supplementary Material Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution Jie Liang*, Hu

null 102 Jun 24, 2022
Official PyTorch code for Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution (MANet, ICCV2021)

Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution (MANet, ICCV2021) This repository is the official PyTorc

Jingyun Liang 118 Jun 25, 2022
The official pytorch implemention of the CVPR paper "Temporal Modulation Network for Controllable Space-Time Video Super-Resolution".

This is the official PyTorch implementation of TMNet in the CVPR 2021 paper "Temporal Modulation Network for Controllable Space-Time VideoSuper-Resolu

Gang Xu 86 Jun 13, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 146 Jun 21, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 145 Jun 11, 2022
Repository for "Exploring Sparsity in Image Super-Resolution for Efficient Inference", CVPR 2021

SMSR Reposity for "Exploring Sparsity in Image Super-Resolution for Efficient Inference" [arXiv] Highlights Locate and skip redundant computation in S

Longguang Wang 206 Jun 27, 2022
PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning This repository is for EMSRDPN introduced in the foll

null 7 Feb 10, 2022
A framework for joint super-resolution and image synthesis, without requiring real training data

SynthSR This repository contains code to train a Convolutional Neural Network (CNN) for Super-resolution (SR), or joint SR and data synthesis. The met

null 69 Apr 28, 2022
PyTorch code for our paper "Attention in Attention Network for Image Super-Resolution"

Under construction... Attention in Attention Network for Image Super-Resolution (A2N) This repository is an PyTorch implementation of the paper "Atten

Haoyu Chen 62 May 13, 2022
PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

HAN PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network" This repository is for HAN introduced in the

五维空间 126 Jun 14, 2022
Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

Computational Photography Lab @ SFU 850 Jun 20, 2022
pytorch implementation for Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network arXiv:1609.04802

PyTorch SRResNet Implementation of Paper: "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network"(https://arxiv.org/abs

Jiu XU 423 Jun 18, 2022
Official implementation of Unfolded Deep Kernel Estimation for Blind Image Super-resolution.

Unfolded Deep Kernel Estimation for Blind Image Super-resolution Hongyi Zheng, Hongwei Yong, Lei Zhang, "Unfolded Deep Kernel Estimation for Blind Ima

Z80 11 May 11, 2022
The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution.

WSRGlow The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution. Audio sa

Kexun Zhang 80 May 24, 2022