Repository for "Exploring Sparsity in Image Super-Resolution for Efficient Inference", CVPR 2021

Related tags

Deep Learning SMSR
Overview

SMSR

Reposity for "Exploring Sparsity in Image Super-Resolution for Efficient Inference"

[arXiv]

Highlights

  • Locate and skip redundant computation in SR networks at a fine-grained level for efficient inference.
  • Maintain state-of-the-art performance with significant FLOPs reduction and a speedup on mobile devices.
  • Efficient implementation of sparse convolution based on original Pytorch APIs for easier migration and deployment.

Network Architecture

Implementation of Sparse Convolution

For easier migration and deployment, we use an efficient implementation of sparse convolution based on original Pytorch APIs rather than the commonly applied CUDA-based implementation. Specifically, sparse features are first extracted from the input, as shown in the following figure. Then, matrix multiplication is executed to produce the output features.

Requirements

  • Python 3.6
  • PyTorch == 1.1.0
  • numpy
  • skimage
  • imageio
  • matplotlib
  • cv2

Train

Prepare training data

  1. Download DIV2K training data (800 training + 100 validtion images) from DIV2K dataset or SNU_CVLab.

  2. Specify '--dir_data' based on the HR and LR images path. In option.py, '--ext' is set as 'sep_reset', which first convert .png to .npy. If all the training images (.png) are converted to .npy files, then set '--ext sep' to skip converting files.

For more informaiton, please refer to EDSR(PyTorch).

Begin to train

python main.py --model SMSR --save SMSR_X2 --scale 2 --patch_size 96 --batch_size 16

Test

Prepare test data

Download benchmark datasets (e.g., Set5, Set14 and other test sets) and prepare HR/LR images in testsets/benchmark following the example of testsets/benchmark/Set5.

Demo

python main.py --dir_data testsets --data_test Set5 --scale 2 --model SMSR --save SMSR_X2 --pre_train experiment/SMSR_X2/model/model_1000.pt --test_only --save_results

Results

Visualization of Sparse Masks

Citation

@InProceedings{Wang2020Exploring,
  author    = {Wang, Longguang and Dong, Xiaoyu and Wang, Yingqian and Ying, Xinyi and Lin, Zaiping and An, Wei and Guo, Yulan},
  title     = {Exploring Sparsity in Image Super-Resolution for Efficient Inference},
  booktitle = {CVPR},
  year      = {2021},
}

Acknowledgements

This code is built on EDSR (PyTorch). We thank the authors for sharing the codes.

Comments
  • question about train/test target

    question about train/test target

    thanks for your work, it is interesting! I have some question about train & test(figure 5 in paper).

    1. In paper, the figure.5 looks like a (0 or 1) mask in training. But I find the code (mask module) it just like a attention module(pixel attention and channel attention)
    2. Because question 1), the test target and train target are different. In training, the pixel and channel just times a weight (not zero through mask module), But in test, it skip the computation(times zero or one).
    3. The channel mask will change through input sample (in paper) , But the code looks like it is a invariable after trained. So it is just a channel prune?https://github.com/The-Learning-And-Vision-Atelier-LAVA/SMSR/blob/daac49c9a107778c95e11a16fd5b4a8b45513678/model/smsr.py#L58
    opened by wangqiim 2
  • difference with paper

    difference with paper

    https://github.com/LongguangWang/SMSR/blob/af9b04ceb5e6820f5df5e372f58f66bb8f85d024/model/smsr.py#L253

    I suggest that the subscript of ch_mask may be [:,:,i-1:i,:1]

    opened by nicetomeetu21 2
  • The model uses image patch as input, how to calculate the PSNR value of the whole image

    The model uses image patch as input, how to calculate the PSNR value of the whole image

    Hi, I'm a beginner of image super resolution, thank you for your great job, which has taught me a lot. I have a problem with calculating the PSNR value of the whole image, I have a question about calculating the PSNR value of the whole image. Do you calculate the PSNR value of each patch in an image first and then average it to get the PSNR value of the whole image，and how to deal with the problem that the image size can't divide the crop size?

    opened by GZ-T 2
  • Export onnx error

    Export onnx error

    Dear all,

    Hi!

    I tried to export the onnx model using torch.onnx.export() but it failed. The error message is as follows：

    TyperError: state_dict() takes 1 positional argument but 3 were given

    Does anyone know what's going on? Can anyone share the experience of successfully exporting the onnx model?

    Thank U all！

    Kind Regards

    opened by haikunzhang95 1
  • IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

    IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed

    hi, I tried to train the SMSR with my data but i got this error. IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed can you help me to solve it :)

    opened by afefbnsaid 2
  • ZeroDivisionError: integer division or modulo by zero

    ZeroDivisionError: integer division or modulo by zero

    File "/home/SR_code/SMSR-master/data/multiscalesrdata.py", line 86, in init self.repeat = args.test_every // (len(self.images_hr) // args.batch_size) ZeroDivisionError: integer division or modulo by zero

    I met this error, can you help me？

    opened by YCHang686 1
Owner
Longguang Wang
Longguang Wang
GitHub
[CVPR 21] Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.

Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting, CVPR 2021. Ayan Kumar Bhunia, Pinaki nath Chowdhury, Yongxin Yan

Ayan Kumar Bhunia 44 Dec 12, 2022
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 328 Dec 17, 2022
Repository of our paper 'Refer-it-in-RGBD' in CVPR 2021

Refer-it-in-RGBD This is the repository of our paper 'Refer-it-in-RGBD: A Bottom-up Approach for 3D Visual Grounding in RGBD Images' in CVPR 2021 Pape

Haolin Liu 34 Nov 7, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 68 Dec 6, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 83 Dec 14, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 38 Dec 27, 2022
This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effects in Video."

Omnimatte in PyTorch This repository contains a re-implementation of the code for the CVPR 2021 paper "Omnimatte: Associating Objects and Their Effect

Erika Lu 728 Dec 28, 2022
This repository is the offical Pytorch implementation of ContextPose: Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021).

Context Modeling in 3D Human Pose Estimation: A Unified Perspective (CVPR 2021) Introduction This repository is the offical Pytorch implementation of

null 37 Nov 21, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

TUCH This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright License fo

Lea Müller 45 Jan 7, 2023
This repository contains the code for the CVPR 2021 paper "GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields"

GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields Project Page | Paper | Supplementary | Video | Slides | Blog | Talk If

null 1.1k Dec 30, 2022
[CVPR 2022] CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation

CoTTA Code for our CVPR 2022 paper Continual Test-Time Domain Adaptation Prerequisite Please create and activate the following conda envrionment. To r

Qin Wang 87 Jan 8, 2023
Official repository for Few-shot Image Generation via Cross-domain Correspondence (CVPR '21)

Few-shot Image Generation via Cross-domain Correspondence Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, Richard Zh

Utkarsh Ojha 251 Dec 11, 2022
Official repository for HOTR: End-to-End Human-Object Interaction Detection with Transformers (CVPR'21, Oral Presentation)

Official PyTorch Implementation for HOTR: End-to-End Human-Object Interaction Detection with Transformers (CVPR'2021, Oral Presentation) HOTR: End-to-

Kakao Brain 114 Nov 28, 2022
This repository contains the code for the CVPR 2020 paper "Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision"

Differentiable Volumetric Rendering Paper | Supplementary | Spotlight Video | Blog Entry | Presentation | Interactive Slides | Project Page This repos

null 697 Jan 6, 2023
This is the official implementation code repository of Underwater Light Field Retention : Neural Rendering for Underwater Imaging (Accepted by CVPR Workshop2022 NTIRE)

Underwater Light Field Retention : Neural Rendering for Underwater Imaging (UWNR) (Accepted by CVPR Workshop2022 NTIRE) Authors: Tian Ye†, Sixiang Che

jmucsx 17 Dec 14, 2022
Repository for "Toward Practical Monocular Indoor Depth Estimation" (CVPR 2022)

Toward Practical Monocular Indoor Depth Estimation Cho-Ying Wu, Jialiang Wang, Michael Hall, Ulrich Neumann, Shuochen Su [arXiv] [project site] DistDe

Meta Research 122 Dec 13, 2022
Official repository for the paper "Self-Supervised Models are Continual Learners" (CVPR 2022)

Self-Supervised Models are Continual Learners This is the official repository for the paper: Self-Supervised Models are Continual Learners Enrico Fini

Enrico Fini 73 Dec 18, 2022
This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametric Head Model (CVPR 2022)".

HeadNeRF: A Real-time NeRF-based Parametric Head Model This repository contains a pytorch implementation of "HeadNeRF: A Real-time NeRF-based Parametr

null 294 Jan 1, 2023
CVPR 2021 Challenge on Super-Resolution Space

Learning the Super-Resolution Space Challenge NTIRE 2021 at CVPR Learning the Super-Resolution Space challenge is held as a part of the 6th edition of

andreas 104 Oct 26, 2022