MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution (CVPR2021)

Overview

MASA-SR

Official PyTorch implementation of our CVPR2021 paper MASA-SR: Matching Acceleration and Spatial Adaptation for Reference-Based Image Super-Resolution

Dependencies

  • python 3
  • pytorch >= 1.1.0
  • torchvision >= 0.4.0

Prepare Dataset

  1. Download CUFED train set and CUFED test set
  2. Place the datasets in this structure:
    CUFED
    ├── train
    │   ├── input
    │   └── ref 
    └── test
        └── CUFED5  
    

Get Started

  1. Clone this repo
    git clone https://github.com/Jia-Research-Lab/MASA-SR.git
    cd MASA-SR
    
  2. Download the dataset. Modify the argument --data_root in test.py and train.py according to your data path.

Evaluation

  1. Download the pre-trained models and place them into the pretrained_weights/ folder

    • Pre-trained models can be downloaded from Google Drive
      • masa_rec.pth: trained with only reconstruction loss
      • masa.pth: trained with all losses
  2. Run test.sh. See more details in test.sh (if you are using cpu, please add --gpu_ids -1 in the command)

    sh test.sh
    
  3. The testing results are in the test_results/ folder

Training

  1. First train masa-rec only with the reconstruction loss.
    python train.py --use_tb_logger --data_augmentation --max_iter 160 --loss_l1 --name train_masa_rec
    
  2. After getting masa-rec, train masa with all losses, which is based on the pretrained masa-rec.
    python train.py --use_tb_logger --max_iter 50 --loss_l1 --loss_adv --loss_perceptual --name train_masa_gan --resume ./weights/train_masa_rec/snapshot/net_best.pth --resume_optim ./weights/train_masa_rec/snapshot/optimizer_G_best.pth --resume_scheduler ./weights/train_masa_rec/snapshot/scheduler_best.pth
    
  3. The training results are in the weights/ folder
Issues
  • How to train on Sun80  or Urban100?

    How to train on Sun80 or Urban100?

    This is my first time learning RefSR. Sun80 or Urban100 sounds like SISR dataset. May I know how to train RefSR code on them? Thanks!

    opened by Lkinyuu 18
  • CUDA out of memory

    CUDA out of memory

    CUDA out of memory! How much memory does test need(for CUFED5)? Looking forward to your reply!

    opened by hbw945 2
  • Question

    Question

    Hi. I'm confused that why the test results are not ideal as I get started and train by following the readme tutorial. 000_0 003_0 Thanks!

    opened by unnc-wqw 2
  • Quite bad results on the linked test data ?

    Quite bad results on the linked test data ?

    Hi!

    First of all I have to say that I am quite impressed by the possibility your current work has shown. However, after downloading the test data that you linked on your repo (CUFED5) and running the shell script for testing it using both the pre-trained models made available , I have been quite disappointed by the accompanying results :

    image

    (the reference images have been scaled to just fit into the presentation but I think they are of 500x332 resolution)

    I have not looked too much into the test script that you have provided but I did look at the architecture you proposed in the paper and here are some observations/comments/questions I have:

    1. shouldn't the Reference Image Resolution be 4 times the LR ? (this does not seem to be the case with the data provided)

    2. currently upon running the shell script for test, the Resolution seems to be the same as the input, so is the shared implementation more like detail enhancement rather than SR ?

    3. The input and Reference look to be of the same quality, so technically I would not be expecting massive improvements in the SR result, however if you could do something like DSLR based reference on a phones camera, it would be quite novel and interesting experiment. (This is the goal I hope to achieve using your methodology)

    4. More theoretically, what kind of results would you expect when the input and reference are completely mismatched ?

    Extremely sorry for the many naive questions that I might have asked but I hope to hear your views on the same!

    opened by mylifeasazucchini 1
  • Code Optimization

    Code Optimization

    Hi, I found this part of code can be optimized. When idx_x1.size(0) is a big number, the loop takes a lot of time. https://github.com/dvlab-research/MASA-SR/blob/9f0cccb71beafa764dadc5984fc8751cc0740c79/models/archs/MASA_arch.py#L343-L350 And here is my reimplementation

    ind_y = idx_y1.view(-1, 1).repeat(1, diameter_x * s).view(-1)
    ind_x = idx_x1.unsqueeze(-1).repeat(1, 1, diameter_y * s).permute(0, 2, 1).contiguous().view(-1)
    
    opened by Wardwarf-Li 1
  • Question of the downsample of encoder

    Question of the downsample of encoder

    Dear author. The paper says that "The encoder consists of three building blocks – the second and third blocks halve the size of the feature maps with stride 2". However, the code's param "n_blks = [4, 4, 4]" in class MASA indicates that the second and third blocks have 2^4 times downsampling by stride 2, which is inconsistent with the paper. So, how to explain that?

    opened by tiger990111 1
  • some questions

    some questions

    null

    opened by zhh99 0
  • hello

    hello

    可以提供下测试集Sun80吗 找了很多没找到全的,谢谢

    opened by beiyu555 0
  • 请教

    请教

    为什么不直接LR和Ref进行匹配?

    opened by beiyu555 1
  • 有一些代码问题请教你

    有一些代码问题请教你

    image 请问这个代码中Search_org 和 Search 分别是代表在patch搜索和在block搜索的意思吗

    opened by beiyu555 2
  • About PSNR/SSIM in Table 1

    About PSNR/SSIM in Table 1

    I have read a few RefSR papers recently. Why is everyone's comparative effect data the same over the years?

    opened by ximu1211 2
  • 你好

    你好

    你好,我想问问训练是在几张卡上完成的,大概多长时间,谢谢。

    opened by beiyu555 4
  • some question

    some question

    1.stage1中的LR与Ref↓大小不应该都是40x40吗,为什么图中画的不一样大 2.后两层索引图Dk的尺寸小于x2,x4参考特征的大小,怎么根据Dk取这些尺寸不一致的参考特征中取特征 3.dilation=1就是patch=3x3吗 4。LR先变成LR_down,再变回LR进行后续处理的意义是什么

    opened by zhh99 10
  • cuda out of memory

    cuda out of memory

    test时12G显存不够用,报错内存不够,训练才占8G,是什么原因

    opened by zhh99 1
  • PSNR of CUFED ?

    PSNR of CUFED ?

    train as python train.py --use_tb_logger --data_augmentation --max_iter 250 --loss_l1 --name train_masa_rec

    but the best PSNR of CUFED only 27.31

    opened by Lkinyuu 6
  • How many epochs were performed before the testing phase mentioned in the paper?

    How many epochs were performed before the testing phase mentioned in the paper?

    Is 250 L1 and 50 rec the same as on github?

    opened by ximu1211 1
  • Can be applied to the field of image repair/denoising?

    Can be applied to the field of image repair/denoising?

    Thank you very much for your work. This lightweight network structure is really great, but when I actually use my own data, I find that its training loss does not decrease much, and the output visual image shows that the model has been repaired to some extent, but there are still some deficiencies in texture. In addition, I hope it can be applied to the field of repair or noise reduction. I wonder if you have done relevant experiments, and I hope you can tell me whether it is feasible

    opened by 834799106 1
Owner
DV Lab
Deep Vision Lab
DV Lab
PyTorch code for our paper "Image Super-Resolution with Non-Local Sparse Attention" (CVPR2021).

Image Super-Resolution with Non-Local Sparse Attention This repository is for NLSN introduced in the following paper "Image Super-Resolution with Non-

null 89 Jan 11, 2022
EFENet: Reference-based Video Super-Resolution with Enhanced Flow Estimation

EFENet EFENet: Reference-based Video Super-Resolution with Enhanced Flow Estimation Code is a bit messy now. I woud clean up soon. For training the EF

Yaping Zhao 6 Oct 20, 2021
(CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic

ClassSR (CVPR2021) ClassSR: A General Framework to Accelerate Super-Resolution Networks by Data Characteristic Paper Authors: Xiangtao Kong, Hengyuan

Xiangtao Kong 242 Jan 12, 2022
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 10 Nov 15, 2021
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 118 Jan 17, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 121 Jan 19, 2022
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 94 Jan 5, 2022
A Python implementation of the Locality Preserving Matching (LPM) method for pruning outliers in image matching.

LPM_Python A Python implementation of the Locality Preserving Matching (LPM) method for pruning outliers in image matching. The code is established ac

AoxiangFan 8 Sep 8, 2021
PyTorch code for the paper "Curriculum Graph Co-Teaching for Multi-target Domain Adaptation" (CVPR2021)

PyTorch code for the paper "Curriculum Graph Co-Teaching for Multi-target Domain Adaptation" (CVPR2021) This repo presents PyTorch implementation of M

Evgeny 61 Dec 2, 2021
Code for CVPR2021 "Visualizing Adapted Knowledge in Domain Transfer". Visualization for domain adaptation. #explainable-ai

Visualizing Adapted Knowledge in Domain Transfer @inproceedings{hou2021visualizing, title={Visualizing Adapted Knowledge in Domain Transfer}, auth

Yunzhong Hou 56 Jan 14, 2022
(CVPR2021) DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation

DANNet: A One-Stage Domain Adaptation Network for Unsupervised Nighttime Semantic Segmentation CVPR2021(oral) [arxiv] Requirements python3.7 pytorch==

W-zx-Y 54 Jan 17, 2022
[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation

[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation [Paper] Prerequisites To install requirements: pip install -r requirements.txt

Guangrui Li 57 Jan 18, 2022
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching(CVPR2021)

CFNet(CVPR 2021) This is the implementation of the paper CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching, CVPR 2021, Zhelun Shen, Yuch

null 72 Jan 1, 2022
Adversarial Adaptation with Distillation for BERT Unsupervised Domain Adaptation

Knowledge Distillation for BERT Unsupervised Domain Adaptation Official PyTorch implementation | Paper Abstract A pre-trained language model, BERT, ha

Minho Ryu 26 Oct 18, 2021
A framework for joint super-resolution and image synthesis, without requiring real training data

SynthSR This repository contains code to train a Convolutional Neural Network (CNN) for Super-resolution (SR), or joint SR and data synthesis. The met

null 66 Dec 16, 2021
PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021.

GCResNet PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021. The code will

null 8 Jan 13, 2022