pytorch implementation for Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network arXiv:1609.04802

Overview

PyTorch SRResNet

Implementation of Paper: "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network"(https://arxiv.org/abs/1609.04802) in PyTorch

Usage

Training

usage: main_srresnet.py [-h] [--batchSize BATCHSIZE] [--nEpochs NEPOCHS]
                        [--lr LR] [--step STEP] [--cuda] [--resume RESUME]
                        [--start-epoch START_EPOCH] [--threads THREADS]
                        [--pretrained PRETRAINED] [--vgg_loss] [--gpus GPUS]

optional arguments:
  -h, --help            show this help message and exit
  --batchSize BATCHSIZE
                        training batch size
  --nEpochs NEPOCHS     number of epochs to train for
  --lr LR               Learning Rate. Default=1e-4
  --step STEP           Sets the learning rate to the initial LR decayed by
                        momentum every n epochs, Default: n=500
  --cuda                Use cuda?
  --resume RESUME       Path to checkpoint (default: none)
  --start-epoch START_EPOCH
                        Manual epoch number (useful on restarts)
  --threads THREADS     Number of threads for data loader to use, Default: 1
  --pretrained PRETRAINED
                        path to pretrained model (default: none)
  --vgg_loss            Use content loss?
  --gpus GPUS           gpu ids (default: 0)

An example of training usage is shown as follows:

python main_srresnet.py --cuda --vgg_loss --gpus 0

demo

usage: demo.py [-h] [--cuda] [--model MODEL] [--image IMAGE]
               [--dataset DATASET] [--scale SCALE] [--gpus GPUS]

optional arguments:
  -h, --help         show this help message and exit
  --cuda             use cuda?
  --model MODEL      model path
  --image IMAGE      image name
  --dataset DATASET  dataset name
  --scale SCALE      scale factor, Default: 4
  --gpus GPUS        gpu ids (default: 0)

We convert Set5 test set images to mat format using Matlab, for simple image reading An example of usage is shown as follows:

python demo.py --model model/model_srresnet.pth --dataset Set5 --image butterfly_GT --scale 4 --cuda

Eval

usage: eval.py [-h] [--cuda] [--model MODEL] [--dataset DATASET]
               [--scale SCALE] [--gpus GPUS]

optional arguments:
  -h, --help         show this help message and exit
  --cuda             use cuda?
  --model MODEL      model path
  --dataset DATASET  dataset name, Default: Set5
  --scale SCALE      scale factor, Default: 4
  --gpus GPUS        gpu ids (default: 0)

We convert Set5 test set images to mat format using Matlab. Since PSNR is evaluated on only Y channel, we import matlab in python, and use rgb2ycbcr function for converting rgb image to ycbcr image. You will have to setup the matlab python interface so as to import matlab library. An example of usage is shown as follows:

python eval.py --model model/model_srresnet.pth --dataset Set5 --cuda

Prepare Training dataset

  • Please refer Code for Data Generation for creating training files.
  • Data augmentations including flipping, rotation, downsizing are adopted.

Performance

  • We provide a pretrained model trained on 291 images with data augmentation
  • Instance Normalization is applied instead of Batch Normalization for better performance
  • So far performance in PSNR is not as good as paper, any suggestion is welcome
Dataset SRResNet Paper SRResNet PyTorch
Set5 32.05 31.80
Set14 28.49 28.25
BSD100 27.58 27.51

Result

From left to right are ground truth, bicubic and SRResNet

Comments
  • Training Error

    Training Error

    When I run the python main.py --cuda --batchSize 4, I got this error.

    Namespace(batchSize=4, clip=0.1, cuda=True, lr=0.0001, momentum=0.9, nEpochs=500, pretrained='', resume='', start_epoch=1, step=500, threads=1, weight_decay=0)
    ('Random Seed: ', 4270)
    ===> Loading datasets
    ===> Building model
    ===> Setting GPU
    ===> Setting Optimizer
    ===> Training
    epoch = 1 lr = 0.0001
    Traceback (most recent call last):
      File "main.py", line 130, in <module>
        main()
      File "main.py", line 82, in main
        train(training_data_loader, optimizer, model, criterion, epoch)
      File "main.py", line 108, in train
        loss = criterion(model(input), target)
      File "/home/shuai/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/shuai/SRResNet/srresnet.py", line 65, in forward
        out = self.relu(self.conv_input(x))
      File "/home/shuai/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 206, in __call__
        result = self.forward(*input, **kwargs)
      File "/home/shuai/.local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 237, in forward
        self.padding, self.dilation, self.groups)
      File "/home/shuai/.local/lib/python2.7/site-packages/torch/nn/functional.py", line 40, in conv2d
        return f(input, weight, bias)
    RuntimeError: CUDNN_STATUS_BAD_PARAM
    

    I generated the trianing data with generate_train.m from VDSR.

    opened by yuanshuai220 3
  • Problem in 3-channeled image output

    Problem in 3-channeled image output

    Hi, When i'm displaying my 3 channeled output image, it is not at all good and looks something like this:- rgb

    When I'm displaying my single channeled output ,it still shows proper image features ,like this:- singlechannel

    Though I've started to train my model and it isn't even 50 epochs till now, but I really want to know why is this happening.

    Thank you .

    opened by sonal-bansal 2
  • How can I test in my own .mat files?

    How can I test in my own .mat files?

    I have converted my images to .mat files, later I realised that my files are not same as the testsets provided in the repository. Did anybody faced similar situation as mine?

    opened by surajitsaikia27 2
  • AttributeError: 'NoneType' object has no attribute 'shape'

    AttributeError: 'NoneType' object has no attribute 'shape'

    Dear all, I encounter this problem at the start of my training. Could anyone help? The error message is as follows:

    user@userVM:~/\u4e0b\u8f7d/SRResNet$ python main_srresnet.py --cuda --vgg_loss --gpus 0 /home/user/anaconda3/lib/python3.6/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Namespace(batchSize=16, cuda=True, gpus='0', lr=0.0001, nEpochs=500, pretrained='', resume='', start_epoch=1, step=200, threads=0, vgg_loss=True) => use gpu id: '0' Random Seed: 3780 ===> Loading datasets ===> Loading VGG model ===> Building model /home/user/anaconda3/lib/python3.6/site-packages/torch/nn/functional.py:52: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead. warnings.warn(warning.format(ret)) ===> Setting GPU ===> Setting Optimizer ===> Training Epoch=1, lr=0.0001 Traceback (most recent call last): File "main_srresnet.py", line 166, in main() File "main_srresnet.py", line 104, in main train(training_data_loader, optimizer, model, criterion, epoch) File "main_srresnet.py", line 122, in train for iteration, batch in enumerate(training_data_loader, 1): File "/home/user/anaconda3/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 313, in next indices = next(self.sample_iter) # may raise StopIteration File "/home/user/anaconda3/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 138, in iter for idx in self.sampler: File "/home/user/anaconda3/lib/python3.6/site-packages/torch/utils/data/sampler.py", line 51, in iter return iter(torch.randperm(len(self.data_source)).tolist()) File "/home/user/\u4e0b\u8f7d/SRResNet/dataset.py", line 16, in len return self.data.shape[0] AttributeError: 'NoneType' object has no attribute 'shape'

    The generate_train_srresnet.m script is extremely slow. To speed up, I fix downsizes to 0.5 and generate a srresnet_x4.h5 (2.3GB). I checked the subim_input and subim_input, which seem valid. But I just do not understand why this happens. Could anyone help me with this? Thanks.

    opened by niuqun 1
  • UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 918: ordinal not in range(128)

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 918: ordinal not in range(128)

    UnicodeDecodeError Traceback (most recent call last) in () ----> 1 weights = torch.load('Saved_model//model_srresnet.pth') 2 model.load_state_dict(weights['model'].state_dict())

    ~\Anaconda3\lib\site-packages\torch\serialization.py in load(f, map_location, pickle_module) 301 f = open(f, 'rb') 302 try: --> 303 return _load(f, map_location, pickle_module) 304 finally: 305 if new_fd:

    ~\Anaconda3\lib\site-packages\torch\serialization.py in _load(f, map_location, pickle_module) 467 unpickler = pickle_module.Unpickler(f) 468 unpickler.persistent_load = persistent_load --> 469 result = unpickler.load() 470 471 deserialized_storage_keys = pickle_module.load(f)

    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 918: ordinal not in range(128)

    I've also tried your previous answers of changing encoding type in serialization.py as 'latin1' , it didn't work.

    opened by sonal-bansal 0
  • test erro

    test erro

    @twtygqyy ,Hi,I run python eval.py with pytorch0.4 and python3.5 and get erro like this: Traceback (most recent call last): File "/home/hpl/code/2018/Super_Resolution/SRResNet/pytorch-SRResNet-master_imaginput_twosatge/eval.py", line 42, in model = torch.load(opt.model)["model"] File "/home/hpl/anaconda2/envs/pytorch0.3_p3.5/lib/python3.5/site-packages/torch/serialization.py", line 229, in load return _load(f, map_location, pickle_module) File "/home/hpl/anaconda2/envs/pytorch0.3_p3.5/lib/python3.5/site-packages/torch/serialization.py", line 377, in _load result = unpickler.load() AttributeError: Can't get attribute '_rebuild_tensor_v2' on <module 'torch._utils' from '/home/hpl/anaconda2/envs/pytorch0.3_p3.5/lib/python3.5/site-packages/torch/_utils.py'>

    opened by HPL123 0
  • LeakyReLU?

    LeakyReLU?

    Regarding your SRResNet implementation (the generator), I was wondering why you used a LeakyReLU instead of a PReLU with a trainable "a". Reading the paper and seing your implementation raised this question, because the two are related, but a bit different. It seems to me that the author meant a trainable PReLU in the original implementation.

    Also, why did you use InstanceNorm2d instead of BatchNorm2d? Were you feeding the network with single images instead of a 16 sized batch as the paper specifies?

    My last question would be what dataset you used to train your implementation? I've seen you used data augmentation, but I didn't find info on the used dataset.

    I'm studying the paper and planning on running it on Imagenet, as the author did, expecting to get similar results. I also will definetly take a look on your evaluation code implementation, the Matlab tip, which increases PSNR values, seems valuable.

    My implementation is a bit different than yours and it's getting 29,75dB PSNR on Set5. To get these results I trained it with a sampled 5500 images from Imagenet and I'm hoping to get an improvement when I train it with the 350k images they mention in the paper.

    opened by BrunoVox 0
  • Training Loss

    Training Loss

    Hi , I'm training srresnet on DIV2K from scratch(no pretrained) at lr: e-4 , and it is decayed by 0.1 for step=200, but I'm having MSE loss(at 402 epoch) as:-

    Epoch=402, lr=1.0000000000000002e-06 Epoch [402/500], Step [1/50], Loss: 74.55887,Total_loss_old:74.55887 Epoch [402/500], Step [2/50], Loss : 77.55219,Total_loss_old:152.11106 Epoch [402/500], Step [3/50], Loss: 58.70598,Total_loss_old:210.81704 Epoch [402/500], Step [4/50],Loss: 78.69186,Total_loss_old:289.50890 Epoch [402/500], Step [5/50], Loss: 87.79494,Total_loss_old:377.30384 Epoch [402/500], Step [6/50], Loss: 24.87956,Total_loss_old:402.18340 ..... Overall Total_loss: 72.45043201446533

    I dont know if actually training or not. Please check code once:- code.zip This is my result:- result_images.docx

    opened by sonal-bansal 0
  • Memory issue when Eval

    Memory issue when Eval

    @twtygqyy ,Hi, I trained normally but the test showed memory issue error like this. I don't know why it needs so much memory. RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda bld/pytorch_1501969512886/work/pytorch-0.1.12/torch/lib/THC/generic/THCStorage.cu:66

    opened by HPL123 0
Owner
Jiu XU
Computer Vision Engineering Manager @ Apple
Jiu XU
Official implementation of the paper 'Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution' in CVPR 2022

LDL Paper | Supplementary Material Details or Artifacts: A Locally Discriminative Learning Approach to Realistic Image Super-Resolution Jie Liang*, Hu

null 150 Dec 26, 2022
Super-Fast-Adversarial-Training - A PyTorch Implementation code for developing super fast adversarial training

Super-Fast-Adversarial-Training This is a PyTorch Implementation code for develo

LBK 26 Dec 2, 2022
PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network"

HAN PyTorch code for our ECCV 2020 paper "Single Image Super-Resolution via a Holistic Attention Network" This repository is for HAN introduced in the

五维空间 140 Nov 23, 2022
【Arxiv】Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution

SANet Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution Dependencies numpy==1.18.5 scikit_image==0.16.2 torchvision==0.8.1 to

null 36 Jan 5, 2023
PyTorch code for 'Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning'

Efficient Single Image Super-Resolution Using Dual Path Connections with Multiple Scale Learning This repository is for EMSRDPN introduced in the foll

null 7 Feb 10, 2022
Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

alias-free-gan-pytorch Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) This implementation

Kim Seonghyeon 502 Jan 3, 2023
PyTorch version of the paper 'Enhanced Deep Residual Networks for Single Image Super-Resolution' (CVPRW 2017)

About PyTorch 1.2.0 Now the master branch supports PyTorch 1.2.0 by default. Due to the serious version problem (especially torch.utils.data.dataloade

Sanghyun Son 2.1k Jan 1, 2023
Practical Single-Image Super-Resolution Using Look-Up Table

Practical Single-Image Super-Resolution Using Look-Up Table [Paper] Dependency Python 3.6 PyTorch glob numpy pillow tqdm tensorboardx 1. Training deep

Younghyun Jo 116 Dec 23, 2022
Torch implementation of "Enhanced Deep Residual Networks for Single Image Super-Resolution"

NTIRE2017 Super-resolution Challenge: SNU_CVLab Introduction This is our project repository for CVPR 2017 Workshop (2nd NTIRE). We, Team SNU_CVLab, (B

Bee Lim 625 Dec 30, 2022
Augmentation for Single-Image-Super-Resolution

SRAugmentation Augmentation for Single-Image-Super-Resolution Implimentation CutBlur Cutout CutMix Cutup CutMixup Blend RGBPermutation Identity OneOf

Yubo 6 Jun 27, 2022
Single Image Super-Resolution (SISR) with SRResNet, EDSR and SRGAN

Single Image Super-Resolution (SISR) with SRResNet, EDSR and SRGAN Introduction Image super-resolution (SR) is the process of recovering high-resoluti

null 8 Apr 15, 2022
PyTorch code for our paper "Attention in Attention Network for Image Super-Resolution"

Under construction... Attention in Attention Network for Image Super-Resolution (A2N) This repository is an PyTorch implementation of the paper "Atten

Haoyu Chen 71 Dec 30, 2022
Official PyTorch code for Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution (MANet, ICCV2021)

Mutual Affine Network for Spatially Variant Kernel Estimation in Blind Image Super-Resolution (MANet, ICCV2021) This repository is the official PyTorc

Jingyun Liang 139 Dec 29, 2022
The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution.

WSRGlow The official implementation of the Interspeech 2021 paper WSRGlow: A Glow-based Waveform Generative Model for Audio Super-Resolution. Audio sa

Kexun Zhang 96 Jan 3, 2023
Implementation of paper: "Image Super-Resolution Using Dense Skip Connections" in PyTorch

SRDenseNet-pytorch Implementation of paper: "Image Super-Resolution Using Dense Skip Connections" in PyTorch (http://openaccess.thecvf.com/content_ICC

wxy 114 Nov 26, 2022
Official implementation of the paper 'Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution'

DASR Paper Efficient and Degradation-Adaptive Network for Real-World Image Super-Resolution Jie Liang, Hui Zeng, and Lei Zhang. In arxiv preprint. Abs

null 81 Dec 28, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 159 Dec 30, 2022
Official PyTorch code for Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021)

Hierarchical Conditional Flow: A Unified Framework for Image Super-Resolution and Image Rescaling (HCFlow, ICCV2021) This repository is the official P

Jingyun Liang 159 Dec 30, 2022
Pytorch implementation of Deep Recursive Residual Network for Super Resolution (DRRN)

DRRN-pytorch This is an unofficial implementation of "Deep Recursive Residual Network for Super Resolution (DRRN)", CVPR 2017 in Pytorch. [Paper] You

yun_yang 192 Dec 12, 2022