Code repo for "Cross-Scale Internal Graph Neural Network for Image Super-Resolution" (NeurIPS'20)

Overview

IGNN

Code repo for "Cross-Scale Internal Graph Neural Network for Image Super-Resolution"   [paper] [supp]

Prepare datasets

1 Download training dataset and test datasets from here.

2 Crop training dataset DIV2K to sub-images.

python ./datasets/prepare_DIV2K_subimages.py

Remember to modify the 'input_folder' and 'save_folder' in the above script.

Dependencies and Installation

The denoising code is tested with Python 3.7, PyTorch 1.1.0 and Cuda 9.0 but is likely to run with newer versions of PyTorch and Cuda.

1 Create conda environment.

conda create --name ignn
conda activate ignn
conda install pytorch=1.1.0 torchvision=0.3.0 cudatoolkit=9.0 -c pytorch

2 Install PyInn.

pip install git+https://github.com/szagoruyko/pyinn.git@master

3 Install matmul_cuda.

bash install.sh

4 Install other dependencies.

pip install -r requirements.txt

Pretrained Models

Downloading the pretrained models from this link and put them into ./ckpt

Training

Use the following command to train the network:

python runner.py
        --gpu [gpu_id]\
        --phase 'train'\
        --scale [2/3/4]\
        --dataroot [dataset root]\
        --out [output path]

Use the following command to resume training the network:

python runner.py 
        --gpu [gpu_id]\
        --phase 'resume'\
        --weights './ckpt/IGNN_x[2/3/4].pth'\
        --scale [2/3/4]\
        --dataroot [dataset root]\
        --out [output path]

You can also use the following simple command with different settings in config.py:

python runner.py

Testing

Use the following command to test the network on benchmark datasets (w/ GT):

python runner.py \
        --gpu [gpu_id]\
        --phase 'test'\
        --weights './ckpt/IGNN_x[2/3/4].pth'\
        --scale [2/3/4]\
        --dataroot [dataset root]\
        --testname [Set5, Set14, BSD100, Urban100, Manga109]\
        --out [output path]

Use the following command to test the network on your demo images (w/o GT):

python runner.py \
        --gpu [gpu_id]\
        --phase 'test'\
        --weights './ckpt/IGNN_x[2/3/4].pth'\
        --scale [2/3/4]\
        --demopath [test folder path]\
        --testname 'Demo'\
        --out [output path]

You can also use the following simple command with different settings in config.py:

python runner.py

Visual Results (x4)

For visual comparison on the 5 benchmarks, you can download our IGNN results from here.

Some examples

image

image

Citation

If you find our work useful for your research, please consider citing the following papers :)

@inproceedings{zhou2020cross,
title={Cross-scale internal graph neural network for image super-resolution},
author={Zhou, Shangchen and Zhang, Jiawei and Zuo, Wangmeng and Loy, Chen Change},
booktitle={Advances in Neural Information Processing Systems},
year={2020}
}

Contact

We are glad to hear from you. If you have any questions, please feel free to contact [email protected].

License

This project is open sourced under MIT license.

Comments
  • Question about cupy

    Question about cupy

    Thank you for the nice work! I can install pyinn&cupy,matmul successfuly,but error when run runner.py. Some of my configure: CUDA version: nvcc -V _nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2017 NVIDIA Corporation Built on Fri_Nov__3_21:07:56_CDT_2017 Cuda compilation tools, release 9.1, V9.1.85 Python 3.7, PyTorch 1.1.0

    Traceback (most recent call last): File "runner.py", line 15, in from core.build import bulid_net File "/home/lixiaohan/workspace/IGNN/core/build.py", line 16, in from models.IGNN import IGNN File "/home/lixiaohan/workspace/IGNN/models/IGNN.py", line 6, in from models.submodules import * File "/home/lixiaohan/workspace/IGNN/models/submodules.py", line 9, in from models.graph_agg import * File "/home/lixiaohan/workspace/IGNN/models/graph_agg.py", line 13, in from models.lib import ops File "/home/lixiaohan/workspace/IGNN/models/lib/ops.py", line 10, in import pyinn File "/home/lixiaohan/.conda/envs/ignn/lib/python3.7/site-packages/pyinn/init.py", line 1, in from .ncrelu import ncrelu File "/home/lixiaohan/.conda/envs/ignn/lib/python3.7/site-packages/pyinn/ncrelu.py", line 2, in from pyinn.utils import Stream, Dtype, load_kernel File "/home/lixiaohan/.conda/envs/ignn/lib/python3.7/site-packages/pyinn/utils.py", line 18, in @cupy.util.memoize(for_each_device=True) AttributeError: module 'cupy' has no attribute 'util'

    opened by MEYE66 4
  • The PSNR is lower than its in the paper

    The PSNR is lower than its in the paper

    Hello, I train the IGNN model as the config in README. But when the training is convergence, I can't get the PSNR as high as its in the paper: | | Set5 | Set14 | B100 | Urban100 | Manga109 | | ---- | ---- | ----- | ---- | -------- | -------- | |IGNN(Paper)|38.24/0.9613|34.07/0.9217|32.41/0.9025|33.23/0.9383|39.35/0.9786| | IGNN(my test) | 38.15/0.9610 | 33.85/0.9199 | 32.30/0.9012 | 32.70/0.9337 | 38.94/0.9776 |

    Are there any other tricks in training? thank you!

    opened by jamekuma 3
  • Loss value appears NAN

    Loss value appears NAN

    Hello, when I use the cross-scale attention module in your code and my own data for training, the Loss value becomes NAN as the number of training iterations increases. I can make sure that there is no NAN data in my data. When I reduce the size of the image for training, the training loss becomes normal. Is it because the cross-scale attention module may cause gradient explosion? How can I improve it?

    opened by qiqiing 1
  • About the supplementary material

    About the supplementary material

    image

    Hello, in your paper, you mentioned: More detailed structures are provided in the supplementary material.Could you tell me how to find these supplementary materials?

    opened by qiqiing 1
  • out of  memory

    out of memory

    Your paper is very good and enlightening to me. When I apply your GraphAgg to my code, it will appear out of memory. Why is this? How should I deal with this problem? thank you!

    opened by PASSENGER128 1
  • About ‘matmul_cuda’

    About ‘matmul_cuda’

    Hello, if my pytorch>1.1.0, how should I use ‘matmul_cuda’? Will there be any performance impact if I don’t use this package? Or is it just an impact on speed? Currently, after installing ‘matmul’, I run the code and report an error: ‘untimeError: CUDA error: no kernel image is available for execution on the device’. How should it be solved?

    opened by qiqiing 0
  • how did you get GraphAgg*

    how did you get GraphAgg*

    When you are proving ‘Effectiveness of Graph Aggregation Module’, how did you get GraphAgg*. I processed it like this: directly compare the input low-resolution image and its down-sampling form in the image domain to obtain idx_k, and then aggregate to obtain z_sr (using the GraphAggregation in your code) ). What is wrong with this approach? The PSNR of my aggregation result is lower than the result obtained by directly performing ‘bicubic’.

    opened by qiqiing 2
  • the question of Nan

    the question of Nan

    Hello, I used your graph construction, graph aggregation module and my own data for training. During the training process, I found that increasing the size of the input data will cause Loss to appear'Nan'. At this time, if the batchsize is set to 1, loss will become normal. I can guarantee that my input data is correct and does not contain the ‘Nan’ value. So what is the reason for this situation, is it because the backpropagation of the imtopatch operation may cause the gradient to explode? Is there any solution?

    opened by qiqiing 0
  • Segmentation fault

    Segmentation fault

    python runner.py \
            --gpu 0\
            --phase 'test'\
            --weights './ckpt/IGNN_x4.pth'\
            --scale 4\
            --demopath [my_path]\
            --testname 'Demo'
    

    By running the command, I got:

    ...
    Using GPUs NUMBER: 1
    [INFO] 2021-03-14 16:35:38.938372 Complete collecting files for TEST. Total test images: 100.
    [DEBUG] 2021-03-14 16:35:41.253575 Parameters in IGNN: 49512553.
    [INFO] 2021-03-14 16:35:44.150203 Recovering from ./ckpt/IGNN_x4.pth ...
    [INFO] 2021-03-14 16:35:44.262738 Recover complete. Current Epoch #0, Best_PSNR = 0 at Epoch #0.
    srun: error: SH-IDC2-172-20-21-49: task 0: Segmentation fault (core dumped)
    

    Why Segmentation fault? I found that this error is related to memory, it may be related to the acceleration of CUDA. Could I run the model on Pytorch only?

    opened by Yshuo-Li 0
Owner
Shangchen Zhou
Ph.D. student at MMLab@NTU.
Shangchen Zhou
The repo contains the code of the ACL2020 paper `Dice Loss for Data-imbalanced NLP Tasks`

Dice Loss for NLP Tasks This repository contains code for Dice Loss for Data-imbalanced NLP Tasks at ACL2020. Setup Install Package Dependencies The c

null 223 Dec 17, 2022
This repo contains the official code of our work SAM-SLR which won the CVPR 2021 Challenge on Large Scale Signer Independent Isolated Sign Language Recognition.

Skeleton Aware Multi-modal Sign Language Recognition By Songyao Jiang, Bin Sun, Lichen Wang, Yue Bai, Kunpeng Li and Yun Fu. Smile Lab @ Northeastern

Isen (Songyao Jiang) 128 Dec 8, 2022
A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python

Mesh-Keys A repo that contains all the mesh keys needed for mesh backend, along with a code example of how to use them in python Have been seeing alot

Joseph 53 Dec 13, 2022
We will release the code of "ConTNet: Why not use convolution and transformer at the same time?" in this repo

ConTNet Introduction ConTNet (Convlution-Tranformer Network) is proposed mainly in response to the following two issues: (1) ConvNets lack a large rec

null 93 Nov 8, 2022
Repo for the Video Person Clustering dataset, and code for the associated paper

Video Person Clustering Repo for the Video Person Clustering dataset, and code for the associated paper. This reporsitory contains the Video Person Cl

Andrew Brown 47 Nov 2, 2022
This repo contains the code and data used in the paper "Wizard of Search Engine: Access to Information Through Conversations with Search Engines"

Wizard of Search Engine: Access to Information Through Conversations with Search Engines by Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zh

null 19 Oct 27, 2022
This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

Dynamic-Vision-Transformer (Pytorch) This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT). Not All Ima

null 210 Dec 18, 2022
This repo will contain code to reproduce and build upon understanding transfer learning

What is being transferred in transfer learning? This repo contains the code for the following paper: Behnam Neyshabur*, Hanie Sedghi*, Chiyuan Zhang*.

null 4 Jun 16, 2021
Code Repo for the ACL21 paper "Common Sense Beyond English: Evaluating and Improving Multilingual LMs for Commonsense Reasoning"

Common Sense Beyond English: Evaluating and Improving Multilingual LMs for Commonsense Reasoning This is the Github repository of our paper, "Common S

INK Lab @ USC 19 Nov 30, 2022
Code repo for realtime multi-person pose estimation in CVPR'17 (Oral)

Realtime Multi-Person Pose Estimation By Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh. Introduction Code repo for winning 2016 MSCOCO Keypoints Cha

Zhe Cao 4.9k Dec 31, 2022
Code repo for EMNLP21 paper "Zero-Shot Information Extraction as a Unified Text-to-Triple Translation"

Zero-Shot Information Extraction as a Unified Text-to-Triple Translation Source code repo for paper Zero-Shot Information Extraction as a Unified Text

cgraywang 88 Dec 31, 2022
This repo includes our code for evaluating and improving transferability in domain generalization (NeurIPS 2021)

Transferability for domain generalization This repo is for evaluating and improving transferability in domain generalization (NeurIPS 2021), based on

gordon 9 Nov 29, 2022
Code repo for "RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network" (Machine Learning and the Physical Sciences workshop in NeurIPS 2021).

RBSRICNN: Raw Burst Super-Resolution through Iterative Convolutional Neural Network An official PyTorch implementation of the RBSRICNN network as desc

Rao Muhammad Umer 6 Nov 14, 2022
This repo is the code release of EMNLP 2021 conference paper "Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories".

Connect-the-Dots: Bridging Semantics between Words and Definitions via Aligning Word Sense Inventories This repo is the code release of EMNLP 2021 con

null 12 Nov 22, 2022
This repo contains the code required to train the multivariate time-series Transformer.

Multi-Variate Time-Series Transformer This repo contains the code required to train the multivariate time-series Transformer. Download the data The No

Gregory Duthé 4 Nov 24, 2022
This is the official source code for SLATE. We provide the code for the model, the training code, and a dataset loader for the 3D Shapes dataset. This code is implemented in Pytorch.

SLATE This is the official source code for SLATE. We provide the code for the model, the training code and a dataset loader for the 3D Shapes dataset.

Gautam Singh 66 Dec 26, 2022
Official Repo for Ground-aware Monocular 3D Object Detection for Autonomous Driving

Visual 3D Detection Package: This repo aims to provide flexible and reproducible visual 3D detection on KITTI dataset. We expect scripts starting from

Yuxuan Liu 305 Dec 19, 2022
AI grand challenge 2020 Repo (Speech Recognition Track)

KorBERT를 활용한 한국어 텍스트 기반 위협 상황인지(2020 인공지능 그랜드 챌린지) 본 프로젝트는 ETRI에서 제공된 한국어 korBERT 모델을 활용하여 폭력 기반 한국어 텍스트를 분류하는 다양한 분류 모델들을 제공합니다. 본 개발자들이 참여한 2020 인공지

Young-Seok Choi 23 Jan 25, 2022
Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

null 105 Dec 23, 2022