PyTorch implementation of EGVSR: Efficcient & Generic Video Super-Resolution (VSR)

Related tags

Deep Learning real-time super-resolution vsr
Overview

EGVSR-PyTorch

GitHub | Gitee码云


VSR x4: EGVSR; Upscale x4: Bicubic Interpolation

Contents

Introduction

This is a PyTorch implementation of EGVSR: Efficcient & Generic Video Super-Resolution (VSR), using subpixel convolution to optimize the inference speed of TecoGAN VSR model. Please refer to the official implementation ESPCN and TecoGAN for more information.

Features

  • Unified Framework: This repo provides a unified framework for various state-of-the-art DL-based VSR methods, such as VESPCN, SOFVSR, FRVSR, TecoGAN and our EGVSR.
  • Multiple Test Datasets: This repo offers three types of video datasets for testing, i.e., standard test dataset -- Vid4, Tos3 used in TecoGAN and our new dataset -- Gvt72 (selected from Vimeo site and including more scenes).
  • Better Performance: This repo provides model with faster inferencing speed and better overall performance than prior methods. See more details in Benchmarks section.

Dependencies

  • Ubuntu >= 16.04
  • NVIDIA GPU + CUDA & CUDNN
  • Python 3
  • PyTorch >= 1.0.0
  • Python packages: numpy, matplotlib, opencv-python, pyyaml, lmdb (requirements.txt & req.txt)
  • (Optional) Matlab >= R2016b

Datasets

A. Training Dataset

Download the official training dataset based on the instructions in TecoGAN-TensorFlow, rename to VimeoTecoGAN and then place under ./data.

B. Testing Datasets

  • Vid4 -- Four video sequences: city, calendar, foliage and walk;
  • Tos3 -- Three video sequences: bridge, face and room;
  • Gvt72 -- Generic VSR Test Dataset: 72 video sequences (including natural scenery, culture scenery, streetscape scene, life record, sports photography, etc, as shown below)

You can get them at ⏬ 百度网盘 (提取码:8tqc) and put them into 📁 Datasets. The following shows the structure of the above three datasets.

data
  ├─ Vid4
    ├─ GT                # Ground-Truth (GT) video sequences
      └─ calendar
        ├─ 0001.png
        └─ ...
    ├─ Gaussian4xLR      # Low Resolution (LR) video sequences in gaussian degradation and x4 down-sampling
      └─ calendar
        ├─ 0001.png
        └─ ...
  └─ ToS3
    ├─ GT
    └─ Gaussian4xLR
  └─ Gvt72
    ├─ GT
    └─ Gaussian4xLR

Benchmarks

Experimental Environment

Version Info.
System Ubuntu 18.04.5 LTS X86_64
CPU Intel i9-9900 3.10GHz
GPU Nvidia RTX 2080Ti 11GB GDDR6
Memory DDR4 2666 32GB×2

A. Test on Vid4 Dataset


1.LR 2.VESPCN 3.SOFVSR 4.DUF 5.Ours:EGVSR 6.GT
Objective metrics for visual quality evaluation[1]

B. Test on Tos3 Dataset


1.VESPCN 2.SOFVSR 3. FRVSR 4.TecoGAN 5.Ours:EGVSR 6.GT

C. Test on Gvt72 Dataset


1.LR 2.VESPCN 3.SOFVSR 4.DUF 5.Ours:EGVSR 6.GT
Objective metrics for visual quality and temporal coherence evaluation[1]

D. Optical-Flow based Motion Compensation

Please refer to FLOW_walk, FLOW_foliage and FLOW_city.

E. Comprehensive Performance


Comparison of various SOTA VSR model on video quality score and speed performance[3]

[1] ⬇️ :smaller value for better performance, ⬆️ : on the contrary; Red: stands for Top1, Blue: Top2. [2] The calculation formula of video quality score considering both spatial and temporal domain, using lambda1=lambda2=lambda3=1/3. [3] FLOPs & speed are computed on RGB with resolution 960x540 to 3840x2160 (4K) on NVIDIA GeForce GTX 2080Ti GPU.

License & Citations

This EGVSR project is released under the MIT license. See more details in LICENSE. The provided implementation is strictly for academic purposes only. If EGVSR helps your research or work, please consider citing EGVSR. The following is a BibTeX reference:

@misc{thmen2021egvsr,
  author =       {Yanpeng Cao, Chengcheng Wang, Changjun Song, Yongming Tang and He Li},
  title =        {EGVSR},
  howpublished = {\url{https://github.com/Thmen/EGVSR}},
  year =         {2021}
}

Yanpeng Cao, Chengcheng Wang, Changjun Song, Yongming Tang and He Li. EGVSR. https://github.com/Thmen/EGVSR, 2021.

Acknowledgements

This code is built on the following projects. We thank the authors for sharing their codes.

  1. ESPCN
  2. BasicSR
  3. VideoSuperResolution
  4. TecoGAN-PyTorch
Comments
  • how to run the test program

    how to run the test program

    I put some images into dir /data/Vid4/GT for test, but I don't know what's the mean of "data/Vid4/Gaussian4xLR",

    python codes/main.py --exp_dir test --mode test --model EGVSR --opt /opt/EGVSR-master/experiments_BD/EGVSR/001/test.yml

    2021-07-22 11:01:23,123 [INFO]: ======================================== 2021-07-22 11:01:23,123 [INFO]: Testing model: EGVSR_iter420000 2021-07-22 11:01:23,123 [INFO]: ======================================== 2021-07-22 11:01:24,793 [INFO]: Testing on test1: Vid4 Traceback (most recent call last): File "codes/main.py", line 341, in test(opt) File "codes/main.py", line 182, in test test_loader = create_dataloader(opt, dataset_idx=dataset_idx) File "/opt/daizhsh/EGVSR-master/codes/data/init.py", line 58, in create_dataloader dataset = PairedFolderDataset(data_opt, scale=opt['scale']) File "/opt/daizhsh/EGVSR-master/codes/data/paired_folder_dataset.py", line 21, in init lr_keys = sorted(os.listdir(self.lr_seq_dir)) FileNotFoundError: [Errno 2] No such file or directory: 'data/Vid4/Gaussian4xLR'

    Thank you for you reply!

    opened by daizzhisheng 2
  • Your Results in New Super-Resolution Benchmarks

    Your Results in New Super-Resolution Benchmarks

    Hello,

    MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

    Your method achieved 12th place in Super-Resolution for Video Compression Benchmark in 'x265 compression' category. We look forward to your future work!

    We would be grateful for your feedback on our work.

    opened by EvgeneyBogatyrev 0
  • Significant artifact appears at frame still area when using a long frame sequence

    Significant artifact appears at frame still area when using a long frame sequence

    Some artifacts like ringing are getting more and more significant as the sequence moving forwards, and become insufferable at frame about 100 which don't show up in test datasets because of their short length.

    Do you have some advice to solve this problem?

    opened by HahaOptimus 0
  • metric is NaN

    metric is NaN

    I met this problem while I run the test.sh root@Ic75ba579f00d014e0:/hy-tmp/EGVSR# bash test.sh ./experiments_BD/EGVSR/001/ 2022-07-25 15:10:19,482 [INFO]: ======================================== 2022-07-25 15:10:19,482 [INFO]: Testing model: EGVSR_iter420000 2022-07-25 15:10:19,482 [INFO]: ======================================== 2022-07-25 15:10:23,526 [INFO]: Testing on test1: Vid4 2022-07-25 15:10:24,329 [INFO]: Average /usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /usr/local/lib/python3.8/dist-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) 2022-07-25 15:10:24,331 [INFO]: PSNR: nan (x1.0) 2022-07-25 15:10:24,331 [INFO]: LPIPS: nan (x1.0) 2022-07-25 15:10:24,331 [INFO]: tOF: nan (x1.0) 2022-07-25 15:10:24,331 [INFO]: ---------------------------------------- 2022-07-25 15:10:24,331 [INFO]: Testing on test2: ToS3 2022-07-25 15:10:25,094 [INFO]: Average 2022-07-25 15:10:25,095 [INFO]: PSNR: nan (x1.0) 2022-07-25 15:10:25,095 [INFO]: LPIPS: nan (x1.0) 2022-07-25 15:10:25,095 [INFO]: tOF: nan (x1.0) 2022-07-25 15:10:25,095 [INFO]: ---------------------------------------- 2022-07-25 15:10:25,095 [INFO]: Finish testing 2022-07-25 15:10:25,095 [INFO]: ========================================

    So I should do what and I am troubled

    opened by JacoboJin 0
  • bash test.sh

    bash test.sh

    Bash test How should I write the parameters after sh-- exp_ dir ./ experiments_$ {degradation}/${model}/${exp_id}
    --mode test
    --model ${model}
    --opt test. yml
    --gpu_ id ${gpu_id} Can you give me an example? There is no problem with my path. It keeps reminding me that I can't find the file

    opened by JiaChenGe 1
Owner
null
GitHub
[CVPR 2022] Official PyTorch Implementation for "Reference-based Video Super-Resolution Using Multi-Camera Video Triplets"

Reference-based Video Super-Resolution (RefVSR) Official PyTorch Implementation of the CVPR 2022 Paper Project | arXiv | RealMCVSR Dataset This repo c

Junyong Lee 151 Dec 30, 2022
The official pytorch implemention of the CVPR paper "Temporal Modulation Network for Controllable Space-Time Video Super-Resolution".

This is the official PyTorch implementation of TMNet in the CVPR 2021 paper "Temporal Modulation Network for Controllable Space-Time VideoSuper-Resolu

Gang Xu 95 Oct 24, 2022
A PyTorch Reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution

TecoGAN-PyTorch Introduction This is a PyTorch reimplementation of TecoGAN: Temporally Coherent GAN for Video Super-Resolution (VSR). Please refer to

null 165 Dec 17, 2022
Pytorch implementation of our method for high-resolution (e.g. 2048x1024) photorealistic video-to-video translation.

vid2vid Project | YouTube(short) | YouTube(full) | arXiv | Paper(full) Pytorch implementation for high-resolution (e.g., 2048x1024) photorealistic vid

NVIDIA Corporation 8.1k Jan 1, 2023
Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference

RawVSR This repo contains the official codes for our paper: Exploit Camera Raw Data for Video Super-Resolution via Hidden Markov Model Inference Xiaoh

Xiaohong Liu 23 Oct 8, 2022
BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond

BasicVSR BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond Ported from https://github.com/xinntao/BasicSR Dependencie

Holy Wu 8 Jun 7, 2022
BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment

BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment

Holy Wu 35 Jan 1, 2023
EFENet: Reference-based Video Super-Resolution with Enhanced Flow Estimation

EFENet EFENet: Reference-based Video Super-Resolution with Enhanced Flow Estimation Code is a bit messy now. I woud clean up soon. For training the EF

Yaping Zhao 6 Oct 20, 2021
Fast and Context-Aware Framework for Space-Time Video Super-Resolution (VCIP 2021)

Fast and Context-Aware Framework for Space-Time Video Super-Resolution Preparation Dependencies PyTorch 1.2.0 CUDA 10.0 DCNv2 cd model/DCNv2 bash make

Xueheng Zhang 1 Mar 29, 2022
MoCoPnet - Deformable 3D Convolution for Video Super-Resolution

Deformable 3D Convolution for Video Super-Resolution Pytorch implementation of l

Xinyi Ying 28 Dec 15, 2022
Official repository of "BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment"

BasicVSR_PlusPlus (CVPR 2022) [Paper] [Project Page] [Code] This is the official repository for BasicVSR++. Please feel free to raise issue related to

Kelvin C.K. Chan 227 Jan 1, 2023
Super-Fast-Adversarial-Training - A PyTorch Implementation code for developing super fast adversarial training

Super-Fast-Adversarial-Training This is a PyTorch Implementation code for develo

LBK 26 Dec 2, 2022
PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021.

GCResNet PyTorch implementation of Graph Convolutional Networks in Feature Space for Image Deblurring and Super-resolution, IJCNN 2021. The code will

null 11 May 19, 2022
Implementation of paper: "Image Super-Resolution Using Dense Skip Connections" in PyTorch

SRDenseNet-pytorch Implementation of paper: "Image Super-Resolution Using Dense Skip Connections" in PyTorch (http://openaccess.thecvf.com/content_ICC

wxy 114 Nov 26, 2022
Pytorch implementation of Deep Recursive Residual Network for Super Resolution (DRRN)

DRRN-pytorch This is an unofficial implementation of "Deep Recursive Residual Network for Super Resolution (DRRN)", CVPR 2017 in Pytorch. [Paper] You

yun_yang 192 Dec 12, 2022
PyTorch implementation of 1712.06087 "Zero-Shot" Super-Resolution using Deep Internal Learning

Unofficial PyTorch implementation of "Zero-Shot" Super-Resolution using Deep Internal Learning Unofficial Implementation of 1712.06087 "Zero-Shot" Sup

Jacob Gildenblat 196 Nov 27, 2022
pytorch implementation for Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network arXiv:1609.04802

PyTorch SRResNet Implementation of Paper: "Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network"(https://arxiv.org/abs

Jiu XU 436 Jan 9, 2023
Unoffical implementation about Image Super-Resolution via Iterative Refinement by Pytorch

Image Super-Resolution via Iterative Refinement Paper | Project Brief This is a unoffical implementation about Image Super-Resolution via Iterative Re

LiangWei Jiang 2.5k Jan 2, 2023
PyTorch Implementation of "Light Field Image Super-Resolution with Transformers"

LFT PyTorch implementation of "Light Field Image Super-Resolution with Transformers", arXiv 2021. [pdf]. Contributions: We make the first attempt to a

Squidward 62 Nov 28, 2022