Official repository of "BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment"

Overview

BasicVSR_PlusPlus (CVPR 2022)

[Paper] [Project Page] [Code]

This is the official repository for BasicVSR++. Please feel free to raise issue related to BasicVSR++! If you are also interested in RealBasicVSR, which is also accepted to CVPR 2022, please don't hesitate to star!

Authors: Kelvin C.K. Chan, Shangchen Zhou, Xiangyu Xu, Chen Change Loy, Nanyang Technological University

Acknowedgement: Our work is built upon MMEditing. Please follow and star this repository and MMEditing!

News

  • 2 Dec 2021: Colab demo released google colab logo
  • 18 Apr 2022: Code released. Also merged into MMEditing

TODO

  • Add data processing scripts
  • Add checkpoints for deblur and denoise
  • Add configs for deblur and denoise
  • Add Colab demo

Pre-trained Weights

You can find the pre-trained weights for deblurring and denoising in this link. For super-resolution and compressed video enhancement, please refer to MMEditing.

Installation

  1. Install PyTorch
  2. pip install openmim
  3. mim install mmcv-full
  4. git clone https://github.com/ckkelvinchan/BasicVSR_PlusPlus.git
  5. cd BasicVSR_PlusPlus
  6. pip install -v -e .

Inference a Video

  1. Download pre-trained weights
  2. python demo/restoration_video_demo.py ${CONFIG} ${CHKPT} ${IN_PATH} ${OUT_PATH}

For example, you can download the VSR checkpoint here to chkpts/basicvsr_plusplus_reds4.pth, then run

python demo/restoration_video_demo.py configs/basicvsr_plusplus_reds4.py chkpts/basicvsr_plusplus_reds4.pth data/demo_000 results/demo_000

You can also replace ${IN_PATH} ${OUT_PATH} by your video path (e.g., xxx/yyy.mp4) to input/output videos.

Training Models

  1. Put the dataset in the designated locations specified in the configuration file.
  2. sh tools/dist_train.sh ${CONFIG} ${NGPUS}

Data Preprocessing

To be added...

Related Work

Our BasicVSR series:

  1. BasicVSR: The Search for Essential Components in Video Super-Resolution and Beyond, CVPR 2021
  2. Investigating Tradeoffs in Real-World Video Super-Resolution, CVPR 2022

More about deformable alignment:

Citations

@inproceedings{chan2022basicvsrpp,
  author = {Chan, Kelvin C.K. and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change},
  title = {{BasicVSR++}: Improving video super-resolution with enhanced propagation and alignment},
  booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
  year = {2022}
}
@article{chan2022generalization,
  title={On the Generalization of {BasicVSR++} to Video Deblurring and Denoising},
  author={Chan, Kelvin CK and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change},
  journal={arXiv preprint arXiv:2204.05308},
  year={2022}
}
Comments
  • denoise pretrained model  didn't work

    denoise pretrained model didn't work

    Hi @ckkelvinchan , I have tried to run the denoise pretrianed model with DAVIS-test dataset, but the result images has no difference between the input images. My experiment is as below: pretrained model: basicvsr_plusplus_denoise-28f6920c.pth optical flow model: I download it to my local path dataset: I use the the tractor folder in DAIVS-test dataset which is used in your paper. I add the sigma 50 gaussian noise to the pictures as input. result: image

    It seem that it run successfully, but the result has no change between the input. Is their any parameter or option error in my attempt? By the way, basicvsr_plusplus_c64n7_8x1_600k_reds4.pth can run successfully and the result is excellent!

    opened by bjiale 8
  • Question for the computation of cond__n1

    Question for the computation of cond__n1

    why the cond_n1 is the out of flow_wrap(feat_prop, flow_n1.permute(0, 2, 3, 1)) while feat_prop is zeros? why not flow_wrap(feat_n1,flow_n1.permute(0, 2, 3, 1) and feat_n1 = feats[module_name][-1]

    opened by xingbo-jiang 4
  • Encountered error while trying to install package mmcv-full

    Encountered error while trying to install package mmcv-full

    I installed pytorch using the following command conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch I am trying to install mmcv-full. I am facing the error

    opened by purna821 1
  • Your Results in New Super-Resolution Benchmarks

    Your Results in New Super-Resolution Benchmarks

    Hello,

    MSU Graphics & Media Lab Video Group has recently launched two new Super-Resolution Benchmarks.

    Your method achieved 22nd place in Video Upscalers Benchmark: Quality Enhancement in 'Animation 2x' category. We look forward to your future work!

    We would be grateful for your feedback on our work.

    opened by EvgeneyBogatyrev 0
  • Data Preparation

    Data Preparation

    Hi, thank you very much for your excellent work. I wonder could you please show us the folders of the dataset like the following forms? So that I can recollect the dataset into the same form as yours. image

    opened by linjing7 0
  • Question for the flow based DCN

    Question for the flow based DCN

    First of all, thanks for your great work. I have a question for a piece of code which is about flow based DCN

    offset_1 = offset_1 + flow_1.flip(1).repeat(1, offset_1.size(1) // 2, 1, 1)

    Could you explain why flip(1) is needed for flow, that is why need to switch x and y

    opened by Craigie-Hill 2
  • dastset of denoising task

    dastset of denoising task

    In the denoising task, what are the training data set, verification data set and test data set respectively, can you provide them? Is the Set8 dataset a validation set in the code? But in Table5 and Table6 of the paper, we see that Set8 is used for testing

    opened by Estherisok666 0
  • BasicVSR++2 and Training Speed

    BasicVSR++2 and Training Speed

    In the technical report for deblurring and denoising (https://arxiv.org/pdf/2204.05308.pdf), there are two versions of the BasicVSR++ for video denoising, the one which reduces the resolution by 2 (BasicVSR++2, superior performance but less speed) and by 4 (BasicVSR++4, more speed but worse performance). Current code, as I understand, implements the version which reduces the resolution by 4. How do I change the code exactly to switch to the BasicVSR++2 (the one which reduces the resolution by 2)? Or could you please share the code?

    In addition, I am trying to speed up the training for denoising. I tried increasing samples_per_gpu config variable from 1 to 3 and decreasing the number of iterations from 600,000 to 200,000 (so that the amount of data that the model uses for training is the same), which helped to improve the time quite well (from 13 days to 6 days of training). Is there any other way to make the training faster without hurting the performance after the training?

    Thank you!

    opened by mikolez 2
  • I used demo to run my own video file in linux environment. After a period of time, the process running demo was killed

    I used demo to run my own video file in linux environment. After a period of time, the process running demo was killed

    My Enviorment: PyTorch 1.11.0 Python 3.8(ubuntu20.04) Cuda 11.3 GPU:RTX A5000 * 1;video memory:24GB CPU:14core Intel(R) Xeon(R) Gold 6330 CPU @ 2.00GHz ram:30GB system disk:20 GB data disk:50GB SSD While running the demo based on the basicvsr_plusplus_reds4.pth checkpoint, I put in a blurry video of myself for several tens of seconds, and after a while the process was killed. I tried to set -- max-seq-len=1 but still the process was killed.How can I solve this problem?

    opened by Dylan-Jinx 2
  • BasicVSR++ Denoise Train Error

    BasicVSR++ Denoise Train Error

    When I try to run the training for denoising using the suggested script and configuration file, I get "ValueError: The input sequence is not long enough to support the current choice of [interval] or [num_input_frames]" at BasicVSR_PlusPlus/mmedit/datasets/pipelines/augmentations.py, line 1053. I investigated the code a bit and it seems that the problem is with the way data is loaded. It looks like there is a variable at some point of loading the data which is called sequence_length and it is always zero because it calculates number of png images. However, DAVIS dataset only consists of jpg images, so that's why the sequence_length is zero. I assume the data is preprocessed for denoising task in some way and saved in png format in the end. The data preprocessing step is unfortunately missing in this repo, so I am wondering could it be the reason for my error? Or there is no preprocessing when training for denoising? I am trying to run the training with the DAVIS dataset.

    Curious if you have any clue why my error is occurring.

    Thank you!

    opened by mikolez 3
Owner
Kelvin C.K. Chan
Kelvin C.K. Chan
Official repository for "PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation"

pair-emnlp2020 Official repository for the paper: Xinyu Hua and Lu Wang: PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long

Xinyu Hua 31 Oct 13, 2022
Official repository for Few-shot Image Generation via Cross-domain Correspondence (CVPR '21)

Few-shot Image Generation via Cross-domain Correspondence Utkarsh Ojha, Yijun Li, Jingwan Lu, Alexei A. Efros, Yong Jae Lee, Eli Shechtman, Richard Zh

Utkarsh Ojha 251 Dec 11, 2022
Official repository for Jia, Raghunathan, Göksel, and Liang, "Certified Robustness to Adversarial Word Substitutions" (EMNLP 2019)

Certified Robustness to Adversarial Word Substitutions This is the official GitHub repository for the following paper: Certified Robustness to Adversa

Robin Jia 38 Oct 16, 2022
Official repository for the ICLR 2021 paper Evaluating the Disentanglement of Deep Generative Models with Manifold Topology

Official repository for the ICLR 2021 paper Evaluating the Disentanglement of Deep Generative Models with Manifold Topology Sharon Zhou, Eric Zelikman

Stanford Machine Learning Group 34 Nov 16, 2022
The repository offers the official implementation of our paper in PyTorch.

Cloth Interactive Transformer (CIT) Cloth Interactive Transformer for Virtual Try-On Bin Ren1, Hao Tang1, Fanyang Meng2, Runwei Ding3, Ling Shao4, Phi

Bingoren 49 Dec 1, 2022
Official code repository of the paper Learning Associative Inference Using Fast Weight Memory by Schlag et al.

Learning Associative Inference Using Fast Weight Memory This repository contains the offical code for the paper Learning Associative Inference Using F

Imanol Schlag 18 Oct 12, 2022
Official repository for "Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems"

Action-Based Conversations Dataset (ABCD) This respository contains the code and data for ABCD (Chen et al., 2021) Introduction Whereas existing goal-

ASAPP Research 49 Oct 9, 2022
Official repository for HOTR: End-to-End Human-Object Interaction Detection with Transformers (CVPR'21, Oral Presentation)

Official PyTorch Implementation for HOTR: End-to-End Human-Object Interaction Detection with Transformers (CVPR'2021, Oral Presentation) HOTR: End-to-

Kakao Brain 114 Nov 28, 2022
Official repository for "Intriguing Properties of Vision Transformers" (2021)

Intriguing Properties of Vision Transformers Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, & Ming-Hsuan Yang P

Muzammal Naseer 155 Dec 27, 2022
Competitive Programming Club, Clinify's Official repository for CP problems hosting by club members.

Clinify-CPC_Programs This repository holds the record of the competitive programming club where the competitive coding aspirants are thriving hard and

Clinify Open Sauce 4 Aug 22, 2022
Official repository for "On Improving Adversarial Transferability of Vision Transformers" (2021)

Improving-Adversarial-Transferability-of-Vision-Transformers Muzammal Naseer, Kanchana Ranasinghe, Salman Khan, Fahad Khan, Fatih Porikli arxiv link A

Muzammal Naseer 47 Dec 2, 2022
This is the official repository of XVFI (eXtreme Video Frame Interpolation)

XVFI This is the official repository of XVFI (eXtreme Video Frame Interpolation), https://arxiv.org/abs/2103.16206 Last Update: 20210607 We provide th

Jihyong Oh 195 Dec 29, 2022
The official repository for BaMBNet

BaMBNet-Pytorch Paper

Junjun Jiang 18 Dec 4, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

selfcontact This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] It includes the main function

Lea Müller 68 Dec 6, 2022
CVPR 2021 - Official code repository for the paper: On Self-Contact and Human Pose.

SMPLify-XMC This repo is part of our project: On Self-Contact and Human Pose. [Project Page] [Paper] [MPI Project Page] License Software Copyright Lic

Lea Müller 83 Dec 14, 2022
Official repository for the paper "Going Beyond Linear Transformers with Recurrent Fast Weight Programmers"

Recurrent Fast Weight Programmers This is the official repository containing the code we used to produce the experimental results reported in the pape

IDSIA 36 Nov 15, 2022
Official repository for the paper "Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks"

Easy-To-Hard The official repository for the paper "Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks". Gett

Avi Schwarzschild 52 Sep 8, 2022
Official repository for the CVPR 2021 paper "Learning Feature Aggregation for Deep 3D Morphable Models"

Deep3DMM Official repository for the CVPR 2021 paper Learning Feature Aggregation for Deep 3D Morphable Models. Requirements This code is tested on Py

null 38 Dec 27, 2022