Dataset and Code for ICCV 2021 paper "Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme"

Overview

Dataset and Code for RealVSR

Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme
Xi Yang, Wangmeng Xiang, Hui Zeng and Lei Zhang
International Conference on Computer Vision, 2021.

Dataset

The dataset is hosted on Google Drive and Baidu Drive (code: 43ph). Some example scenes are shown below.

dataset_samples

The structure of the dataset is illustrated below.

File Description
GT.zip All ground truth sequences in RGB format
LQ.zip All low quality sequences in RGB format
GT_YCbCr.zip All ground truth sequences in YCbCr format
LQ_YCbCr.zip All low quality sequences in YCbCr format
GT_test.zip Ground truth test sequences in RGB format
LQ_test.zip Low Quality test sequences in RGB format
GT_YCbCr_test.zip Ground truth test sequences in YCbCr format
LQ_YCbCr_test.zip Low Quality test sequences in YCbCr format

Code

Dependencies

  • Linux (tested on Ubuntu 18.04)
  • Python 3 (tested on python 3.7)
  • NVIDIA GPU + CUDA (tested on CUDA 10.2 and 11.1)

Installation

# Create a new anaconda python environment (realvsr)
conda create -n realvsr python=3.7 -y

# Activate the created environment
conda activate realvsr

# Install dependencies
pip install -r requirements.txt

# Bulid the DCN module
cd codes/models/archs/dcn
python setup.py develop

Training

Modify the configuration files accordingly in codes/options/train folder and run the following command (current we did not implement distributed training):

python train.py -opt xxxxx.yml

Testing

Test on RealVSR testing set sequences:

Modify the configuration in test_RealVSR_wi_GT.py and run the following command:

python test_RealVSR_wi_GT.py

Test on real-world captured sequences:

Modify the configuration in test_RealVSR_wo_GT.py and run the following command:

python test_RealVSR_wo_GT.py

Pre-trained Models

Some pretrained models could be found on Google Drive and Baidu Drive (code: n1n0).

License

This project is released under the Apache 2.0 license.

Citation

If you find this code useful in your research, please consider citing:

@article{yang2021real,
  title={Real-world Video Super-resolution: A Benchmark Dataset and A Decomposition based Learning Scheme},
  author={YANG, Xi and Xiang, Wangmeng and Zeng, Hui and Zhang, Lei},
  journal=ICCV,
  year={2021}
}

Acknowledgement

This implementation largely depends on EDVR. Thanks for the excellent codebase! You may also consider migrating it to BasicSR.

Comments
  • Unable to reproduce the results

    Unable to reproduce the results

    Hi guys, Thanks for the nice paper and the code. I tried to reproduce the button results in Fig. 6 from the paper without success. I am just trying to run an inference with the pretrained EDVR model but faced the following problems:

    1. In fig. 6 you show x2 upscale but the model outputs the same image size as LR! I presume that this should act as a sharpening model, but is there another model for x2 upscale that you show in the paper?
    2. After applying output = data_util.ycbcr2bgr(output), the result is bgr. I converted it to rgb in order to save as png with cv2.imwrite but the result is very poor and far from the rather sharp outcome that you present in the paper.

    Will appreciate your answer.

    opened by ramiben 2
  • Visualize the Y channel and Chrominance channel

    Visualize the Y channel and Chrominance channel

    Hi @IanYeung Given the YCbCr format of RealVSR Dataset, how could I visualize the Y channel and chrominance channel like you did in your paper? image Could you please give me some suggestions? Thx a lot.

    opened by susan-sun1999 2
  • About the Laplacian Pyramid Loss Function

    About the Laplacian Pyramid Loss Function

    In the paper and program implementation, after decomposing the luminance channel, the charbonnier loss is used to calculate the loss, the selection mode is sum, and each decomposition loss is added. If the mode is chosen to be mean, the different layers of the pyramid cannot be summed due to inconsistent sizes. I chose mean and upsampled the pyramid losses of different scales, and added them after the scales were consistent. From this, the return tensor of the mean type is constructed. But the final reconstruction effect is that there are many regular small dots on the image. Do you have any comments?

    opened by dzz416 0
  • About the loss

    About the loss

    I want to know what l_d_fake and l_d_real mean, l_g_gan is the sum of discriminator and generator? And what the 'd' and 'g' in l_d_fake and l_g_pix_s stand for?

    opened by SongYxing 9
  • About the RGB to YCbCr

    About the RGB to YCbCr

    Can you tell me how you convert RGB to YCbCr. The results I get with OPENCV conversion function are inconsistent with yours. Or can you show your code of RGB to YCbCr?

    opened by SongYxing 2
  • what does it refer to test_frames_YCbCr

    what does it refer to test_frames_YCbCr

    in test_RealVSR_wo_GT.py

    dataset

    read_folder = '/home/yangxi/datasets/RealVSR/test_frames_YCbCr'
    

    the test_frames_YCbCr' in this path where did it come from?There is no file with this name in the downloaded dataset

    opened by wei-hub 1
  • Problem running

    Problem running

    How can I solve this problem when I run it?

    (realvsr) lzw@computer3:~/code/RealVSR/RealVSR-main/codes$ python test_RealVSR_wi_GT.py 22-04-06 17:05:11.072 - INFO: Data: RealVSR - /home/lzw/code/data/ycbcr/LQ_YCbCr_test 22-04-06 17:05:11.073 - INFO: Padding mode: replicate 22-04-06 17:05:11.073 - INFO: Model path: /home/lzw/code/data/ycbcr/RealVSR-Models/001_EDVR_NoUp_woTSA_scratch_lr1e-4_150k_RealVSR_3frame_WiCutBlur_YCbCr_LapPyr+GW.pth 22-04-06 17:05:11.073 - INFO: Save images: True Traceback (most recent call last): File "test_RealVSR_wi_GT.py", line 220, in main() File "test_RealVSR_wi_GT.py", line 98, in main imgs = data_util.read_img_seq(subfolder, color=color) File "/home/lzw/code/RealVSR/RealVSR-main/codes/data/util.py", line 116, in read_img_seq img_l = [read_img(None, v) for v in img_path_l] File "/home/lzw/code/RealVSR/RealVSR-main/codes/data/util.py", line 116, in img_l = [read_img(None, v) for v in img_path_l] File "/home/lzw/code/RealVSR/RealVSR-main/codes/data/util.py", line 95, in read_img img = img.astype(np.float32) / 255. AttributeError: 'NoneType' object has no attribute 'astype'

    opened by wei-hub 3
  • About image registration

    About image registration

    您好,请教几个关于数据对齐的问题 1、请问图像配准是直接用Real-SR项目中代码做吗,有没有做了什么改进. 2、Real-SR中提到图像要先经过ps进行镜头矫正后再进行迭代配准,请问本文数据是否也同样做了矫正呢? 3、请问你们配准成功率大概多少? 谢谢了

    Hello, I have some questions about data alignment

    1. Is the image registration done directly with the code in the Real-SR project?
    2. In Real-SR, the images were import into PS to correct the lens distortion before the image registration. Does the data in this paper also has lens distortion?
    3. What is the registration success rate? Thanks
    opened by Wenju-Huang 2
Owner
Xi Yang
PhD Candidate @ PolyU, working on low-level computer vision
Xi Yang
code for ICCV 2021 paper 'Generalized Source-free Domain Adaptation'

G-SFDA Code (based on pytorch 1.3) for our ICCV 2021 paper 'Generalized Source-free Domain Adaptation'. [project] [paper]. Dataset preparing Download

Shiqi Yang 84 Dec 26, 2022
Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators..

ARAPReg Code for ICCV 2021 paper: ARAPReg: An As-Rigid-As Possible Regularization Loss for Learning Deformable Shape Generators.. Installation The cod

Bo Sun 132 Nov 28, 2022
Code for the ICCV 2021 paper "Pixel Difference Networks for Efficient Edge Detection" (Oral).

Pixel Difference Convolution This repository contains the PyTorch implementation for "Pixel Difference Networks for Efficient Edge Detection" by Zhuo

Alex 236 Dec 21, 2022
Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization

Sync2Gen Code for ICCV 2021 paper: Scene Synthesis via Uncertainty-Driven Attribute Synchronization 0. Environment Environment: python 3.6 and cuda 10

Haitao Yang 62 Dec 30, 2022
Code for the paper "Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds" (ICCV 2021)

Spatio-temporal Self-Supervised Representation Learning for 3D Point Clouds This is the official code implementation for the paper "Spatio-temporal Se

Hesper 63 Jan 5, 2023
Code release for ICCV 2021 paper "Anticipative Video Transformer"

Anticipative Video Transformer Ranked first in the Action Anticipation task of the CVPR 2021 EPIC-Kitchens Challenge! (entry: AVT-FB-UT) [project page

Facebook Research 123 Dec 13, 2022
Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

Official code release for ICCV 2021 paper SNARF: Differentiable Forward Skinning for Animating Non-rigid Neural Implicit Shapes.

null 235 Dec 26, 2022
Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Sensor-Guided Optical Flow Demo code for "Sensor-Guided Optical Flow", ICCV 2021 This code is provided to replicate results with flow hints obtained f

null 10 Mar 16, 2022
Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Code for ICCV 2021 paper "HuMoR: 3D Human Motion Model for Robust Pose Estimation"

Davis Rempe 367 Dec 24, 2022
Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

Unified-EPT Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation. Installation Linux, CUDA>=10.0,

null 29 Aug 23, 2022
Starter code for the ICCV 2021 paper, 'Detecting Invisible People'

Detecting Invisible People [ICCV 2021 Paper] [Website] Tarasha Khurana, Achal Dave, Deva Ramanan Introduction This repository contains code for Detect

Tarasha Khurana 28 Sep 16, 2022
HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021)

Code for HDR Video Reconstruction HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021) Guanying Chen, Cha

Guanying Chen 64 Nov 19, 2022
VIL-100: A New Dataset and A Baseline Model for Video Instance Lane Detection (ICCV 2021)

Preparation Please see dataset/README.md to get more details about our datasets-VIL100 Please see INSTALL.md to install environment and evaluation too

null 82 Dec 15, 2022
Official implementation of the ICCV 2021 paper "Joint Inductive and Transductive Learning for Video Object Segmentation"

JOINT This is the official implementation of Joint Inductive and Transductive learning for Video Object Segmentation, to appear in ICCV 2021. @inproce

Yunyao 35 Oct 16, 2022
Official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

Vision Transformer with Progressive Sampling This is the official implementation of the paper Vision Transformer with Progressive Sampling, ICCV 2021.

yuexy 123 Jan 1, 2023
PyTorch implementation of paper: AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer, ICCV 2021.

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer [Paper] [PyTorch Implementation] [Paddle Implementation] Overview This reposit

null 148 Dec 30, 2022
Homepage of paper: Paint Transformer: Feed Forward Neural Painting with Stroke Prediction, ICCV 2021.

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction [Paper] [PaddlePaddle Implementation] Homepage of paper: Paint Transformer: Fee

null 442 Dec 16, 2022
Official Repository for the ICCV 2021 paper "PixelSynth: Generating a 3D-Consistent Experience from a Single Image"

PixelSynth: Generating a 3D-Consistent Experience from a Single Image (ICCV 2021) Chris Rockwell, David F. Fouhey, and Justin Johnson [Project Website

Chris Rockwell 95 Nov 22, 2022
Official implementation of the ICCV 2021 paper "Conditional DETR for Fast Training Convergence".

The DETR approach applies the transformer encoder and decoder architecture to object detection and achieves promising performance. In this paper, we handle the critical issue, slow training convergence, and present a conditional cross-attention mechanism for fast DETR training. Our approach is motivated by that the cross-attention in DETR relies highly on the content embeddings and that the spatial embeddings make minor contributions, increasing the need for high-quality content embeddings and thus increasing the training difficulty.

null 281 Dec 30, 2022