Blind Video Temporal Consistency via Deep Video Prior

Overview

deep-video-prior (DVP)

Code for NeurIPS 2020 paper: Blind Video Temporal Consistency via Deep Video Prior

PyTorch implementation | paper | project website

Introduction

Our method is a general framework to improve the temporal consistency of video processed by image algorithms. For example, combining image colorization or image dehazing algorithm with our framework, we can achieve the goal of video colorization or video dehazing.

Dependencey

Environment

This code is based on tensorflow. It has been tested on Ubuntu 18.04 LTS.

Anaconda is recommended: Ubuntu 18.04 | Ubuntu 16.04

After installing Anaconda, you can setup the environment simply by

conda env create -f environment.yml
conda activate deep-video-prior

Download VGG model

cd deep-video-prior
python download_VGG.py
unzip VGG_Model.zip

Inference

Demo

bash test.sh

The results are placed in ./result

Use your own data

For the video with unimodal inconsistency:

python dvp_video_consistency.py --input PATH_TO_YOUR_INPUT_FOLDER --processed PATH_TO_YOUR_PROCESSED_FOLDER --task NAME_OF_YOUR_MODEL  --output ./result/OWN_DATA

For the video with multimodal inconsistency:

python dvp_video_consistency.py --input PATH_TO_YOUR_INPUT_FOLDER --processed PATH_TO_YOUR_PROCESSED_FOLDER --task NAME_OF_YOUR_MODEL --with_IRT 1 --IRT_initialization 1 --output ./result/OWN_DATA

Other information

  -h, --help            show this help message and exit
  --task TASK           Name of task
  --input INPUT         Dir of input video
  --processed PROCESSED
                        Dir of processed video
  --output OUTPUT       Dir of output video
  --use_gpu USE_GPU     Use gpu or not
  --loss {perceptual,l1,l2}
                        Chooses which loss to use. perceptual, l1, l2
  --network {unet}      Chooses which model to use. unet, fcn
  --coarse_to_fine_speedup COARSE_TO_FINE_SPEEDUP
                        Use coarse_to_fine_speedup for training
  --with_IRT WITH_IRT   Sse IRT or not, set this to 1 if you want to solve
                        multimodal inconsistency
  --IRT_initialization IRT_INITIALIZATION
                        Sse initialization for IRT
  --large_video LARGE_VIDEO
                        Set this to 1 when the number of video frames are
                        large, e.g., more than 1000 frames
  --save_freq SAVE_FREQ
                        Save frequency of epochs
  --max_epoch MAX_EPOCH
                        The max number of epochs for training
  --format FORMAT       Format of output image

Citation

If you find this work useful for your research, please cite:

@inproceedings{lei2020dvp,
  title={Blind Video Temporal Consistency via Deep Video Prior},
  author={Lei, Chenyang and Xing, Yazhou and Chen, Qifeng},
  booktitle={Advances in Neural Information Processing Systems},
  year={2020}
}                

Contact

Please contact me if there is any question (Chenyang Lei, [email protected])

Beyond the tasks we evaluated

Researcher found that Blind Temporal Consistency (e.g., DVP) can be applied to many more tasks!

Comments
  • bash test.sh Traceback... error

    bash test.sh Traceback... error

    File "main_IRT.py", line 7, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow' Traceback (most recent call last): File "main_IRT.py", line 7, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow' Traceback (most recent call last): File "main_IRT.py", line 7, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow' Traceback (most recent call last): File "main_IRT.py", line 7, in import tensorflow as tf ModuleNotFoundError: No module named 'tensorflow'

    opened by TechHalloman 6
  • 在老视频上色的处理中,颜色不准确受到之前帧图片颜色影响

    在老视频上色的处理中,颜色不准确受到之前帧图片颜色影响


    上面是计算出来的图片,具体命令如下: python main_IRT.py --max_epoch 50 --input demo/colorization/ai_input1000 --processed demo/colorization/ai_processed1000 --model enhancement --with_IRT 1 --IRT_initialization 1 --output ./result/colorization1000/ 表现效果为:第三张图片偏黄,且受到了之前帧炮弹爆炸画面黄光的影响。

    同样的model colorization参数也试过,表现效果为:图片直接变成了晚上,深蓝色的天空,与实际上色图片差距太大。

    小弟才学有限,对大佬十分敬仰,期待您的解答。

    opened by stone100010 6
  • Processing larger videos

    Processing larger videos

    HI, I am trying to process a large video (10 min) to deflicker after colorization. I have left my system on to run for about 24 hours after witch it trained for 50 epochs. After 50 epochs the result is still quite poor and looks more like the original black and white video than the colorized one. Did you ever test on longer video’s? Is the result going to improve with more training epochs? Or is it impossible to use your model on a video this long? Thanks in advance for your answer!

    opened by Saartjes 5
  • Traceback (most recent call last):   File import scipy.io ModuleNotFoundError: No module named 'scipy'">

    Traceback (most recent call last): File "dvp_video_consistency.py", line 6, in import scipy.io ModuleNotFoundError: No module named 'scipy'

    Traceback (most recent call last): File "dvp_video_consistency.py", line 6, in import scipy.io ModuleNotFoundError: No module named 'scipy'

    opened by SolomGfxcolor 4
  • Video Super-Resolution via Deep Video Prior

    Video Super-Resolution via Deep Video Prior

    Thanks for your excellent work!

    I have two questions about the Deep Video Prior.

    1、Have you attempted to do some experiments about VSR with the deep video prior? if yes, how about the performance?

    2、In my opinion, the temporal consistency of output frames highly depends on the temporal consistency of input frames. If the input frames don't have good temporal consistency, can this deep video prior work?

    Hope you can help me solve above problem.

    opened by littlewhitesea 3
  • Wrong resolution of output frames.

    Wrong resolution of output frames.

    After running supplied demo (regardless of used task ) resolution of output video frames is different from than resolution of input/processed frames. e.g. resolution of "goat_input and goat_processed" frames is 854x480 however output result is 832x480. Processing my own frames (regardless of used task) with 1440x1080 delivers frames with 1440x1056.

    opened by semel1 2
  • Wrong version

    Wrong version

    When attempting to run main_IRT.py with 6000 frames causes RuntimeError: CUDA out of memory. This could be fixed by reducing batch size or use memory when needed. Also, I`m using google Colab and yes the ram that google collab had offered did not reach the full amount of ram instead, it stopped at 2/3 of it and caused this error

    Traceback (most recent call last): File "main_IRT.py", line 139, in net_in = torch.from_numpy(net_in).permute(0,3,1,2).float().to(device) RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.73 GiB total capacity; 12.13 GiB already allocated; 5.88 MiB free; 13.79 GiB reserved in total by PyTorch)

    opened by myname1111 2
  • 待处理数据文件比较多,CPU占用爆表,电脑卡死了

    待处理数据文件比较多,CPU占用爆表,电脑卡死了

    我做了一个戴口罩的视频修改功能,videoFrames文件夹下是视频拆帧数据,processed/colorization文件夹下是戴面具处理的数据,两文件夹各6000张图片(1920*720),使用的是: python main_IRT.py --max_epoch 1 --input videoFrames --processed processed/colorization --model colorization --with_IRT 1 --IRT_initialization 1 --output ./result/colorization/AddMask 我的CPU直接干到100%, 内存128也不够用,电脑就卡住了,具体配置如下: "--use_gpu", default=1, tensorflow-gpu=1.13.1

    期待您的解答,感谢您!

    opened by stone100010 2
  • The code to calculate Ewarp

    The code to calculate Ewarp

    Hi, thanks a lot for your amazing work. I wonder whether you could share your code to calculate Ewarp, which is mentioned in your paper to evaluate temporal consistency. Thanks a lot.

    opened by wangjk666 2
  • The url of downloading VGG.zip is invalid now

    The url of downloading VGG.zip is invalid now

    Dear author, I tried to implement the code for learning, however, an error appeared when I compiled the command [ python download_VGG.py ], which points to that the url of downloading VGG.zip is invalid now. Therefore, if convenient, could you please show me another one?

    opened by Ssakura-go 1
  • Final result is flickering

    Final result is flickering

    Hi, First all, thank you for your good work ! We have one question: Final video flickering a little more using you source code. Video source is under your demo/spatial_white_balance. Attach is test result. video.zip Would you like give us some suggestion ? Many thanks.

    opened by delldu 1
  • Can you please share a google colab notebook?

    Can you please share a google colab notebook?

    Hey I was creating a colab notebook for your code but I am facing some issues a lot of requirements are missing can you please create a colab notebook? Please

    opened by mau021 0
  • attempt to get argmax of an empty sequence

    attempt to get argmax of an empty sequence

    Upon running "main_IRT.py", got an error:

    Exception has occurred: ValueError
    attempt to get argmax of an empty sequence
      File "D:\colorization\deep-video-prior-master-pytorch\main_IRT.py", line 49, in <module>
        for x in subprocess.Popen("nvidia-smi -q -d Memory | grep -A4 GPU | grep Free", shell=True, stdout=subprocess.PIPE).stdout.readlines()]))
    

    Happened in both Windows 10 and WSL2 Nvidia RTX 3090

    opened by semel1 0
Owner
Chenyang LEI
CS Ph.D. student at HKUST
Chenyang LEI
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Code for Blind Image Decomposition (BID) and Blind Image Decomposition network (BIDeN).

arXiv, porject page, paper Blind Image Decomposition (BID) Blind Image Decomposition is a novel task. The task requires separating a superimposed imag

null 64 Dec 20, 2022
A python software that can help blind people find things like laptops, phones, etc the same way a guide dog guides a blind person in finding his way.

GuidEye A python software that can help blind people find things like laptops, phones, etc the same way a guide dog guides a blind person in finding h

Munal Jain 0 Aug 9, 2022
Self-Learned Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence

In this paper, we address the problem of rain streaks removal in video by developing a self-learned rain streak removal method, which does not require any clean groundtruth images in the training process.

Yang Wenhan 44 Dec 6, 2022
Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis [Paper] [Online Demo] The following results are obtained by our SCUNet with purely syn

Kai Zhang 312 Jan 7, 2023
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 5 Jul 14, 2022
Source code for paper "Deep Superpixel-based Network for Blind Image Quality Assessment"

DSN-IQA Source code for paper "Deep Superpixel-based Network for Blind Image Quality Assessment" Requirements Python >=3.8.0 Pytorch >=1.7.1 Usage wit

null 7 Oct 13, 2022
Official implementation of Unfolded Deep Kernel Estimation for Blind Image Super-resolution.

Unfolded Deep Kernel Estimation for Blind Image Super-resolution Hongyi Zheng, Hongwei Yong, Lei Zhang, "Unfolded Deep Kernel Estimation for Blind Ima

Z80 15 Dec 26, 2022
Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

MEGVII Research 141 Dec 30, 2022
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

emmanuel 47 Nov 6, 2022
Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging, ICCV2021 [PyTorch Code]

Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging, ICCV2021 [PyTorch Code]

Jian Zhang 20 Oct 24, 2022
Code of the paper "Deep Human Dynamics Prior" in ACM MM 2021.

Code of the paper "Deep Human Dynamics Prior" in ACM MM 2021. Figure 1: In the process of motion capture (mocap), some joints or even the whole human

Shinny cui 3 Oct 31, 2022
PyTorch code for SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised DA

PyTorch Code for SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation Viraj Prabhu, Shivam Khare, Deeks

Viraj Prabhu 46 Dec 24, 2022
[PyTorch] Official implementation of CVPR2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency". https://arxiv.org/abs/2103.05465

PointDSC repository PyTorch implementation of PointDSC for CVPR'2021 paper "PointDSC: Robust Point Cloud Registration using Deep Spatial Consistency",

null 153 Dec 14, 2022
Unsupervised Video Interpolation using Cycle Consistency

Unsupervised Video Interpolation using Cycle Consistency Project | Paper | YouTube Unsupervised Video Interpolation using Cycle Consistency Fitsum A.

NVIDIA Corporation 100 Nov 30, 2022
python library for invisible image watermark (blind image watermark)

invisible-watermark invisible-watermark is a python library and command line tool for creating invisible watermark over image.(aka. blink image waterm

Shield Mountain 572 Jan 7, 2023
[CVPR 2021] Unsupervised Degradation Representation Learning for Blind Super-Resolution

DASR Pytorch implementation of "Unsupervised Degradation Representation Learning for Blind Super-Resolution", CVPR 2021 [arXiv] Overview Requirements

Longguang Wang 318 Dec 24, 2022
DAN: Unfolding the Alternating Optimization for Blind Super Resolution

DAN-Basd-on-Openmmlab DAN: Unfolding the Alternating Optimization for Blind Super Resolution We reproduce DAN via mmediting based on open-sourced code

AlexZou 72 Dec 13, 2022