[NeurIPS 2020] Blind Video Temporal Consistency via Deep Video Prior

Yazhou XING

Last update: Oct 19, 2022

Related tags

Deep Learning pytorch-deep-video-prior

Overview

pytorch-deep-video-prior (DVP)

Official PyTorch implementation for NeurIPS 2020 paper: Blind Video Temporal Consistency via Deep Video Prior

TensorFlow implementation | paper | project website

Introduction

Our method is a general framework to improve the temporal consistency of video processed by image algorithms.

For example, combining single image colorization or single image dehazing algorithm with our framework, we can achieve the goal of video colorization or video dehazing.

Dependency

Environment

This code is based on PyTorch. It has been tested on Ubuntu 18.04 LTS.

Anaconda is recommended: Ubuntu 18.04 | Ubuntu 16.04

After installing Anaconda, you can setup the environment simply by

conda env create -f environment.yml

Inference

Demo

bash test.sh

The results will be saved in ./result

Use your own data

For the video with unimodal inconsistency:

python main_IRT.py --max_epoch 25 --input PATH_TO_YOUR_INPUT_FOLDER --processed PATH_TO_YOUR_PROCESSED_FOLDER --model NAME_OF_YOUR_MODEL --with_IRT 0 --IRT_initialization 0 --output ./result/OWN_DATA

For the video with multimodal inconsistency:

python main_IRT.py --max_epoch 25 --input PATH_TO_YOUR_INPUT_FOLDER --processed PATH_TO_YOUR_PROCESSED_FOLDER --model NAME_OF_YOUR_MODEL --with_IRT 1 --IRT_initialization 1 --output ./result/OWN_DATA

Citation

If you find this work useful for your research, please cite:

@inproceedings{lei2020dvp,
  title={Blind Video Temporal Consistency via Deep Video Prior},
  author={Lei, Chenyang and Xing, Yazhou and Chen, Qifeng},
  booktitle={Advances in Neural Information Processing Systems},
  year={2020}
}

Contact

Feel free to contact me if there is any question. (Yazhou Xing, [email protected])

Comments

Temporally-consistent version of depth estimation

Is it possible to generate temporally consistent video from depth map sequence obtained by depth estimation (MIDAS) similar to "Blind Video Temporal Consistency project" https://github.com/lulu1315/BlindConsistency or https://github.com/nbonneel/blindconsistency?

opened by semel1 4
CUDA out of memory

When attempting to run main_IRT.py with 6000 frames causes RuntimeError: CUDA out of memory. This could be fixed by reducing batch size or use memory when needed. Also, I`m using google Colab and yes the ram that google collab had offered did not reach the full amount of ram instead, it stopped at 2/3 of it and caused this error

Traceback (most recent call last): File "main_IRT.py", line 139, in net_in = torch.from_numpy(net_in).permute(0,3,1,2).float().to(device) RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.73 GiB total capacity; 12.13 GiB already allocated; 5.88 MiB free; 13.79 GiB reserved in total by PyTorch)

opened by myname1111 3
copy of another issue

When attempting to run main_IRT.py with 6000 frames causes RuntimeError: CUDA out of memory. This could be fixed by reducing batch size or use memory when needed. Also, I`m using google Colab and yes the ram that google collab had offered did not reach the full amount of ram instead, it stopped at 2/3 of it and caused this error

Traceback (most recent call last): File "main_IRT.py", line 139, in net_in = torch.from_numpy(net_in).permute(0,3,1,2).float().to(device) RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 14.73 GiB total capacity; 12.13 GiB already allocated; 5.88 MiB free; 13.79 GiB reserved in total by PyTorch)

opened by myname1111 2
输出结果尺寸与原序列不一致

Hi，首先非常感谢您愿意给出pytroch版本的DVP，对于学习DVP真的帮大忙了！在复现您论文结果的过程中，发现DVP输出结果尺寸与原图像序列不一致。例如，在demo中的上色序列为854 * 480，而test.sh输出结果为 832 * 480。在demo其他任务中也是这样。

请问这种情况正常吗？因为我希望计算结果的PSNR值，但是被这个问题所困扰着。希望您能为我解答这个问题，非常感谢！

opened by EderOdan 2
Reproducing Evaluation Results

Hello authors, thank you for your great work!

I am writing a blog post about your work and I was interested in reproducing the results completely (on DAVIS/the Bonneel dataset). Would it be possible for you to provide evaluation code or instructions as to evaluating E_warp and F_data for your method on these datasets?

All the best,

opened by greeneggsandyaml 2
CUDA out of memory

I noticed that a similar issue has been closed. However, I have the same problem. So, "You can also load the frame just before each iteration. I think torch.utils.data.DataLoader can be a better option for this issue. We will update the code using pytorch dataloader." Any code update? Could you please provide me with more detail on how to " load the frame just before each iteration"

opened by semel1 1
IRT loss

Hi, I see that the loss used for IRT in the code is different than the one in the paper. Can u pls explain why is there a difference pls?

Also, i see u divide the pixelwise difference of O_main and the net_gt by the image. What's the idea behind this? Thank you!

opened by nicolas-dufour 1
Run code with CPU only

Is it possible to run this code using CPU only? Tried "conda install pytorch torchvision torchaudio cpuonly -c pytorch" - no sucssess. Running Windows 10.

opened by semel1 0

attempt to get argmax of an empty sequence

Upon running "main_IRT.py", got an error:

Exception has occurred: ValueError
attempt to get argmax of an empty sequence
  File "D:\colorization\deep-video-prior-master-pytorch\main_IRT.py", line 49, in <module>
    for x in subprocess.Popen("nvidia-smi -q -d Memory | grep -A4 GPU | grep Free", shell=True, stdout=subprocess.PIPE).stdout.readlines()]))

Happened in both Windows 10 and WSL2 Nvidia RTX 3090

opened by semel1 2

Owner

Yazhou XING

Ph.D. Candidate at HKUST

GitHub

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

5 Sep 16, 2022

Code for Blind Image Decomposition (BID) and Blind Image Decomposition network (BIDeN).

arXiv, porject page, paper Blind Image Decomposition (BID) Blind Image Decomposition is a novel task. The task requires separating a superimposed imag

64 Dec 20, 2022

A python software that can help blind people find things like laptops, phones, etc the same way a guide dog guides a blind person in finding his way.

GuidEye A python software that can help blind people find things like laptops, phones, etc the same way a guide dog guides a blind person in finding h

0 Aug 9, 2022

Self-Learned Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence

In this paper, we address the problem of rain streaks removal in video by developing a self-learned rain streak removal method, which does not require any clean groundtruth images in the training process.

44 Dec 6, 2022

Implementation of "Fast and Flexible Temporal Point Processes with Triangular Maps" (Oral @ NeurIPS 2020)

Fast and Flexible Temporal Point Processes with Triangular Maps This repository includes a reference implementation of the algorithms described in "Fa

20 Dec 2, 2022

Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

MTTS-CAN: Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement Paper Xin Liu, Josh Fromm, Shwetak Patel, Daniel M

106 Dec 30, 2022

PyTorch implementation for the visual prior component (i.e. perception module) of the Visually Grounded Physics Learner [Li et al., 2020].

VGPL-Visual-Prior PyTorch implementation for the visual prior component (i.e. perception module) of the Visually Grounded Physics Learner (VGPL). Give

8 Dec 29, 2022

[NeurIPS 2021] A weak-shot object detection approach by transferring semantic similarity and mask prior.

49 Jul 27, 2022

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis

Practical Blind Denoising via Swin-Conv-UNet and Data Synthesis [Paper] [Online Demo] The following results are obtained by our SCUNet with purely syn

312 Jan 7, 2023

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency[ECCV 2020]

Self-Supervised Monocular 3D Face Reconstruction by Occlusion-Aware Multi-view Geometry Consistency(ECCV 2020) This is an official python implementati

304 Jan 3, 2023

Unofficial implementation of "TTNet: Real-time temporal and spatial video analysis of table tennis" (CVPR 2020)

TTNet-Pytorch The implementation for the paper "TTNet: Real-time temporal and spatial video analysis of table tennis" An introduction of the project c

438 Dec 29, 2022

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

5 Jul 14, 2022

Source code for paper "Deep Superpixel-based Network for Blind Image Quality Assessment"

DSN-IQA Source code for paper "Deep Superpixel-based Network for Blind Image Quality Assessment" Requirements Python >=3.8.0 Pytorch >=1.7.1 Usage wit

7 Oct 13, 2022

Official implementation of Unfolded Deep Kernel Estimation for Blind Image Super-resolution.

Unfolded Deep Kernel Estimation for Blind Image Super-resolution Hongyi Zheng, Hongwei Yong, Lei Zhang, "Unfolded Deep Kernel Estimation for Blind Ima

15 Dec 26, 2022

Official PyTorch implementation of the paper "Deep Constrained Least Squares for Blind Image Super-Resolution", CVPR 2022.

Deep Constrained Least Squares for Blind Image Super-Resolution [Paper] This is the official implementation of 'Deep Constrained Least Squares for Bli

141 Dec 30, 2022

CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

63 Sep 27, 2022

Official Pytorch implementation of ICLR 2018 paper Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge.

Deep Learning for Physical Processes: Integrating Prior Scientific Knowledge: Official Pytorch implementation of ICLR 2018 paper Deep Learning for Phy

47 Nov 6, 2022

Dense Deep Unfolding Network with 3D-CNN Prior for Snapshot Compressive Imaging, ICCV2021 [PyTorch Code]

20 Oct 24, 2022

Code of the paper "Deep Human Dynamics Prior" in ACM MM 2021.

Code of the paper "Deep Human Dynamics Prior" in ACM MM 2021. Figure 1: In the process of motion capture (mocap), some joints or even the whole human

3 Oct 31, 2022