Self-Learned Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence

Related tags

Deep Learning CVPR-2020-Self-Rain-Removal
Overview

Self-Learned Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence (CVPR'2020)

Wenhan Yang, Robby T. Tan, Shiqi Wang, and Jiaying Liu

[Paper Link] [Project Page] [Slides](TBA)[Video](TBA) (CVPR'2020 Poster)

Abstract

In this paper, we address the problem of rain streaks removal in video by developing a self-learned rain streak removal method, which does not require any clean groundtruth images in the training process. The method is inspired by fact that the adjacent frames are highly correlated and can be regarded as different versions of identical scene, and rain streaks are randomly distributed along the temporal dimension. With this in mind, we construct a two-stage Self-Learned Deraining Network (SLDNet) to remove rain streaks based on both temporal correlation and consistency. In the first stage, SLDNet utilizes the temporal correlations and learns to predict the clean version of the current frame based on its adjacent rain video frames. In the second stage, SLDNet enforces the temporal consistency among different frames. It takes both the current rain frame and adjacent rain video frames to recover the structural details. The first stage is responsible for reconstructing main structures, and the second stage is responsible for extracting structural details. We build our network architecture with two sub-tasks, i.e. motion estimation and rain region detection, and optimize them jointly. Our extensive experiments demonstrate the effectiveness of our method, offering better results both quantitatively and qualitatively.

If you find the resource useful, please cite the following :- )

@InProceedings{Yang_2020_CVPR,
author = {Yang, Wenhan and Tan, Robby T. and Wang, Shiqi and Liu, Jiaying},
title = {Self-Learning Video Rain Streak Removal: When Cyclic Consistency Meets Temporal Correspondence},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

## Prerequisites - Linux or macOS - Python 3 - NVIDIA GPU + CUDA cuDNN - PyTorch 0.4

Installation

  1. Clone this repo;
  2. Install PyTorch and dependencies from http://pytorch.org;
  3. Download FLowNet V2.0 from https://pan.baidu.com/s/14xPBvYcnGjAJ2adsQOVKhA (extracted code: 3is9). Put the file FlowNet2_checkpoint.pth.tar into SLDNet_code/pretrained_models/FlowNet2_checkpoint.pth.tar;
  4. Download NTURain Dataset (Only including b1_Rain) from https://pan.baidu.com/s/1nsBl6uhj-MWVgr1uBcsy1w (extraced code:rufg). For other sequences, please download them from https://github.com/hotndy/SPAC-SupplementaryMaterials. Unzip b1_Rain.zip and put the file into SLDNet_code/data_NTU/train/b1_Rain/ and SLDNet_code/data_NTU/test/b1_Rain/.

Train

cd SLDNet_code; sh train_video_rain.sh;

Test

cd SLDNet_code; sh test_video_rain.sh;

Contact

If you have questions, you can contact [email protected].

Comments
  • Some questions about testing

    Some questions about testing

    Hello, your work is very interesting and it has helped me a lot. I have trained 100 epochs according to your source code. Now I am ready to test it. I have some questions I want to ask you

    1. You trained 100 epochs, But when testing, you used the model after three epochs. Why is this?
    2. What testset did you choose? Is it all of the training set? Hope to get your reply as soon as possible, thank you.
    opened by booker-max 2
  • Some issues with missing files

    Some issues with missing files

    Hello, your work is great and gives me a lot of inspiration but when I run train.py, it gives me the following error: No such file or directory: '/home/yangwenhan/.torch/models/vgg16-397923af.pth' Can you share this pre-trained model? Can you share this pre-trained model?

    opened by booker-max 0
  • Please upload the pretrained model

    Please upload the pretrained model

    The uploaded pretrained model only contains the parameters of FlowNet2.0. Such an excellent method will be of great help to people who study the same subject. Please upload the complete pretrained model. Thank you.

    opened by z-xf 0
  • About the model after training

    About the model after training

    Hi, author~ Add the reference of neighbor frames into consideration and repair it in a temporal manner is really a cool thing. I'd like to finish the test phase, however, I can't get the pretrained model yet. Could you please share it with us?

    opened by Ssakura-go 0
  • Link to pretrained models is not working

    Link to pretrained models is not working

    Hi authors, thanks for your paper and code. Actually the link provided in this repo for pretrained models is not working. Please change the link. Thanks

    opened by EkjotSingh898 0
  • An error in networks.resample2d_package

    An error in networks.resample2d_package

    when i run the codes , I face the problem.

    File "train_video_rain.py", line 17, in from networks.resample2d_package.modules.resample2d import Resample2d File "/cache/sldnet_code/networks/init.py", line 1, in from .FlowNet2 import * File "/cache/sldnet_code/networks/FlowNet2.py", line 8, in from networks.resample2d_package.modules.resample2d import Resample2d File "/cache/sldnet_code/networks/resample2d_package/modules/resample2d.py", line 3, in from ..functions.resample2d import Resample2dFunction File "/cache/sldnet_code/networks/resample2d_package/functions/resample2d.py", line 3, in from .._ext import resample2d File "/cache/sldnet_code/networks/resample2d_package/_ext/resample2d/init.py", line 3, in from ._resample2d import lib as _lib, ffi as _ffi ModuleNotFoundError: No module named 'networks.resample2d_package._ext.resample2d._resample2d'

    It seems that the _resample2d.py is lost. Could you help me check these problem. Thank you.

    opened by laulampaul 5
Owner
Yang Wenhan
Yang Wenhan
GitHub
RCDNet: A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020)

RCDNet: A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020) Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng [PDF] [Supplementary M

Hong Wang 6 Sep 27, 2022
MPRNet-Cloud-removal: Progressive cloud removal

MPRNet-Cloud-removal Progressive cloud removal Requirements 1.Pytorch >= 1.0 2.Python 3 3.NVIDIA GPU + CUDA 9.0 4.Tensorboard Installation 1.Clone the

Semi 95 Dec 18, 2022
[NeurIPS 2020] Blind Video Temporal Consistency via Deep Video Prior

pytorch-deep-video-prior (DVP) Official PyTorch implementation for NeurIPS 2020 paper: Blind Video Temporal Consistency via Deep Video Prior TensorFlo

Yazhou XING 90 Oct 19, 2022
git git《Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking》(CVPR 2021) GitHub:git2] 《Masksembles for Uncertainty Estimation》(CVPR 2021) GitHub:git3]

Transformer Meets Tracker: Exploiting Temporal Context for Robust Visual Tracking Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li Accepted by CVPR

NingWang 236 Dec 22, 2022
Learning Correspondence from the Cycle-consistency of Time (CVPR 2019)

TimeCycle Code for Learning Correspondence from the Cycle-consistency of Time (CVPR 2019, Oral). The code is developed based on the PyTorch framework,

Xiaolong Wang 706 Nov 29, 2022
[ICLR 2021] "CPT: Efficient Deep Neural Network Training via Cyclic Precision" by Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin

CPT: Efficient Deep Neural Network Training via Cyclic Precision Yonggan Fu, Han Guo, Meng Li, Xin Yang, Yining Ding, Vikas Chandra, Yingyan Lin Accep

null 26 Oct 25, 2022
Code release for SLIP Self-supervision meets Language-Image Pre-training

SLIP: Self-supervision meets Language-Image Pre-training What you can find in this repo: Pre-trained models (with ViT-Small, Base, Large) and code to

Meta Research 621 Dec 31, 2022
Semi-supervised Video Deraining with Dynamical Rain Generator (CVPR, 2021, Pytorch)

S2VD Semi-supervised Video Deraining with Dynamical Rain Generator (CVPR, 2021) Requirements and Dependencies Ubuntu 16.04, cuda 10.0 Python 3.6.10, P

Zongsheng Yue 53 Nov 23, 2022
Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Zhengzhong Tu 5 Sep 16, 2022
Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

This repository is the official PyTorch implementation of Cascaded Deep Video Deblurring Using Temporal Sharpness Prior and Non-local Spatial-Temporal Similarity

hippopmonkey 4 Dec 11, 2022
Self-Supervised Methods for Noise-Removal

SSMNR | Self-Supervised Methods for Noise Removal Image denoising is the task of removing noise from an image, which can be formulated as the task of

null 1 Jan 16, 2022
PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization using Augmented-Self Reference and Dense Semantic Correspondence) and pre-trained model on ImageNet dataset

Reference-Based-Sketch-Image-Colorization-ImageNet This is a PyTorch implementation of CVPR 2020 paper (Reference-Based Sketch Image Colorization usin

Yuzhi ZHAO 11 Jul 28, 2022
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

null 75 Nov 24, 2022
This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking".

SCT This is the official code for the paper "Tracker Meets Night: A Transformer Enhancer for UAV Tracking" The spatial-channel Transformer (SCT) enhan

Intelligent Vision for Robotics in Complex Environment 27 Nov 23, 2022
Pythonic particle-based (super-droplet) warm-rain/aqueous-chemistry cloud microphysics package with box, parcel & 1D/2D prescribed-flow examples in Python, Julia and Matlab

PySDM PySDM is a package for simulating the dynamics of population of particles. It is intended to serve as a building block for simulation systems mo

Atmospheric Cloud Simulation Group @ Jagiellonian University 32 Oct 18, 2022
ConvMAE: Masked Convolution Meets Masked Autoencoders

ConvMAE ConvMAE: Masked Convolution Meets Masked Autoencoders Peng Gao1, Teli Ma1, Hongsheng Li2, Jifeng Dai3, Yu Qiao1, 1 Shanghai AI Laboratory, 2 M

Alpha VL Team of Shanghai AI Lab 345 Jan 8, 2023
CVPR2021: Temporal Context Aggregation Network for Temporal Action Proposal Refinement

Temporal Context Aggregation Network - Pytorch This repo holds the pytorch-version codes of paper: "Temporal Context Aggregation Network for Temporal

Zhiwu Qing 63 Sep 27, 2022
Self-supervised Augmentation Consistency for Adapting Semantic Segmentation (CVPR 2021)

Self-supervised Augmentation Consistency for Adapting Semantic Segmentation This repository contains the official implementation of our paper: Self-su

Visual Inference Lab @TU Darmstadt 132 Dec 21, 2022
the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

G2S This is the official code for ICRA 2021 Paper: Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation by Hemang

NeurAI 4 Jul 27, 2022