Asymmetric Bilateral Motion Estimation for Video Frame Interpolation, ICCV2021

Related tags

Deep Learning ABME
Overview

ABME (ICCV2021)

PWC PWC

Junheum Park, Chul Lee, and Chang-Su Kim

Official PyTorch Code for "Asymmetric Bilateral Motion Estimation for Video Frame Interpolation" [paper]

Requirements

  • PyTorch 1.7
  • CUDA 11.0
  • CuDNN 8.0.5
  • python 3.8

Installation

Create conda environment:

    $ conda create -n ABME python=3.8 anaconda
    $ conda activate ABME
    $ pip install opencv-python
    $ conda install pytorch==1.7 torchvision cudatoolkit=11.0 -c pytorch

Download repository:

    $ git clone https://github.com/JunHeum/ABME.git

Download pre-trained model parameters:

    $ unzip ABME_Weights.zip

Check your nvcc version:

    $ nvcc --version
  • To install correlation layer, you should match your nvcc version with cudatoolkit version of your conda environment. [nvcc_setting]

Install correlation layer:

    $ cd correlation_package
    $ python setup.py install

Quick Usage

Generate an intermediate frame on your pair of frames:

    $ python run.py --first images/im1.png --second images/im3.png --output images/im2.png

Test

  1. Download the datasets.
  2. Copy the path of the test dataset. (e.g., /hdd/vimeo_interp_test)
  3. Parse this path into the --dataset_root argument.
  4. (optional) You can ignore the --is_save. But, it yields a slightly different performance than evaluation on saved images.
    $ python test.py --name ABME --is_save --Dataset ucf101 --dataset_root /where/is/your/ucf101_dataset/path
    $ python test.py --name ABME --is_save --Dataset vimeo --dataset_root /where/is/your/vimeo_dataset/path
    $ python test.py --name ABME --is_save --Dataset SNU-FILM-all --dataset_root /where/is/your/FILM_dataset/path
    $ python test.py --name ABME --is_save --Dataset Xiph_HD --dataset_root /where/is/your/Xiph_dataset/path
    $ python test.py --name ABME --is_save --Dataset X4K1000FPS --dataset_root /where/is/your/X4K1000FPS_dataset/path

Experimental Results

We provide interpolated frames on test datasets for fast comparison or users with limited GPU memory. Especially, the test on X4K1000FPS requires at least 20GB of GPU memory.

Table

Train

We plan to share train codes soon!

Citation

Please cite the following paper if you feel this repository useful.

    @inproceedings{park2021ABME,
        author    = {Park, Junheum and Lee, Chul and Kim, Chang-Su}, 
        title     = {Asymmetric Bilateral Motion Estimation for Video Frame Interpolation}, 
        booktitle = {International Conference on Computer Vision},
        year      = {2021}
    }

License

See MIT License

Comments
  • About the asymmetric motion field in the paper

    About the asymmetric motion field in the paper

    企业微信截图_16417930957228 Hi, I am confused about the asymmetric motion field figure above, which you put in the paper. Since we only have t=0 and t=1, it is hard to judge whether the motion is symmetric or not. In my opinion, if the right foot in the above image is lower a little bit, then the motion will be symmetric and we don't have to take asymmetric motion field into account.

    opened by JasonSheng-atp 3
  • Census Loss Implementation for training

    Census Loss Implementation for training

    Hi @JunHeum, Thanks for the wonderful work! I'm trying to re-implement ABME, but I don not know how to implement census loss. Even if in the reference paper I cannot get enough information about it. Could you please kindly provide an implementation of census loss? Thanks a lot!

    opened by jzyustc 1
  • bugs

    bugs

    bicorr6 = self.bilateral_corr6(c16, c26, torch.zeros(c16.size(0), 2, c16.size(2), c16.size(3)).to(c16)) to bicorr6 = self.bilateral_corr6(c16, c26, torch.zeros(c16.size(0), 2, c16.size(2), c16.size(3)).to(c16.device)) ?

    opened by hiredd 1
  • How to test the model time comsuming?

    How to test the model time comsuming?

    I want to reproduce the 0.22s for 480*640 image reported by your paper, could you please provide some code? I implemented some code by myself but failed, about 2 times slower.

    opened by P2Oileen 0
  • SNU-FILM dataset

    SNU-FILM dataset

    We checked SNU-FILM dataset is not available in this link.

    Therefore, we share the SNU-FILM dataset until that link provides the dataset available.

    You should download two zip files. RGB image files Sequence index files

    opened by JunHeum 0
  • For arbitrary time t

    For arbitrary time t

    Hi, thanks for your awesome work.

    I have a question as an extension to Issue #8 . I understand that the provided pretrained model can be used recursively to produce results for more moments than t=0.5. However as far as I understood, according to the paper and your responses, it seems like the conclusion leads to: 'although the provided model is for predicting an intermediate frame t=0.5, it can be rebuilt for estimating frame at random t.'

    I thus tried to reproduce your model for arbitrary time t based on your code and figured that it doesn't seem possible to be extended for arbitrary time. Specifically, I doubt that Eq. 15 of the paper cannot hold for arbitrary time t. The pixel intensity seems to be intended to sum to 1. The equation holds as long as t=0.5, but for example, when t=0.3, M_t1=0, M_t0=1, the pixel intensity becomes 1.4 . I wanted to ask if there is anything that I misunderstood or got wrong... Please, correct me if I am wrong.

    opened by JHLew 0
  • About validation dataset

    About validation dataset

    Why use vimeo90k 'testlist.txt' provided for validation. As far as i know, when training the test dataset is not visible.

    And another question, what optimizer and lr_scheduler do you use when training? Can give some details? Thanks.

    opened by DachunKai 0
  • Question about the SBMNet.

    Question about the SBMNet.

    Hi, As what mentioned in the paper, the reliability map Z is computed by image where I think, Z is related to the difference about the interped frame in time t by V_(t->0) and V_(t->1). However, in the code,

    image where frame2_1 is compute by the flow SBM_ * (-1), which I think is flow from 0 to 1, not time t to 1.
    In '''frame2_1, Mask2_1 = warp(frame1_, SBM_ * (-1), return_mask=True)''', frame2_1 will be a warpped frame from frame1 to frame3, not frame2. Hope I have maken a clear statement of my question.

    opened by JerryLeolfl 1
  • how to interpolate at arbitrary time t

    how to interpolate at arbitrary time t

    Hi team,

    I think the paper mentions it is possible to interpolate at any time t. But the codes provided only perform a t=0.5 interpolation. Is it possible to modify the code and interpolate at a given time t? Thank you!

    opened by qianqianwang68 2
Owner
Junheum Park
BS: EE, Korea University Grad: EE, Korea University (Current)
Junheum Park
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation Ported from https://github.com/hzwer/arXiv2020-RIFE Dependencies NumPy

null 49 Jan 7, 2023
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE - Real Time Video Interpolation arXiv | YouTube | Colab | Tutorial | Demo Table of Contents Introduction Collection Usage Evaluation Training and

hzwer 3k Jan 4, 2023
RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation YouTube | BiliBili 16X interpolation results from two input images: Introd

旷视天元 MegEngine 28 Dec 9, 2022
This is the official repository of XVFI (eXtreme Video Frame Interpolation)

XVFI This is the official repository of XVFI (eXtreme Video Frame Interpolation), https://arxiv.org/abs/2103.16206 Last Update: 20210607 We provide th

Jihyong Oh 195 Dec 29, 2022
Repository relating to the CVPR21 paper TimeLens: Event-based Video Frame Interpolation

TimeLens: Event-based Video Frame Interpolation This repository is about the High Speed Event and RGB (HS-ERGB) dataset, used in the 2021 CVPR paper T

Robotics and Perception Group 544 Dec 19, 2022
an implementation of Revisiting Adaptive Convolutions for Video Frame Interpolation using PyTorch

revisiting-sepconv This is a reference implementation of Revisiting Adaptive Convolutions for Video Frame Interpolation [1] using PyTorch. Given two f

Simon Niklaus 59 Dec 22, 2022
An implementation of Video Frame Interpolation via Adaptive Separable Convolution using PyTorch

This work has now been superseded by: https://github.com/sniklaus/revisiting-sepconv sepconv-slomo This is a reference implementation of Video Frame I

Simon Niklaus 984 Dec 16, 2022
Video Frame Interpolation with Transformer (CVPR2022)

VFIformer Official PyTorch implementation of our CVPR2022 paper Video Frame Interpolation with Transformer Dependencies python >= 3.8 pytorch >= 1.8.0

DV Lab 63 Dec 16, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
Code of paper "CDFI: Compression-Driven Network Design for Frame Interpolation", CVPR 2021

CDFI (Compression-Driven-Frame-Interpolation) [Paper] (Coming soon...) | [arXiv] Tianyu Ding*, Luming Liang*, Zhihui Zhu, Ilya Zharkov IEEE Conference

Tianyu Ding 95 Dec 4, 2022
A motion tracking system for any arbitaray points in a video frame.

PointTracking This code is written by Majid Masoumi @ [email protected] I have used lucas kanade optical flow technique to track the points b

Dr. Majid Masoumi 1 Feb 9, 2022
Speech Enhancement Generative Adversarial Network Based on Asymmetric AutoEncoder

ASEGAN: Speech Enhancement Generative Adversarial Network Based on Asymmetric AutoEncoder 中文版简介 Readme with English Version 介绍 基于SEGAN模型的改进版本,使用自主设计的非

Nitin 53 Nov 17, 2022
(SIGIR2020) “Asymmetric Tri-training for Debiasing Missing-Not-At-Random Explicit Feedback’’

Asymmetric Tri-training for Debiasing Missing-Not-At-Random Explicit Feedback About This repository accompanies the real-world experiments conducted i

yuta-saito 19 Dec 1, 2022
the code for our CVPR 2021 paper Bilateral Grid Learning for Stereo Matching Network [BGNet]

BGNet This repository contains the code for our CVPR 2021 paper Bilateral Grid Learning for Stereo Matching Network [BGNet] Environment Python 3.6.* C

3DCV developer 87 Nov 29, 2022
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Code of Adverse Weather Image Translation with Asymmetric and Uncertainty aware GAN

Adverse Weather Image Translation with Asymmetric and Uncertainty-aware GAN (AU-GAN) Official Tensorflow implementation of Adverse Weather Image Trans

Jeong-gi Kwak 36 Dec 26, 2022
PyTorch implementation of Asymmetric Siamese (https://arxiv.org/abs/2204.00613)

Asym-Siam: On the Importance of Asymmetry for Siamese Representation Learning This is a PyTorch implementation of the Asym-Siam paper, CVPR 2022: @inp

Meta Research 89 Dec 18, 2022
BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

Bilateral Denoising Diffusion Models (BDDMs) This is the official PyTorch implementation of the following paper: BDDM: BILATERAL DENOISING DIFFUSION M

null 172 Dec 23, 2022
Official implementation of the network presented in the paper "M4Depth: A motion-based approach for monocular depth estimation on video sequences"

M4Depth This is the reference TensorFlow implementation for training and testing depth estimation models using the method described in M4Depth: A moti

Michaël Fonder 76 Jan 3, 2023