An implementation of Video Frame Interpolation via Adaptive Separable Convolution using PyTorch

Overview

This work has now been superseded by: https://github.com/sniklaus/revisiting-sepconv

sepconv-slomo

This is a reference implementation of Video Frame Interpolation via Adaptive Separable Convolution [1] using PyTorch. Given two frames, it will make use of adaptive convolution [2] in a separable manner to interpolate the intermediate frame. Should you be making use of our work, please cite our paper [1].

Paper

For a reimplemntation of our work, see: https://github.com/martkartasev/sepconv
And for another adaptation, consider: https://github.com/HyeongminLEE/pytorch-sepconv
For softmax splatting, please see: https://github.com/sniklaus/softmax-splatting

setup

The separable convolution layer is implemented in CUDA using CuPy, which is why CuPy is a required dependency. It can be installed using pip install cupy or alternatively using one of the provided binary packages as outlined in the CuPy repository.

If you plan to process videos, then please also make sure to have pip install moviepy installed.

usage

To run it on your own pair of frames, use the following command. You can either select the l1 or the lf model, please see our paper for more details. In short, the l1 model should be used for quantitative evaluations and the lf model for qualitative comparisons.

python run.py --model lf --one ./images/one.png --two ./images/two.png --out ./out.png

To run in on a video, use the following command.

python run.py --model lf --video ./videos/car-turn.mp4 --out ./out.mp4

For a quick benchmark using examples from the Middlebury benchmark for optical flow, run python benchmark.py. You can use it to easily verify that the provided implementation runs as expected.

video

Video

license

The provided implementation is strictly for academic purposes only. Should you be interested in using our technology for any commercial use, please feel free to contact us.

references

[1]  @inproceedings{Niklaus_ICCV_2017,
         author = {Simon Niklaus and Long Mai and Feng Liu},
         title = {Video Frame Interpolation via Adaptive Separable Convolution},
         booktitle = {IEEE International Conference on Computer Vision},
         year = {2017}
     }
[2]  @inproceedings{Niklaus_CVPR_2017,
         author = {Simon Niklaus and Long Mai and Feng Liu},
         title = {Video Frame Interpolation via Adaptive Convolution},
         booktitle = {IEEE Conference on Computer Vision and Pattern Recognition},
         year = {2017}
     }

acknowledgment

This work was supported by NSF IIS-1321119. The video above uses materials under a Creative Common license or with the owner's permission, as detailed at the end.

Comments
  • Perceptual loss

    Perceptual loss

    Hi, in your paper you mentioned the use of perceptual loss, did you apply any pre-processing to the training set as basically I'm getting a contrast issue despite performing normalisation using the imageNet mean and standard deviation values. I've tried it without doing this, but the output is more or less the same I'd appreciate it if you had any ideas as to where I'm going wrong in this case Thanks

    opened by issakh 42
  • Error in run.py

    Error in run.py

    When I try to run the python file run.py after using downloading the bash file, I get the following error message:

    Traceback (most recent call last): File "run.py", line 172, in moduleNetwork = Network().cuda().eval() File "run.py", line 124, in init self.load_state_dict(torch.load('./network-' + arguments_strModel + '.pytorch')) File "/home/souvik207/.local/lib/python2.7/site-packages/torch/serialization.py", line 356, in load f = open(f, 'rb') IOError: [Errno 2] No such file or directory: './network-lf--first.pytorch'

    What could be causing this? Is there a way to overcome this error?

    opened by Souvik207 13
  • No module named '_ext'

    No module named '_ext'

    Applause to everyone involved in making this, it is amazing.

    Anyway this is the first time I'm using python, please be gentle. When I try to run "run.py" I get this following error:

    Traceback (most recent call last): File "run.py", line 13, in from SeparableConvolution import SeparableConvolution # the custom SeparableConvolution layer File "D:\Users-\Desktop\Sepconv\pytorch-sepconv-master\SeparableConvolution.py", line 3, in import _ext.cunnex ModuleNotFoundError: No module named '_ext'

    Not sure what I'm missing, I'm using python 3.6.

    opened by 9Ginseng 11
  • Need your help to analyze the reason

    Need your help to analyze the reason

    Hello, I divided a complete video into odd-frame video and even-frame video, using odd frames to interpolate, then I converted the YUV video losslessly into mp4 format and interpolated it, and converted the interpolated mp4 into YUV format to calculate the PSNR result, but this effect is not very rational, I have not yet figured out the reason, What is your opinion?

    opened by LINSEVENQI 8
  • Out of memory... how to increase memory?

    Out of memory... how to increase memory?

    So I got it to work (I guess) but I'm having an issue with larger images. Anything below 720p works fine, but at or above 720p it gives me an out of memory error. Is there a way to extend this? I don't mind if it takes longer to process; or would I have to get a GPU with more VRAM? My current one is a 970gtx with 3.5GB of VRAM. Or would high VRAM not make any difference?

    opened by deama 6
  • "ModuleNotFoundError: no module named torch.utils.serialization"

    Hello,

    I've tried to run the run.py file with my anaconda pytorch setup, however it gives me the: "ModuleNotFoundError: no module named torch.utils.serialization" error. I can't seem to find any information on how to fix this or really where that serialization package is from.

    opened by deama 6
  • How to generate the Separable kernels?

    How to generate the Separable kernels?

    Hi, thank you for your idea and codes. But I could not understand the Separable kernel. The kernels mentioned in the paper are 1D, but in my opinion, the output of the sub-network should be 51x128x128. I am very confused

    opened by wendyWeng 6
  • error in run.py

    error in run.py

    Traceback (most recent call last): File "run.py", line 172, in moduleNetwork = Network().cuda().eval() File "run.py", line 124, in init self.load_state_dict(torch.load('network-' + arguments_strModel + '.pytorch')) File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 358, in load return _load(f, map_location, pickle_module) File "/home/ubuntu/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 532, in _load magic_number = pickle_module.load(f) _pickle.UnpicklingError: invalid load key, '<'.

    When I run the file run.py, the error above occurred. Hope someone can help me.

    opened by SanSLee 5
  • problem with compilation

    problem with compilation

    I installed in ubuntu16.04. When I run thebash install.bash , i get the following error.

    huwenchao@huwenchao-B350GT3:~/cv/pytorch-sepconv-master$ bash install.bashIn file included from /home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/lib/include/THC/THCGeneral.h:5:0,
                     from /home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/lib/include/THC/THC.h:4,
                     from src/SeparableConvolution_kernel.cu:1:
    /home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/lib/include/TH/THAllocator.h:6:28: fatal error: ATen/Allocator.h: No such file or directory
     #include <ATen/Allocator.h> ^
    compilation terminated.
    gcc: error: /home/huwenchao/cv/pytorch-sepconv-master/src/SeparableConvolution_kernel.o: No such file or directory
    Traceback (most recent call last):
      File "install.py", line 32, in <module>
        objectExtension.build()
      File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 189, in build
        _build_extension(ffi, cffi_wrapper_name, target_dir, verbose)
      File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/torch/utils/ffi/__init__.py", line 111, in _build_extension
        outfile = ffi.compile(tmpdir=tmpdir, verbose=verbose, target=libname)
      File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/api.py", line 697, in compile
        compiler_verbose=verbose, debug=debug, **kwds)
      File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/recompiler.py", line 1520, in recompile
        compiler_verbose, debug)
      File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/ffiplatform.py", line 22, in compile
        outputfilename = _build(tmpdir, ext, compiler_verbose, debug)
      File "/home/huwenchao/anaconda2/lib/python2.7/site-packages/cffi/ffiplatform.py", line 58, in _build
        raise VerificationError('%s: %s' % (e.__class__.__name__, e))
    cffi.error.VerificationError: LinkError: command 'gcc' failed with exit status 1
    

    I am a novice to this,please help me.

    opened by pczebra 5
  • The training videos

    The training videos

    Hi, thanks for your sharing. I like your work. Recently I want to retrain your model, and I download some YouTube videos. But I train the model which can not achieve the PSNR 35.73 dB in the Middlebury. So can you offer your training videos list? Thanks for you very much!

    opened by linchuming 4
  • RuntimeError: tried to construct a tensor from a nested float sequence, but found an item of type numpy.float32

    RuntimeError: tried to construct a tensor from a nested float sequence, but found an item of type numpy.float32

    I am using Ubuntu 17.04 with an Nvidia GTX 980 and CUDA 9 When I try to run the run.py script I get the following error:

    alex@alex-desktop:~/pytorch-sepconv$ python run.py --model lf --first ./images/first.png --second ./images/second.png --out ./result.png
    Traceback (most recent call last):
      File "run.py", line 167, in <module>
        tensorInputFirst = torch.FloatTensor(numpy.rollaxis(numpy.asarray(PIL.Image.open(arguments_strFirst))[:,:,::-1], 2, 0).astype(numpy.float32) / 255.0)
    RuntimeError: tried to construct a tensor from a nested float sequence, but found an item of type numpy.float32 at index (0, 0, 0)
    
    opened by alex47 4
Owner
Simon Niklaus
Research Scientist at Adobe
Simon Niklaus
FLAVR is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction

FLAVR is a fast, flow-free frame interpolation method capable of single shot multi-frame prediction. It uses a customized encoder decoder architecture with spatio-temporal convolutions and channel gating to capture and interpolate complex motion trajectories between frames to generate realistic high frame rate videos. This repository contains original source code for the paper accepted to CVPR 2021.

Tarun K 280 Dec 23, 2022
This is the official repository of XVFI (eXtreme Video Frame Interpolation)

XVFI This is the official repository of XVFI (eXtreme Video Frame Interpolation), https://arxiv.org/abs/2103.16206 Last Update: 20210607 We provide th

Jihyong Oh 195 Dec 29, 2022
Repository relating to the CVPR21 paper TimeLens: Event-based Video Frame Interpolation

TimeLens: Event-based Video Frame Interpolation This repository is about the High Speed Event and RGB (HS-ERGB) dataset, used in the 2021 CVPR paper T

Robotics and Perception Group 544 Dec 19, 2022
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation Ported from https://github.com/hzwer/arXiv2020-RIFE Dependencies NumPy

null 49 Jan 7, 2023
Asymmetric Bilateral Motion Estimation for Video Frame Interpolation, ICCV2021

ABME (ICCV2021) Junheum Park, Chul Lee, and Chang-Su Kim Official PyTorch Code for "Asymmetric Bilateral Motion Estimation for Video Frame Interpolati

Junheum Park 86 Dec 28, 2022
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE - Real Time Video Interpolation arXiv | YouTube | Colab | Tutorial | Demo Table of Contents Introduction Collection Usage Evaluation Training and

hzwer 3k Jan 4, 2023
RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation YouTube | BiliBili 16X interpolation results from two input images: Introd

旷视天元 MegEngine 28 Dec 9, 2022
Video Frame Interpolation with Transformer (CVPR2022)

VFIformer Official PyTorch implementation of our CVPR2022 paper Video Frame Interpolation with Transformer Dependencies python >= 3.8 pytorch >= 1.8.0

DV Lab 63 Dec 16, 2022
SE3 Pose Interp - Interpolate camera pose or trajectory in SE3, pose interpolation, trajectory interpolation

SE3 Pose Interpolation Pose estimated from SLAM system are always discrete, and

Ran Cheng 4 Dec 15, 2022
Code of paper "CDFI: Compression-Driven Network Design for Frame Interpolation", CVPR 2021

CDFI (Compression-Driven-Frame-Interpolation) [Paper] (Coming soon...) | [arXiv] Tianyu Ding*, Luming Liang*, Zhihui Zhu, Ilya Zharkov IEEE Conference

Tianyu Ding 95 Dec 4, 2022
This is the official implementation of the paper "Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation".

[CVPRW 2021] - Object Propagation via Inter-Frame Attentions for Temporally Stable Video Instance Segmentation

Anirudh S Chakravarthy 6 May 3, 2022
TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Prediction.

TalkNet 2 [WIP] TalkNet 2: Non-Autoregressive Depth-Wise Separable Convolutional Model for Speech Synthesis with Explicit Pitch and Duration Predictio

Rishikesh (ऋषिकेश) 69 Dec 17, 2022
DeepLabv3+:Encoder-Decoder with Atrous Separable Convolution语义分割模型在tensorflow2当中的实现

DeepLabv3+:Encoder-Decoder with Atrous Separable Convolution语义分割模型在tensorflow2当中的实现 目录 性能情况 Performance 所需环境 Environment 注意事项 Attention 文件下载 Download

Bubbliiiing 31 Nov 25, 2022
【Arxiv】Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution

SANet Exploring Separable Attention for Multi-Contrast MR Image Super-Resolution Dependencies numpy==1.18.5 scikit_image==0.16.2 torchvision==0.8.1 to

null 36 Jan 5, 2023
(CVPR 2021) PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds

PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Clouds by Mutian Xu*, Runyu Ding*, Hengshuang Zhao, and Xiaojuan Qi. Int

CVMI Lab 228 Dec 25, 2022
Adaptive Graph Convolution for Point Cloud Analysis

Adaptive Graph Convolution for Point Cloud Analysis This repository contains the implementation of AdaptConv for point cloud analysis. Adaptive Graph

null 64 Dec 21, 2022
Unsupervised Video Interpolation using Cycle Consistency

Unsupervised Video Interpolation using Cycle Consistency Project | Paper | YouTube Unsupervised Video Interpolation using Cycle Consistency Fitsum A.

NVIDIA Corporation 100 Nov 30, 2022
ICLR 2021, Fair Mixup: Fairness via Interpolation

Fair Mixup: Fairness via Interpolation Training classifiers under fairness constraints such as group fairness, regularizes the disparities of predicti

Ching-Yao Chuang 49 Nov 22, 2022
NeRViS: Neural Re-rendering for Full-frame Video Stabilization

Neural Re-rendering for Full-frame Video Stabilization

Yu-Lun Liu 9 Jun 17, 2022