Unsupervised Learning of Multi-Frame Optical Flow with Occlusions

Overview

This is a Pytorch implementation of

Janai, J., Güney, F., Ranjan, A., Black, M. and Geiger, A., Unsupervised Learning of Multi-Frame Optical Flow with Occlusions. ECCV 2018.

[Link to Paper] [Project Page] [Original Torch Code]

Requirements

  • Runs and tested on Pytorch 0.3.1, it should be compatible with higher versions with little/no modifications.
  • Correlation package is taken from NVIDIA/flownet2-pytorch and it can be installed using
cd correlation_package
bash make.sh

If you are using Pytorch>0.3.1, you can use correlation layer from here.

Usage

To use the model, go to your favorite python environment

from back2future import Model
model = Model(pretrained='pretrained/path_to_your_favorite_model')

There are two pretrained models in pretrained/, that are fine tuned on Sintel and KITTI in an unsupervised way.

Refer to demo.py for more.

Testing

To test performance on KITTI, use

python3 test_back2future.py --pretrained-flow path/to/pretrained/model --kitti-dir path/to/kitti/2015/root

Training

Please use the [original torch code] for training new models.

License

This is a reimplementation. License for the original work can be found at JJanai/back2future.

While using this code, please cite

@inproceedings{Janai2018ECCV,
  title = {Unsupervised Learning of Multi-Frame Optical Flow with Occlusions },
  author = {Janai, Joel and G{"u}ney, Fatma and Ranjan, Anurag and Black, Michael J. and Geiger, Andreas},
  booktitle = {European Conference on Computer Vision (ECCV)},
  volume = {Lecture Notes in Computer Science, vol 11220},
  pages = {713--731},
  publisher = {Springer, Cham},
  month = sep,
  year = {2018},
  month_numeric = {9}
}
Comments
  • Non-shared weights for network branches?

    Non-shared weights for network branches?

    Hi,

    thanks for sharing the code. I have a generic question w.r.t the model's code. I see that you have used three (non-shared weights) branches for 3 images in the lines: https://github.com/anuragranj/back2future.pytorch/blob/master/back2future.py#L147-L166

    In your experience, do you suggest having non-shared weights for image encoding? I have noticed that PWCNet has shared weights for these branches.

    I understand you did not train with pytorch. So, I am not sure if you have shared weights when you trained with Lua torch.

    opened by InnovArul 6
  • Question on the argument for backword warping

    Question on the argument for backword warping

    Hi,

    In your Pytorch implementation, I don't understand why you use the forward upsampled flow for backward warping. should it be the backward upsampled flow? (warping from the past feature (feat5c) into the current frame should use the estimated flow ( past->curr) )

    for example in your code of "back2future.py"

            feat5b_warped = self.warp(feat5b, 0.625*flow6_fwd_up)
            feat5c_warped = self.warp(feat5c, -0.625*flow6_fwd_up) # Q: flow6_fwd_up-> flow6_bwd_up?
    

    It would be great if you answer my question.

    Thank you, Suhong

    opened by suhongkim 2
  • Some problems about the prediction of occlusion

    Some problems about the prediction of occlusion

    Thanks for your great job!

    I try to binary the occlusion by using nn.sigmoid and np.round, but it doesn't seem to be the right way.

    I want to know what is the right way to visualize occlusion? Or can you send to me the files which occlusion visualization on KITTI, Sintel and Middlebury if you have them? I shall be very grateful.:blush:

    opened by Ecalpal 1
  • Confusing of the occlusion map

    Confusing of the occlusion map

    Thanks for your sharing~!

    After read your paper, I noticed that, the occlusion estimation of your work is mapped to (0,1), and the occlusion estimation is used as the weight of photometric loss. While in your paper, the occlusion estimation in Fig.3 is something like a binary map, and in the last of second paragraph of section 4.2, you said 'This clearly shows the benefit of ignoring misleading information in accordance with the occlusion estimates'. It is a little confusing to me. Is the occlusion estimation transformed into binary map? Or you just transform occlusion maps for visualizing?

    Any reply will be appreciated~ Thanks again.

    opened by lelelexxx 1
  • questions regarding multiplicative constant for flow & correlation max_displacement in multi-levels

    questions regarding multiplicative constant for flow & correlation max_displacement in multi-levels

    I am trying to understand the code and the design decisions made while designing this model. In this regard, I have two doubts as below:

    1. The correlation max_displacement is kept constant at 4 for all the levels of feature maps. https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L73

    In your opinion, do you think we have to have different max_displacement for different levels of the network?

    1. I would like to get an intuition regarding the multiplication factors used for flow: 0.625 for level-6 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L178-L179

    1.25 for level-5 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L196-L197

    2.5 for level-4 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L213-L214

    5 for level-3 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L230-L231

    Could you please let me know why do we need these multiplicative factors? I am sorry if this is a basic question to be asked. As the network is learnable, isn't it possible that the network learns this multiplicative factor automatically as well?

    Thanks in advance!

    opened by InnovArul 0
  • How can I fix 'back2future_kitti.pth.tar' error?

    How can I fix 'back2future_kitti.pth.tar' error?

    After making the necessary institutions, I ran demo.py. The error I encountered is this> FileNotFoundError: [Errno 2] No such file or directory: 'pretrained / back2future_kitti.pth.tar'. When I look at it with normal access it gives an error > an error occurred while loading the archive. Is the error caused by the file's corruption?

    Secondly, the output I got when I ran the line you gave as test > python3 test_back2future.py --pretrained-flow path / to / pretrained / model --kitti-dir path / to / kitti / 2015 / root Traceback (most recent call last):    File "test_back2future.py", line 8, in      from path import Path ImportError: cannot import name 'Path'

    What solution can I apply for this? Thank you from now. @anuragranj @JJanai

    opened by ahmetybilgin 3
UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss This repository contains the TensorFlow implementation of the paper UnF

Simon Meister 270 Nov 6, 2022
Self-Supervised Multi-Frame Monocular Scene Flow (CVPR 2021)

Self-Supervised Multi-Frame Monocular Scene Flow 3D visualization of estimated depth and scene flow (overlayed with input image) from temporally conse

Visual Inference Lab @TU Darmstadt 85 Dec 22, 2022
[CVPR 2022] Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions" paper

template-pose Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions

Van Nguyen Nguyen 92 Dec 28, 2022
Demo code for paper "Learning optical flow from still images", CVPR 2021.

Depthstillation Demo code for "Learning optical flow from still images", CVPR 2021. [Project page] - [Paper] - [Supplementary] This code is provided t

null 130 Dec 25, 2022
Learning Optical Flow from a Few Matches (CVPR 2021)

Learning Optical Flow from a Few Matches This repository contains the source code for our paper: Learning Optical Flow from a Few Matches CVPR 2021 Sh

Shihao Jiang (Zac) 159 Dec 16, 2022
A project that uses optical flow and machine learning to detect aimhacking in video clips.

waldo-anticheat A project that aims to use optical flow and machine learning to visually detect cheating or hacking in video clips from fps games. Che

waldo.vision 542 Dec 3, 2022
A project that uses optical flow and machine learning to detect aimhacking in video clips.

waldo-anticheat A project that aims to use optical flow and machine learning to visually detect cheating or hacking in video clips from fps games. Che

waldo.vision 383 Nov 1, 2021
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation Ported from https://github.com/hzwer/arXiv2020-RIFE Dependencies NumPy

null 49 Jan 7, 2023
RIFE: Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE - Real Time Video Interpolation arXiv | YouTube | Colab | Tutorial | Demo Table of Contents Introduction Collection Usage Evaluation Training and

hzwer 3k Jan 4, 2023
RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation

RIFE - Real-Time Intermediate Flow Estimation for Video Frame Interpolation YouTube | BiliBili 16X interpolation results from two input images: Introd

旷视天元 MegEngine 28 Dec 9, 2022
A lightweight deep network for fast and accurate optical flow estimation.

FastFlowNet: A Lightweight Network for Fast Optical Flow Estimation The official PyTorch implementation of FastFlowNet (ICRA 2021). Authors: Lingtong

Tone 161 Jan 3, 2023
Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks

flownet2-pytorch Pytorch implementation of FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. Multiple GPU training is supported, a

NVIDIA Corporation 2.8k Dec 27, 2022
a reimplementation of Optical Flow Estimation using a Spatial Pyramid Network in PyTorch

pytorch-spynet This is a personal reimplementation of SPyNet [1] using PyTorch. Should you be making use of this work, please cite the paper according

Simon Niklaus 269 Jan 2, 2023
A fast model to compute optical flow between two input images.

DCVNet: Dilated Cost Volumes for Fast Optical Flow This repository contains our implementation of the paper: @InProceedings{jiang2021dcvnet, title={

Huaizu Jiang 8 Sep 27, 2021
Demo code for ICCV 2021 paper "Sensor-Guided Optical Flow"

Sensor-Guided Optical Flow Demo code for "Sensor-Guided Optical Flow", ICCV 2021 This code is provided to replicate results with flow hints obtained f

null 10 Mar 16, 2022
MMFlow is an open source optical flow toolbox based on PyTorch

Documentation: https://mmflow.readthedocs.io/ Introduction English | 简体中文 MMFlow is an open source optical flow toolbox based on PyTorch. It is a part

OpenMMLab 688 Jan 6, 2023
Traditional deepdream with VQGAN+CLIP and optical flow. Ready to use in Google Colab

VQGAN-CLIP-Video cat.mp4 policeman.mp4 schoolboy.mp4 forsenBOG.mp4

null 23 Oct 26, 2022
[CVPR 2022] Deep Equilibrium Optical Flow Estimation

Deep Equilibrium Optical Flow Estimation This is the official repo for the paper Deep Equilibrium Optical Flow Estimation (CVPR 2022), by Shaojie Bai*

CMU Locus Lab 136 Dec 18, 2022