Unsupervised Learning of Multi-Frame Optical Flow with Occlusions

Anurag Ranjan

Last update: Nov 2, 2022

Related tags

Overview

This is a Pytorch implementation of

Janai, J., Güney, F., Ranjan, A., Black, M. and Geiger, A., Unsupervised Learning of Multi-Frame Optical Flow with Occlusions. ECCV 2018.

[Link to Paper] [Project Page] [Original Torch Code]

Requirements

Runs and tested on Pytorch 0.3.1, it should be compatible with higher versions with little/no modifications.
Correlation package is taken from NVIDIA/flownet2-pytorch and it can be installed using

cd correlation_package
bash make.sh

If you are using Pytorch>0.3.1, you can use correlation layer from here.

Usage

To use the model, go to your favorite python environment

from back2future import Model
model = Model(pretrained='pretrained/path_to_your_favorite_model')

There are two pretrained models in pretrained/, that are fine tuned on Sintel and KITTI in an unsupervised way.

Refer to demo.py for more.

Testing

To test performance on KITTI, use

python3 test_back2future.py --pretrained-flow path/to/pretrained/model --kitti-dir path/to/kitti/2015/root

Training

Please use the [original torch code] for training new models.

License

This is a reimplementation. License for the original work can be found at JJanai/back2future.

While using this code, please cite

@inproceedings{Janai2018ECCV,
  title = {Unsupervised Learning of Multi-Frame Optical Flow with Occlusions },
  author = {Janai, Joel and G{"u}ney, Fatma and Ranjan, Anurag and Black, Michael J. and Geiger, Andreas},
  booktitle = {European Conference on Computer Vision (ECCV)},
  volume = {Lecture Notes in Computer Science, vol 11220},
  pages = {713--731},
  publisher = {Springer, Cham},
  month = sep,
  year = {2018},
  month_numeric = {9}
}

Comments

Non-shared weights for network branches?

Hi,

thanks for sharing the code. I have a generic question w.r.t the model's code. I see that you have used three (non-shared weights) branches for 3 images in the lines: https://github.com/anuragranj/back2future.pytorch/blob/master/back2future.py#L147-L166

In your experience, do you suggest having non-shared weights for image encoding? I have noticed that PWCNet has shared weights for these branches.

I understand you did not train with pytorch. So, I am not sure if you have shared weights when you trained with Lua torch.

opened by InnovArul 6
Question on the argument for backword warping
Hi,

In your Pytorch implementation, I don't understand why you use the forward upsampled flow for backward warping. should it be the backward upsampled flow? (warping from the past feature (feat5c) into the current frame should use the estimated flow ( past->curr) )

for example in your code of "back2future.py"

feat5b_warped = self.warp(feat5b, 0.625*flow6_fwd_up) feat5c_warped = self.warp(feat5c, -0.625*flow6_fwd_up) # Q: flow6_fwd_up-> flow6_bwd_up?

It would be great if you answer my question.

Thank you, Suhong
opened by suhongkim 2
Some problems about the prediction of occlusion

Thanks for your great job!

I try to binary the occlusion by using nn.sigmoid and np.round, but it doesn't seem to be the right way.

I want to know what is the right way to visualize occlusion? Or can you send to me the files which occlusion visualization on KITTI, Sintel and Middlebury if you have them? I shall be very grateful.:blush:

opened by Ecalpal 1
Confusing of the occlusion map

Thanks for your sharing~!

After read your paper, I noticed that, the occlusion estimation of your work is mapped to (0,1), and the occlusion estimation is used as the weight of photometric loss. While in your paper, the occlusion estimation in Fig.3 is something like a binary map, and in the last of second paragraph of section 4.2, you said 'This clearly shows the benefit of ignoring misleading information in accordance with the occlusion estimates'. It is a little confusing to me. Is the occlusion estimation transformed into binary map? Or you just transform occlusion maps for visualizing?

Any reply will be appreciated~ Thanks again.

opened by lelelexxx 1
questions regarding multiplicative constant for flow & correlation max_displacement in multi-levels
I am trying to understand the code and the design decisions made while designing this model. In this regard, I have two doubts as below:

The correlation max_displacement is kept constant at 4 for all the levels of feature maps. https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L73

In your opinion, do you think we have to have different max_displacement for different levels of the network?

I would like to get an intuition regarding the multiplication factors used for flow: 0.625 for level-6 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L178-L179

1.25 for level-5 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L196-L197

2.5 for level-4 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L213-L214

5 for level-3 flow https://github.com/anuragranj/back2future.pytorch/blob/a3b619a9eb11c91866160565a8593dda690c2da9/back2future.py#L230-L231

Could you please let me know why do we need these multiplicative factors? I am sorry if this is a basic question to be asked. As the network is learnable, isn't it possible that the network learns this multiplicative factor automatically as well?

Thanks in advance!
opened by InnovArul 0
How can I fix 'back2future_kitti.pth.tar' error?

After making the necessary institutions, I ran demo.py. The error I encountered is this> FileNotFoundError: [Errno 2] No such file or directory: 'pretrained / back2future_kitti.pth.tar'. When I look at it with normal access it gives an error > an error occurred while loading the archive. Is the error caused by the file's corruption?

Secondly, the output I got when I ran the line you gave as test > python3 test_back2future.py --pretrained-flow path / to / pretrained / model --kitti-dir path / to / kitti / 2015 / root Traceback (most recent call last): File "test_back2future.py", line 8, in from path import Path ImportError: cannot import name 'Path'

What solution can I apply for this? Thank you from now. @anuragranj @JJanai

opened by ahmetybilgin 3

Owner

Anurag Ranjan

GitHub https://avg.is.tuebingen.mpg.de/research_projects/back2future

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss

UnFlow: Unsupervised Learning of Optical Flow with a Bidirectional Census Loss This repository contains the TensorFlow implementation of the paper UnF

270 Nov 6, 2022

Self-Supervised Multi-Frame Monocular Scene Flow (CVPR 2021)

Self-Supervised Multi-Frame Monocular Scene Flow 3D visualization of estimated depth and scene flow (overlayed with input image) from temporally conse

85 Dec 22, 2022

This is the official repository for evaluation on the NoW Benchmark Dataset. The goal of the NoW benchmark is to introduce a standard evaluation metric to measure the accuracy and robustness of 3D face reconstruction methods from a single image under variations in viewing angle, lighting, and common occlusions.

NoW Evaluation This is the official repository for evaluation on the NoW Benchmark Dataset. The goal of the NoW benchmark is to introduce a standard e

71 Dec 30, 2022

[CVPR 2022] Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions" paper

template-pose Pytorch implementation of "Templates for 3D Object Pose Estimation Revisited: Generalization to New objects and Robustness to Occlusions