Pytorch implementation of MaskFlownet

Overview

MaskFlownet-Pytorch

Unofficial PyTorch implementation of MaskFlownet (https://github.com/microsoft/MaskFlownet).

Tested with:

  • PyTorch 1.5.0
  • CUDA 10.1

Install

The correlation package must be installed first:

cd model/correlation_package
python setup.py install

Inference

Right now, I implemented the inference script for KITTI 2012/2015, MPI Sintel and FlyingChairs.

python predict.py CONFIG -c CHECKPOINT --dataset_cfg DATASET -f ROOT_FOLDER [-b BATCH_SIZE]

For example:

  • python predict.py MaskFlownet.yaml -c 5adNov03-0005_1000000.pth --dataset_cfg sintel.yaml -f ./SINTEL -b 4
  • python predict.py MaskFlownet.yaml -c 8caNov12-1532_300000.pth --dataset_cfg kitti.yaml -f ./KITTI -b 4
  • python predict.py MaskFlownet_S.yaml -c 771Sep25-0735_500000.pth --dataset_cfg chairs.yaml -f ./FLYINGCHAIRS -b 4
  • python predict.py MaskFlownet_S.yaml -c dbbSep30-1206_1000000.pth --dataset_cfg sintel.yaml -f ./SINTEL -b 4

Differences with the original implementation

The results are slightly different from the original implementation:

Checkpoint Network Implementation KITTI2012 KITTI2015 Sintel Clean Sintel Final FlyingChairs
771Sep25 MaskFlownet_S

Original AEPE:
PyTorch AEPE:

4.12
4.18

11.52
11.82

3.38
3.38

4.71
4.70

1.84
1.83

dbbSep30 MaskFlownet_S

Original AEPE:
PyTorch AEPE:

1.27
1.28

1.92
1.93

2.76
2.78

3.29
3.32

2.36
2.36

5adNov03 MaskFlownet

Original AEPE:
PyTorch AEPE:

1.16
1.18

1.66
1.68

2.58
2.59

3.14
3.17

2.23
2.23

8caNov12 MaskFlownet

Original AEPE:
PyTorch AEPE:

0.82
0.82

1.38
1.38

4.34
4.40

5.27
5.33

4.01
3.99

Examples

KITTI Original implementation:

original_visualization

KITTI This implementation:

this_visualization

Sintel Original implementation:

original_visualization

Sintel This implementation:

this_visualization

FlyingChairs Original implementation:

original_visualization

FlyingChairs This implementation:

this_visualization

Notes

If you use my implementation for training, it might happen that you encounter this error:

CUDA error: an illegal memory access was encountered

This is due to a bug in the torchvision implementation of deformable convolutions. (still present in version 0.7.0)

To solve it, you need to use the nightly version of torchvision.

Acknowledgment

Original MXNet implementation: here

correlation_package was taken from flownet2

You might also like...
A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch

This repository holds NVIDIA-maintained utilities to streamline mixed precision and distributed training in Pytorch. Some of the code here will be included in upstream Pytorch eventually. The intention of Apex is to make up-to-date utilities available to users as quickly as possible.

Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch
Objective of the repository is to learn and build machine learning models using Pytorch. 30DaysofML Using Pytorch

30 Days Of Machine Learning Using Pytorch Objective of the repository is to learn and build machine learning models using Pytorch. List of Algorithms

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch
Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Pretrained SOTA Deep Learning models, callbacks and more for research and production with PyTorch Lightning and PyTorch

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.
The Incredible PyTorch: a curated list of tutorials, papers, projects, communities and more relating to PyTorch.

This is a curated list of tutorials, projects, libraries, videos, papers, books and anything related to the incredible PyTorch. Feel free to make a pu

Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks
Amazon Forest Computer Vision: Satellite Image tagging code using PyTorch / Keras with lots of PyTorch tricks

Amazon Forest Computer Vision Satellite Image tagging code using PyTorch / Keras Here is a sample of images we had to work with Source: https://www.ka

A bunch of random PyTorch models using PyTorch's C++ frontend
A bunch of random PyTorch models using PyTorch's C++ frontend

PyTorch Deep Learning Models using the C++ frontend Gettting started Clone the repo 1. https://github.com/mrdvince/pytorchcpp 2. cd fashionmnist or

PyTorch Autoencoders - Implementing a Variational Autoencoder (VAE) Series in Pytorch.

PyTorch Autoencoders Implementing a Variational Autoencoder (VAE) Series in Pytorch. Inspired by this repository Model List check model paper conferen

PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices.

PyTorch-LIT PyTorch-LIT is the Lite Inference Toolkit (LIT) for PyTorch which focuses on easy and fast inference of large models on end-devices. With

Comments
  • flow to image

    flow to image

    I run your code and trying to generate an output image. However, my result is very bad. I guess that the conversion from flow tensor to an image is wrong in my implementation, would you mind showing your code or some samples?

    opened by TengFeiHan0 10
  • errors in installing correlation_package

    errors in installing correlation_package

    I had errors in installing correlation_package. My PyTorch version 1.5.0. My CUDA version is 10.1. My gcc version is 10.1.0.

    running install
    running bdist_egg
    running egg_info
    writing correlation_cuda.egg-info\PKG-INFO
    writing dependency_links to correlation_cuda.egg-info\dependency_links.txt
    writing top-level names to correlation_cuda.egg-info\top_level.txt
    reading manifest file 'correlation_cuda.egg-info\SOURCES.txt'
    writing manifest file 'correlation_cuda.egg-info\SOURCES.txt'
    installing library code to build\bdist.win-amd64\egg
    running install_lib
    running build_ext
    C:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py:237: UserWarning: Error checking compiler version for cl: [WinError 2] The system cannot find the file specified
      warnings.warn('Error checking compiler version for {}: {}'.format(compiler, error))
    building 'correlation_cuda' extension
    Emitting ninja build file C:\Users\lantaoyu\Documents\MaskFlownet-Pytorch-master\MaskFlownet-Pytorch-master\model\correlation_package\build\temp.win-amd64-3.7\Release\build.ninja...
    Compiling objects...
    Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
    [1/2] cl /showIncludes /nologo /Ox /W3 /GL /DNDEBUG /MD /MD /wd4819 /EHsc -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include\TH -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include" -IC:\Users\lantaoyu\Miniconda3\include -IC:\Users\lantaoyu\Miniconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" -c C:\Users\lantaoyu\Documents\MaskFlownet-Pytorch-master\MaskFlownet-Pytorch-master\model\correlation_package\correlation_cuda.cc /FoC:\Users\lantaoyu\Documents\MaskFlownet-Pytorch-master\MaskFlownet-Pytorch-master\model\correlation_package\build\temp.win-amd64-3.7\Release\correlation_cuda.obj -std=c++14 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=correlation_cuda -D_GLIBCXX_USE_CXX11_ABI=0 /std:c++14
    FAILED: C:/Users/lantaoyu/Documents/MaskFlownet-Pytorch-master/MaskFlownet-Pytorch-master/model/correlation_package/build/temp.win-amd64-3.7/Release/correlation_cuda.obj
    cl /showIncludes /nologo /Ox /W3 /GL /DNDEBUG /MD /MD /wd4819 /EHsc -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include\torch\csrc\api\include -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include\TH -IC:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\include" -IC:\Users\lantaoyu\Miniconda3\include -IC:\Users\lantaoyu\Miniconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.26.28801\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" -c C:\Users\lantaoyu\Documents\MaskFlownet-Pytorch-master\MaskFlownet-Pytorch-master\model\correlation_package\correlation_cuda.cc /FoC:\Users\lantaoyu\Documents\MaskFlownet-Pytorch-master\MaskFlownet-Pytorch-master\model\correlation_package\build\temp.win-amd64-3.7\Release\correlation_cuda.obj -std=c++14 -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=correlation_cuda -D_GLIBCXX_USE_CXX11_ABI=0 /std:c++14
    CreateProcess failed: The system cannot find the file specified.
    ninja: fatal: ReadFile: The handle is invalid.
    
    Traceback (most recent call last):
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 1400, in _run_ninja_build
        check=True)
      File "C:\Users\lantaoyu\Miniconda3\lib\subprocess.py", line 512, in run
        output=stdout, stderr=stderr)
    subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "setup.py", line 28, in <module>
        'build_ext': BuildExtension
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\__init__.py", line 161, in setup
        return distutils.core.setup(**attrs)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\core.py", line 148, in setup
        dist.run_commands()
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\dist.py", line 966, in run_commands
        self.run_command(cmd)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\install.py", line 67, in run
        self.do_egg_install()
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\install.py", line 109, in do_egg_install
        self.run_command('bdist_egg')
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\bdist_egg.py", line 173, in run
        cmd = self.call_command('install_lib', warn_dir=0)
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\bdist_egg.py", line 159, in call_command
        self.run_command(cmdname)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\install_lib.py", line 11, in run
        self.build()
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\command\install_lib.py", line 107, in build
        self.run_command('build_ext')
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\dist.py", line 985, in run_command
        cmd_obj.run()
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\build_ext.py", line 87, in run
        _build_ext.run(self)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\command\build_ext.py", line 340, in run
        self.build_extensions()
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 580, in build_extensions
        build_ext.build_extensions(self)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\command\build_ext.py", line 449, in build_extensions
        self._build_extensions_serial()
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\command\build_ext.py", line 474, in _build_extensions_serial
        self.build_extension(ext)
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\setuptools\command\build_ext.py", line 208, in build_extension
        _build_ext.build_extension(self, ext)
      File "C:\Users\lantaoyu\Miniconda3\lib\distutils\command\build_ext.py", line 534, in build_extension
        depends=ext.depends)
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 562, in win_wrap_ninja_compile
        with_cuda=with_cuda)
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 1140, in _write_ninja_file_and_compile_objects
        error_prefix='Error compiling objects for extension')
      File "C:\Users\lantaoyu\Miniconda3\lib\site-packages\torch\utils\cpp_extension.py", line 1413, in _run_ninja_build
        raise RuntimeError(message)
    RuntimeError: Error compiling objects for extension
    
    opened by complexfilter 2
  • Visualization result of Sintel

    Visualization result of Sintel

    Hi, cattaneod! Thank you for implementing the code on pytorch!

    I think there are some mistakes in this re-implement.

    This is the ground truth from sintel training set.(ambush_6 frame_0005) frame_0005

    This is the result which I ran with the original implementation (mxnet MaskFlowNetS). frame_0005

    This is the result which you show on the readme page. image

    This is the result (label) which visualizated from your dataloader. image

    This is the ground truth from sintel training set.(bamboo_1 frame_0001) frame_0001

    Obviously, there are some problems. Can you fix it?

    opened by Ecalpal 1
  • Question about evaluation tabel

    Question about evaluation tabel

    Hi,

    Many thanks for you sharing your nice work, I have two questions about evaluation table.

    1. I didn't find the checkpoint '771Sep25' from original repo, so how did you get it?
    2. The values of the original AEPE in the evaluation table are quite different from the value in maskflownet paper or original repo, so could you please explain it?

    Best regards, Meow

    opened by MeowZheng 1
Owner
Daniele Cattaneo
PostDoc at University of Freiburg. Focus on deep learning for vision-based and LiDAR-based localization, self-driving cars, and sensor fusion.
Daniele Cattaneo
An essential implementation of BYOL in PyTorch + PyTorch Lightning

Essential BYOL A simple and complete implementation of Bootstrap your own latent: A new approach to self-supervised Learning in PyTorch + PyTorch Ligh

Enrico Fini 48 Sep 27, 2022
RealFormer-Pytorch Implementation of RealFormer using pytorch

RealFormer-Pytorch Implementation of RealFormer using pytorch. Includes comparison with classical Transformer on image classification task (ViT) wrt C

Simo Ryu 90 Dec 8, 2022
A PyTorch implementation of the paper Mixup: Beyond Empirical Risk Minimization in PyTorch

Mixup: Beyond Empirical Risk Minimization in PyTorch This is an unofficial PyTorch implementation of mixup: Beyond Empirical Risk Minimization. The co

Harry Yang 121 Dec 17, 2022
A pytorch implementation of Pytorch-Sketch-RNN

Pytorch-Sketch-RNN A pytorch implementation of https://arxiv.org/abs/1704.03477 In order to draw other things than cats, you will find more drawing da

Alexis David Jacq 172 Dec 12, 2022
PyTorch implementation of Advantage async actor-critic Algorithms (A3C) in PyTorch

Advantage async actor-critic Algorithms (A3C) in PyTorch @inproceedings{mnih2016asynchronous, title={Asynchronous methods for deep reinforcement lea

LEI TAI 111 Dec 8, 2022
Pytorch-diffusion - A basic PyTorch implementation of 'Denoising Diffusion Probabilistic Models'

PyTorch implementation of 'Denoising Diffusion Probabilistic Models' This reposi

Arthur Juliani 76 Jan 7, 2023
Fang Zhonghao 13 Nov 19, 2022
RETRO-pytorch - Implementation of RETRO, Deepmind's Retrieval based Attention net, in Pytorch

RETRO - Pytorch (wip) Implementation of RETRO, Deepmind's Retrieval based Attent

Phil Wang 556 Jan 4, 2023
HashNeRF-pytorch - Pure PyTorch Implementation of NVIDIA paper on Instant Training of Neural Graphics primitives

HashNeRF-pytorch Instant-NGP recently introduced a Multi-resolution Hash Encodin

Yash Sanjay Bhalgat 616 Jan 6, 2023
Generic template to bootstrap your PyTorch project with PyTorch Lightning, Hydra, W&B, and DVC.

NN Template Generic template to bootstrap your PyTorch project. Click on Use this Template and avoid writing boilerplate code for: PyTorch Lightning,

Luca Moschella 520 Dec 30, 2022