Just Go with the Flow: Self-Supervised Scene Flow Estimation
Code release for the paper Just Go with the Flow: Self-Supervised Scene Flow Estimation, CVPR 2020 (Oral).
Authors: Himangi Mittal, Brian Okorn, David Held
[arxiv] [Project Page]
Citation
If you find our work useful in your research, please cite:
@InProceedings{Mittal_2020_CVPR,
author = {Mittal, Himangi and Okorn, Brian and Held, David},
title = {Just Go With the Flow: Self-Supervised Scene Flow Estimation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
Introduction
In this work, we propose a method of scene flow estimation using two self-supervised losses, based on nearest neighbors and cycle consistency. These self-supervised losses allow us to train our method on large unlabeled autonomous driving datasets; the resulting method matches current state-of-the-art supervised performance using no real world annotations and exceeds stateof-the-art performance when combining our self-supervised approach with supervised learning on a smaller labeled dataset.
For more details, please refer to our paper or project page.
Installation
Requirements
CUDA 9.0
Tensorflow-gpu 1.9
Python 3.5
g++ 5.4.0
Steps
(a). Clone the repository.
git clone https://github.com/HimangiM/Self-Supervised-Scene-Flow-Estimation.git
(b). Install dependencies
Create a virtualenv
python3 -m venv sceneflowvenv
source sceneflowvenv/bin/activate
cd Self-Supervised-Scene-Flow-Estimation
pip install -r requirements.txt
Check for CUDA-9.0
(c). Compile the operations The TF operators are included under src/tf_ops. Check the CUDA compatability and edit the architecture accordingly in makefiles of each folder (tf_ops/sampling, tf_ops/grouping, tf_ops/3d_interpolation) The authors had used sm_61 as the architecture for CUDA-9.0. Finally, move into each directory and run make. Also, check for the path for CUDA-9.0 and edit the path in the makefiles of each folder. If this method throws error, then run bash make_tf_ops.sh sm_61.
Datasets
Download the kitti dataset from the Google Drive link. Each file is in the .npz format and has three keys: pos1, pos2 and gt, representing the first frame of point cloud, second frame of point cloud and the ground truth scene flow vectors for the points in the first frame. Create a folder with name data_preprocessing and download the kitti dataset in it. The dataset directory should look as follows:
Self-Supervised-Scene-Flow-Estimation
|--data_preprocessing
| |--kitti_self_supervised_flow
| | |--train
| | |--test
The data preprocessing file to run the code on KITTI is present in the src folder: kitti_dataset_self_supervised_cycle.py. To create a dataloader for own dataset, refer to the script:
nuscenes_dataset_self_supervised_cycle.py
Training and Evaluation
To train on own dataset, refer to the scripts:
train_1nn_cycle_nuscenes.py
bash src/commands/command_train_cycle_nuscenes.sh
To evaluate on the KITTI dataset, execute the shell script:
bash src/commands/command_evaluate_kitti.sh
Link to the pretrained model.
Visualization
You can use Open3d to visualize the results. A sample script is given in visualization.py
28 Nov 16, 2022
43 Dec 5, 2022
35 Nov 20, 2022
124 Dec 27, 2022
92 Nov 30, 2022
42 Nov 24, 2022
24 Dec 17, 2022
72 Dec 28, 2022
21 Mar 9, 2022
60 Nov 30, 2022
112 Nov 28, 2022
110 Dec 23, 2022
17 Nov 10, 2022
3 Oct 22, 2021
4 Jul 27, 2022
182 Dec 30, 2022
33 Dec 12, 2022
151 Dec 26, 2022
66 Nov 16, 2022