TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction

Overview

TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction

TSDF++ is a novel multi-object TSDF formulation that can encode multiple object surfaces at each voxel. In a multiple dynamic object tracking and reconstruction scenario, a TSDF++ map representation allows maintaining accurate reconstruction of surfaces even while they become temporarily occluded by other objects moving in their proximity. At the same time, the representation allows maintaining a single volume for the entire scene and all the objects therein, thus solving the fundamental challenge of scalability with respect to the number of objects in the scene and removing the need for an explicit occlusion handling strategy.

Citing

When using TSDF++ in your research, please cite the following publication:

Margarita Grinvald, Federico Tombari, Roland Siegwart, and Juan Nieto, TSDF++: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction, in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021. [Paper] [Video]

@article{grinvald2021tsdf,
  author={M. {Grinvald} and F. {Tombari} and R. {Siegwart} and J. {Nieto}},
  booktitle={IEEE International Conference on Robotics and Automation (ICRA)},
  title={{TSDF++}: A Multi-Object Formulation for Dynamic Object Tracking and Reconstruction},
  year={2021},
}

Installation

The installation has been tested on Ubuntu 16.04 and Ubutnu 20.04.

Requirements

Install dependencies

Install ROS following the instructions at the ROS installation page. The full install (ros-kinetic-desktop-full, ros-melodic-desktop-full) are recommended.

Make sure to source your ROS setup.bash script by following the instructions on the ROS installation page.

Installation on Ubuntu

In your terminal, define the installed ROS version and name of the catkin workspace to use:

export ROS_VERSION=kinetic # (Ubuntu 16.04: kinetic, Ubuntu 18.04: melodic)
export CATKIN_WS=~/catkin_ws

If you don't have a catkin workspace yet, create a new one:

mkdir -p $CATKIN_WS/src && cd $CATKIN_WS
catkin init
catkin config --extend /opt/ros/$ROS_VERSION --merge-devel 
catkin config --cmake-args -DCMAKE_CXX_STANDARD=14 -DCMAKE_BUILD_TYPE=Release
wstool init src

Clone the tsdf-plusplus repository over HTTPS (no Github account required) and automatically fetch dependencies:

cd $CATKIN_WS/src
git clone https://github.com/ethz-asl/tsdf-plusplus.git
wstool merge -t . tsdf-plusplus/tsdf_plusplus_https.rosinstall
wstool update

Alternatively, clone over SSH (Github account required):

cd $CATKIN_WS/src
git clone [email protected]:ethz-asl/tsdf-plusplus.git
wstool merge -t . tsdf-plusplus/tsdf_plusplus_ssh.rosinstall
wstool update

Build and source the TSDF++ packages:

catkin build tsdf_plusplus_ros rgbd_segmentation mask_rcnn_ros cloud_segmentation
source ../devel/setup.bash # (bash shell: ../devel/setup.bash,  zsh shell: ../devel/setup.zsh)

Troubleshooting

Compilation freeze

By default catkin build on a computer with N CPU cores will run N make jobs simultaneously. If compilation seems to hang forever, it might be running low on RAM. Try limiting the number of maximum parallel build jobs through the -jN flag to a value way lower than your CPU count, i.e.

catkin build tsdf_plusplus_ros rgbd_segmentation mask_rcnn_ros cloud_segmentation -j4

If it still freezes at compilation time, you can go as far as limiting the maximum number of parallel build jobs and max load to 1 through the -lN flag:

catkin build tsdf_plusplus_ros rgbd_segmentation mask_rcnn_ros cloud_segmentation -j1 -l1

License

The code is available under the MIT license.

Comments
  • ROS2 Support

    ROS2 Support

    If I would add ROS2 support to your project in a forked repo, would you be open to review it as a PR into your repo (either master or special ros2 branch) ?

    opened by paucarre 1
  • Compilation error: ‘struct std::atomic<bool>’ has no member named ‘set’

    Compilation error: ‘struct std::atomic’ has no member named ‘set’

    Hi, I am trying to build this seemingly wonderful package and can't seem to get over that error message... Does anybody knows what I am supposed to do to fix it ? Thanks in advance.

    Errors     << tsdf_plusplus:make /d/qserdel/ros_ws/voxbloxpp_ws/logs/tsdf_plusplus/build.make.004.log                                                                                                     
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/core/map.cc: In member function ‘void Map::transformLayer(const ObjectID&, const Transformation&)’:
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/core/map.cc:235:25: error: ‘struct std::atomic<bool>’ has no member named ‘set’
      235 |     mo_block->updated().set();
          |                         ^~~
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/core/map.cc:322:27: error: ‘struct std::atomic<bool>’ has no member named ‘set’
      322 |       mo_block->updated().set();
          |                           ^~~
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/core/map.cc: In member function ‘void Map::removeObject(const ObjectID&)’:
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/core/map.cc:356:25: error: ‘struct std::atomic<bool>’ has no member named ‘set’
      356 |     mo_block->updated().set();
          |                         ^~~
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/mesh/mesh_integrator.cc: In member function ‘bool MOMeshIntegrator::generateMesh(bool, bool)’:
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/mesh/mesh_integrator.cc:58:49: error: ‘Update’ has not been declared
       58 |     map_->getMapLayerPtr()->getAllUpdatedBlocks(Update::kMesh, &all_map_blocks);
          |                                                 ^~~~~~
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/mesh/mesh_integrator.cc: In member function ‘void MOMeshIntegrator::generateMeshBlocksFunction(const BlockIndexList&, bool, voxblox::ThreadSafeIndex*)’:
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/mesh/mesh_integrator.cc:104:24: error: ‘struct std::atomic<bool>’ has no member named ‘reset’
      104 |       block->updated().reset(Update::kMesh);
          |                        ^~~~~
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/mesh/mesh_integrator.cc:104:30: error: ‘Update’ has not been declared
      104 |       block->updated().reset(Update::kMesh);
          |                              ^~~~~~
    make[2]: *** [CMakeFiles/tsdf_plusplus.dir/build.make:89: CMakeFiles/tsdf_plusplus.dir/src/core/map.cc.o] Error 1
    make[2]: *** Waiting for unfinished jobs....
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/integrator/integrator.cc: In member function ‘MOVoxel* Integrator::allocateStorageAndGetVoxelPtr(const GlobalIndex&, voxblox::Block<MOVoxel>::Ptr*, voxblox::BlockIndex*)’:
    /d/qserdel/ros_ws/voxbloxpp_ws/src/tsdf-plusplus/tsdf_plusplus/src/integrator/integrator.cc:435:28: error: ‘struct std::atomic<bool>’ has no member named ‘set’
      435 |   (*last_block)->updated().set();
    
    opened by qserdel 0
  • How to run

    How to run

    Hi, your work is amazing and thank you very much for the code! Could you please provide some instructions on how to run it? I would really like to test it on some datasets.

    opened by elDvd96 5
  • ROS Noetic compilation

    ROS Noetic compilation

    When compiling for ROS Noetic on Ubuntu 20.04, I encountered the error

    CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:117 (find_package): Could not find a package configuration file provided by "boost_signals" (requested version 1.71.0) with any of the following names:

    boost_signalsConfig.cmake
    boost_signals-config.cmake
    

    Add the installation prefix of "boost_signals" to CMAKE_PREFIX_PATH or set "boost_signals_DIR" to a directory containing one of the above files. If "boost_signals" provides a separate development package or SDK, be sure it has been installed. Call Stack (most recent call first): /usr/lib/x86_64-linux-gnu/cmake/Boost-1.71.0/BoostConfig.cmake:182 (boost_find_component) /usr/share/cmake-3.16/Modules/FindBoost.cmake:443 (find_package) CMakeLists.txt:6 (find_package)

    This was solved by simply removing signals from src/pcl_catkin/geometry2/tf2/CMakeLists.txt (see this for context).

    Thanks for the amazing work!

    opened by juntuck 1
  • How to preprocess input data before passing it to tsdf_plusplus_node?

    How to preprocess input data before passing it to tsdf_plusplus_node?

    Hi there! Thank you for your great work. But how exactly do I need to preprocess the input data before passing it to the tsdf_plusplus_node on the /segment_pointcloud topic? Thanks in advance!

    opened by HauJulian 0
Owner
ETHZ ASL
ETHZ ASL
Python code to fuse multiple RGB-D images into a TSDF voxel volume.

Volumetric TSDF Fusion of RGB-D Images in Python This is a lightweight python script that fuses multiple registered color and depth images into a proj

Andy Zeng 812 Nov 20, 2022
"MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction" (CVPRW 2022) & (Winner of NTIRE 2022 Challenge on Spectral Reconstruction from RGB)

MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (CVPRW 2022) Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Z

Yuanhao Cai 266 Nov 14, 2022
Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling

TGraM Multi-Object Tracking in Satellite Videos with Graph-Based Multi-Task Modeling, Qibin He, Xian Sun, Zhiyuan Yan, Beibei Li, Kun Fu Abstract Rece

Qibin He 5 Oct 19, 2022
A Planar RGB-D SLAM which utilizes Manhattan World structure to provide optimal camera pose trajectory while also providing a sparse reconstruction containing points, lines and planes, and a dense surfel-based reconstruction.

ManhattanSLAM Authors: Raza Yunus, Yanyan Li and Federico Tombari ManhattanSLAM is a real-time SLAM library for RGB-D cameras that computes the camera

null 115 Nov 5, 2022
Official PyTorch implementation of Joint Object Detection and Multi-Object Tracking with Graph Neural Networks

This is the official PyTorch implementation of our paper: "Joint Object Detection and Multi-Object Tracking with Graph Neural Networks". Our project website and video demos are here.

Richard Wang 442 Nov 15, 2022
Object Detection and Multi-Object Tracking

Object Detection and Multi-Object Tracking

Bobby Chen 1.6k Nov 23, 2022
Python package for multiple object tracking research with focus on laboratory animals tracking.

motutils is a Python package for multiple object tracking research with focus on laboratory animals tracking. Features loads: MOTChallenge CSV, sleap

Matěj Šmíd 2 Sep 5, 2022
The code for the CVPR 2021 paper Neural Deformation Graphs, a novel approach for globally-consistent deformation tracking and 3D reconstruction of non-rigid objects.

Neural Deformation Graphs Project Page | Paper | Video Neural Deformation Graphs for Globally-consistent Non-rigid Reconstruction Aljaž Božič, Pablo P

Aljaz Bozic 132 Nov 1, 2022
MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

Felix Wimbauer 480 Nov 26, 2022
Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation

DynaBOA Code repositoty for the paper: Out-of-Domain Human Mesh Reconstruction via Dynamic Bilevel Online Adaptation Shanyan Guan, Jingwei Xu, Michell

null 186 Nov 19, 2022
OcclusionFusion: realtime dynamic 3D reconstruction based on single-view RGB-D

OcclusionFusion (CVPR'2022) Project Page | Paper | Video Overview This repository contains the code for the CVPR 2022 paper OcclusionFusion, where we

Wenbin Lin 188 Nov 25, 2022
CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image.

CoReNet CoReNet is a technique for joint multi-object 3D reconstruction from a single RGB image. It produces coherent reconstructions, where all objec

Google Research 77 Nov 22, 2022
Dynamic View Synthesis from Dynamic Monocular Video

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer This repository contains code to compute depth from a

Intelligent Systems Lab Org 2.1k Nov 29, 2022
Dynamic View Synthesis from Dynamic Monocular Video

Dynamic View Synthesis from Dynamic Monocular Video Project Website | Video | Paper Dynamic View Synthesis from Dynamic Monocular Video Chen Gao, Ayus

Chen Gao 133 Nov 17, 2022
Dynamic vae - Dynamic VAE algorithm is used for anomaly detection of battery data

Dynamic VAE frame Automatic feature extraction can be achieved by probability di

null 10 Oct 7, 2022
Joint detection and tracking model named DEFT, or ``Detection Embeddings for Tracking.

DEFT: Detection Embeddings for Tracking DEFT: Detection Embeddings for Tracking, Mohamed Chaabane, Peter Zhang, J. Ross Beveridge, Stephen O'Hara

Mohamed Chaabane 250 Nov 12, 2022
The official repo for OC-SORT: Observation-Centric SORT on video Multi-Object Tracking. OC-SORT is simple, online and robust to occlusion/non-linear motion.

OC-SORT Observation-Centric SORT (OC-SORT) is a pure motion-model-based multi-object tracker. It aims to improve tracking robustness in crowded scenes

Jinkun Cao 306 Nov 29, 2022
Tracking code for the winner of track 1 in the MMP-Tracking Challenge at ICCV 2021 Workshop.

Tracking Code for the winner of track1 in MMP-Trakcing challenge This repository contains our tracking code for the Multi-camera Multiple People Track

DamoCV 29 Nov 13, 2022