[ICRA2021] Reconstructing Interactive 3D Scene by Panoptic Mapping and CAD Model Alignment

Last update: Dec 28, 2022

Overview

Interactive Scene Reconstruction

Project Page | Paper

This repository contains the implementation of our ICRA2021 paper Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments. The proposed pipeline reconstructs an interactive indoor scene from RGBD streams, where objects are replaced by (articulated) CAD models. Represented as a contact graph, the reconstructed scene naturally encodes actionable information in terms of environmental kinematics, and can be imported into various simulators to support robot interactions.

The pipeline consists of 3 modules:

A robust panoptic mapping module that accurately reconstruct the semantics and geometry of objects and layouts, which is a modified version of Voxblox++ but with improved robustness. The 2D image segmentation is obtained using [Detectron2] (https://github.com/facebookresearch/detectron2)
An object-based reasoning module that constructs a contact graph from the dense panoptic map and replaces objects with aligned CAD models
An interface that converts a contact graph into a kinematic tree in the URDF format, which can be imported into ROS-based simulators

Todo

Upload code for panoptic mapping
Upload submodules for panoptic mapping
Upload code for CAD replacement
Upload code for URDF conversion and scene visualization
Upload dataset and use cases
Update instructions

1. Installation

1.1 Prerequisites

Ubuntu 16.04 (with ROS Kinetic) or 18.04 (with ROS Melodic)
Python >= 3.7
gcc & g++ >= 5.4
3 <= OpenCV < 4
(Optional) Nvidia GPU (with compatible cuda toolkit and cuDNN) if want to run online segmentation

1.2 Clone the repository & install catkin dependencies

First create and navigate to your catkin workspace

cd <your-working-directory>
mkdir <your-ros-ws>/src && cd <your-ros-ws>

Then, initialize the workspace and configure it. (Remember to replace by your ros version)

catkin init
catkin config --extend /opt/ros/<your-ros-version> --merge-devel 
catkin config --cmake-args -DCMAKE_CXX_STANDARD=14 -DCMAKE_BUILD_TYPE=Release

Download this repository to your ROS workspace src/ folder with submodules via:

cd src
git clone --recursive https://github.com/hmz-15/Interactive-Scene-Reconstruction.git

Then add dependencies specified by .rosinstall using wstool

cd Interactive-Scene-Reconstruction
wstool init dependencies
cd dependencies
wstool merge -t . ../mapping/voxblox-plusplus/voxblox-plusplus_https.rosinstall
wstool merge -t . ../mapping/orb_slam2_ros/orb_slam2_ros_https.rosinstall
wstool update

1.3 Build packages

cd <your-ros-ws>
catkin build orb_slam2_ros perception_ros gsm_node -j2

Comments

Empty visualization in rviz

Hi,

Thank you for the repo!

I tried the visualization instruction in interactive scene but encountered an empty visualization window in rviz. Please see my screenshot for an illustration of the empty visualization window. Have you encountered this issue before and do you by any chance know how to fix it?

I used the reconstructed panoptic segments and estimated contact graph from the provided Google Drive link.

Thank you!

opened by quanvuong 20

Ask about the protobuf version

Dear author,

I am trying to build panoptic_mapping_pipeline by using catkin build panoptic_mapping_pipeline -j2,

but encounter:

Errors     << voxblox:make /opt/ros/melodic/logs/voxblox/build.make.029.log
In file included from /opt/ros/melodic/build/voxblox/Block.pb.cc:4:0:
/opt/ros/melodic/build/voxblox/Block.pb.h:10:10: fatal error: google/protobuf/port_def.inc: No such file or directory
 #include <google/protobuf/port_def.inc>
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [CMakeFiles/voxblox_proto.dir/Block.pb.cc.o] Error 1
make[1]: *** [CMakeFiles/voxblox_proto.dir/all] Error 2
make: *** [all] Error 2
cd /opt/ros/melodic/build/voxblox; catkin build --get-env voxblox | catkin env -si  /usr/bin/make --jobserver-fds=3,4 -j; cd -

while the output of protoc --version is libprotoc 3.13.0. Then I copy the libprotobuf google directory to voxblox directory by using cp -r /miniconda/pkgs/libprotobuf-3.13.0.1-hd408876_0/include/google /opt/ros/melodic/build/voxblox

and build panoptic_mapping_pipeline again, then I face:

Errors     << voxblox:make /opt/ros/melodic/logs/voxblox/build.make.027.log
In file included from /opt/ros/melodic/build/voxblox/Block.pb.cc:4:0:
/opt/ros/melodic/build/voxblox/Block.pb.h:12:2: error: #error This file was generated by a newer version of protoc which is
 #error This file was generated by a newer version of protoc which is
  ^~~~~
/opt/ros/melodic/build/voxblox/Block.pb.h:13:2: error: #error incompatible with your Protocol Buffer headers. Please update
 #error incompatible with your Protocol Buffer headers. Please update
  ^~~~~
/opt/ros/melodic/build/voxblox/Block.pb.h:14:2: error: #error your headers.
 #error your headers.

That says my protobuf version is too low, then I try with protobuf 3.16.0 also with the result of too low and try with protobuf 3.18.0 with the result of too high. Obviously there only remainds protobuf 3.17.0 to be chosen, so I try with protobuf 3.17.0 and it outputs a new error:

/opt/ros/melodic/devel/lib/libvoxblox_proto.so: undefined reference to google::protobuf::internal::InlineGreedyStringParser(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, char const*, google::protobuf::internal::ParseContext*)'
/opt/ros/melodic/devel/lib/libvoxblox_proto.so: undefined reference to google::protobuf::internal::VarintParseSlow64(char const*, unsigned int)'
collect2: error: ld returned 1 exit status
make[2]: *** [/opt/ros/melodic/devel/lib/voxblox/tsdf_to_esdf] Error 1
make[1]: *** [CMakeFiles/tsdf_to_esdf.dir/all] Error 2
make: *** [all] Error 2
cd /opt/ros/melodic/build/voxblox; catkin build --get-env voxblox | catkin env -si  /usr/bin/make --jobserver-fds=3,4 -j; cd -

So I am confused about which version of protobuf should I use. Or is there any other reason that cause the problem?

Looking forward to your reply.

opened by DoctorXK 12

ERROR: Service [/gsm_node/generate_mesh/] is not available.

Hi dear author,

I finally build panoptic_mapping_pipeline successfully, but when I execute Step 4 rosservice call /gsm_node/generate_mesh in Robust Panoptic Mapping I will encounter:

ERROR: Service [/gsm_node/generate_mesh/] is not available.

I cannot find generate_mesh in my running environment and so as rosservice call /gsm_node/extract_instances.

When I reviewed the compilation process I found there was warning message when compiling gsm_node even though it didn't throw compilation ERROR.

Warnings << gsm_node:cmake /opt/ros/melodic/log/gsm_node/build.cmake.000.log 
CMake Warning at /opt/ros/melodic/devel/share/pcl catkin/cmake/pcl-extras.cmake:14 (find package): 
  By not providing "FindovTk.cmake" in cMAkE MODuLE PATH this project has asked CMake to find a package confiquration file provided by"QVTK"but CMake did not find one.
  Could not find a package configuration file provided by "QVTK" with any of
the following names: 
    QVTKConfig.cmake
    qvtk-config.cmake
  Add the installation prefix of "OVTK" to CMAKE PREFIX_PATH or set
  "OVTK DIR" to a directory containing one of the above files.If"OVTK 
  provides a separate development package or sokbe sure it has been 
  installed.
Call Stack (most recent call first):
  /opt/ros/melodic/devel/share/pclcatkin/cmake/pcl_catkinconfiq.cmake:222(include)
  /opt/ros/melodic/devel/share/globalsegment_map/cmake/globalsegment_mapConfig.cmake:197(find_package)
  /opt/ros/melodic/devel/share/catkin_simple/cmake/catkin_simple-extras.cmake:38 (find_package) 
  CMakeLists.txt:12(catkin simple)

cd /opt/ros/melodic/build/gsm_node; catkin build --det-env gsm_node catkin env -si /usr/bin/cmake /opt/ros/melodic/src/Interactive-Scene-Reconstruction/mappina/voxblox-plusplus/global_segment_map_node --no-warn-unused-cli -DCATKIN_DEVEL_PREFIX=/opt/ros/melodic/devel -DCMAKE INSTALL PREFIX=/opt/ros/melodic/install -OCMAKE_CX X_STANDARD=14 -DCMAKE_BUILDTYPE=Release; cd -

Have you ever encounter this situation? Looking forward to your reply. : )

opened by DoctorXK 8

Something wrong when "catkin build map_proc"
Hi, When building the map_proc,there are some errors printed to log. I think the reason is googletest changed the tag to "main" instead of "master", could you please check the errors and modified the code? or maybe I made a wrong build method? thanks. the build log as following: [ 10%] [34m[1mPerforming configure step for 'approxmvbb_src'[0m -- Module path is now set to: /home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src/cmake/ -- Project name is: ApproxMVBB- -- Set make program to /home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/parallelmake.sh -- Meta: find meta -- Meta: Setup External Projext -- /home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/src/meta/include -- Added '' to CMAKE_CXX and CMAKE_C_FLAGS: and
-- Adding ApproxMVBB - Diameter ====================== -- ==================================================== -- Adding ApproxMVBB - GeometryPredicates ============ -- Floating point control: HAVE__FPU_SETCW HAVE_FPSETPREC HAVE__CONTROLFP HAVE__CONTROLFP_S HAVE_FPU_INLINE_ASM_X86 1 -- ==================================================== -- ApproxMVBB: Write config file /home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/include/ApproxMVBB/Config/Config.hpp, -- ApproxMVBB Version: -- v2.1.1 extracted from git tags! -- Dependecy Libs (public): eigenLib -- Downloading/updating googletest -- Configuring done -- Generating done -- Build files have been written to: /home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download make[3]: Entering directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[4]: Entering directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[5]: Entering directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[5]: Leaving directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[5]: Entering directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' [ 11%] Performing download step (git clone) for 'googletest-download' Cloning into 'googletest-src'... fatal: unable to access 'https://github.com/google/googletest.git/': GnuTLS recv error (-54): Error in the pull function. Cloning into 'googletest-src'... -- Had to git clone more than once: 2 times. fatal: invalid reference: master CMake Error at googletest-download/googletest-download-prefix/tmp/googletest-download-gitclone.cmake:40 (message): Failed to checkout tag: 'master'

make[5]: Leaving directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[5]: *** [CMakeFiles/googletest-download.dir/build.make:91: googletest-download-prefix/src/googletest-download-stamp/googletest-download-download] Error 1 make[4]: Leaving directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[4]: *** [CMakeFiles/Makefile2:76: CMakeFiles/googletest-download.dir/all] Error 2 make[3]: Leaving directory '/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-download' make[3]: *** [Makefile:84: all] Error 2 CMake Error at tests/CMakeLists.txt:14 (add_subdirectory): The source directory

/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/thirdparty/googletest-src

does not contain a CMakeLists.txt file.

-- Configuring incomplete, errors occurred! See also "/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/CMakeFiles/CMakeOutput.log". See also "/home/lwh/ros-ws/build/approxmvbb_catkin/approxmvbb_src-prefix/src/approxmvbb_src-build/CMakeFiles/CMakeError.log". make[2]: *** [CMakeFiles/approxmvbb_src.dir/build.make:108: approxmvbb_src-prefix/src/approxmvbb_src-stamp/approxmvbb_src-configure] Error 1 make[1]: *** [CMakeFiles/Makefile2:333: CMakeFiles/approxmvbb_src.dir/all] Error 2 make: *** [Makefile:141: all] Error 2
opened by acclwh 6
How to save the CAD replacement results

Hello,

The options in map_proc/launch/map_processing.launch doesn't seem to cover how to save the results of scene with CAD replacement. However, it seems that it is necesssary to save the CAD replacement results if we want these for post-processing. So I would like to ask how to save this kind of result? Or is there any saving options in the script?

It would be grateful if you can give some suggestions. Thank you very much!!!

opened by DoctorXK 4
Error compiling perception_ros

Error compiling perception_ros： Some people say that both opencv and pcl define "flann", and the function of opencv is called here instead of pcl. What should I do?

opened by canglangzhige 3
Question about the ground truth localization

In the tutorial "https://github.com/hmz-15/Interactive-Scene-Reconstruction/tree/main/mapping", Step 2, The ground truth pose of the sceneNN dataset is used, but the topic "ground truth pose" does not exit by the command "rosbag info scenenn_225.bag" : How is the ground truth pose obtained, and how is it input into the entire system?

opened by canglangzhige 1
ERROR: Service [/gsm_node/generate_mesh] is not available.

I successfully compiled the whole project and ran panoptic_mapping_pipeline successfully. But when I run "rosservice call /gsm_node/generate_mesh", I get an error "ERROR: Service [/gsm_node/generate_mesh] is not available." as https://github.com/hmz-15/Interactive-Scene-Reconstruction/issues/3. I successfully compiled "gsm_node"： I cannot find generate_mesh and extract_instances under rosservice list after： catkin build gsm_node What should I do.

opened by canglangzhige 1

Owner

Ph.D student in Robotics

GitHub

[arXiv'22] Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation Xiao Fu1* Shangzhan Zhang1* Tianrun Chen1 Yichong Lu1 Lanyun Zhu2 Xi

37 May 17, 2022

Code for "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" @ICRA2021

CloudAAE This is an tensorflow implementation of "CloudAAE: Learning 6D Object Pose Regression with On-line Data Synthesis on Point Clouds" Files log:

35 Nov 14, 2022

SSL_SLAM2: Lightweight 3-D Localization and Mapping for Solid-State LiDAR (mapping and localization separated) ICRA 2021

SSL_SLAM2 Lightweight 3-D Localization and Mapping for Solid-State LiDAR (Intel Realsense L515 as an example) This repo is an extension work of SSL_SL

1.3k Jan 8, 2023

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment (ICCV2021)

Seeing Dynamic Scene in the Dark: High-Quality Video Dataset with Mechatronic Alignment This is a pytorch project for the paper Seeing Dynamic Scene i

21 Nov 28, 2022

Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

178 Dec 13, 2022

Implementation of CVPR'2022:Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors

Reconstructing Surfaces for Sparse Point Clouds with On-Surface Priors (CVPR 2022) Personal Web Pages | Paper | Project Page This repository contains

151 Dec 26, 2022

An unofficial personal implementation of UM-Adapt, specifically to tackle joint estimation of panoptic segmentation and depth prediction for autonomous driving datasets.

Semisupervised Multitask Learning This repository is an unofficial and slightly modified implementation of UM-Adapt[1] using PyTorch. This code primar

11 Nov 25, 2022

[TIP 2020] Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion

Multi-Temporal Scene Classification and Scene Change Detection with Correlation based Fusion Code for Multi-Temporal Scene Classification and Scene Ch

33 Dec 12, 2022

Official PyTorch code of DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context Graph and Relation-based Optimization (ICCV 2021 Oral).

DeepPanoContext (DPC) [Project Page (with interactive results)][Paper] DeepPanoContext: Panoramic 3D Scene Understanding with Holistic Scene Context G

66 Nov 16, 2022

Implementation for Panoptic-PolarNet (CVPR 2021)

Panoptic-PolarNet This is the official implementation of Panoptic-PolarNet. [ArXiv paper] Introduction Panoptic-PolarNet is a fast and robust LiDAR po

126 Jan 1, 2023

Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

Implementation for Pixel Consensus Voting (CVPR 2020). This codebase contains the essential ingredients of PCV, including various spatial discretizati

23 Oct 25, 2022

[CVPR 2021] Exemplar-Based Open-Set Panoptic Segmentation Network (EOPSN)

EOPSN: Exemplar-Based Open-Set Panoptic Segmentation Network (CVPR 2021) PyTorch implementation for EOPSN. We propose open-set panoptic segmentation t

49 Dec 30, 2022

[CVPR 2021] Forecasting the panoptic segmentation of future video frames

Panoptic Segmentation Forecasting Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021 [Link to paper] We propose

44 Nov 29, 2022

A PyTorch implementation of the baseline method in Panoptic Narrative Grounding (ICCV 2021 Oral)

52 Dec 19, 2022

PanopticBEV - Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images

Bird's-Eye-View Panoptic Segmentation Using Monocular Frontal View Images This r

63 Dec 16, 2022

Image transformations designed for Scene Text Recognition (STR) data augmentation. Published at ICCV 2021 Workshop on Interactive Labeling and Data Augmentation for Vision.

Data Augmentation for Scene Text Recognition (ICCV 2021 Workshop) (Pronounced as "strog") Paper Arxiv Why it matters? Scene Text Recognition (STR) req

152 Dec 28, 2022

Code for CVPR 2021 oral paper "Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts"

Exploring Data-Efficient 3D Scene Understanding with Contrastive Scene Contexts The rapid progress in 3D scene understanding has come with growing dem

182 Dec 30, 2022

Neural Scene Graphs for Dynamic Scene (CVPR 2021)

Implementation of Neural Scene Graphs, that optimizes multiple radiance fields to represent different objects and a static scene background. Learned representations can be rendered with novel object compositions and views.