ROSEFusion
🌹
This project is based on our SIGGRAPH 2021 paper, ROSEFusion: Random Optimization for Online DenSE Reconstruction under Fast Camera Motion .
Introduction
ROSEFsuion is proposed to tackle the difficulties in fast-motion camera tracking using random optimization with depth information only. Our method attains good quality pose tracking under fast camera motion in a realtime framerate without including loop closure or global pose optimization.
Installation
The code is based on C++ and CUDA with the support of:
- Pangolin
- OpenCV with CUDA (v.4.5 is required, for instance you can follow the link)
- Eigen
- CUDA (v.11 and above is required)
Befor building, please make sure the architecture (sm_xx and compute_xx) in the L22 of CMakeLists.txt is compatible with your own graphics card.
Our code has been tested with Nvidia GeForce RTX 2080 SUPER on Ubuntu 16.04.
[Option] Test with Docker
We have already upload a docker image with all the lib, code and data. Please download the image from the google drive.
Prepare
Make sure you have successfully installed the docker and nvidia docker. Once the environment is ready, you can using following commands to boot the docker image:
sudo docker load -i rosefusion_docker.tar
sudo docker run -it --gpus all jiazhao/rosefusion:v7 /bin/bash
And please check the architecture in the L22 of /home/code/ROSEFusion-main/CMakeList.txt is compatible with your own graphics card. If not, change the sm_xx and compute_xx, then rebuild the ROSEFusion.
QuickStart
All the data and configuration files are ready for using. You can find "run_example.sh" and "run_stairwell.sh" in /home/code/ROSEFusion-main/build. After running the scripts, the trajectory and reconstuciton results woulSd be generated in /home/code/rosefusion_xxx_data.
Configuration File
We use the following configuration files to make the parameters setting easier. There are four types of configuration files.
- seq_generation_config.yaml: data information
- camera_config.yaml: camera and image information.
- data_config.yaml: output path, sequence file path and parameters of the volume.
- controller_config.yaml: visualization, saving and parameters of tacking.
The seq_generation_config.yaml is only used in data preparation, and the other three types of configuration files are necessary to run the fusion part. The configuration files of many common datasets are given in [type]_config/ directory, you can change the settings to fit your own dataset.
Data Preparation
The details of data prepartiation can be found in src/seq_gen.cpp. By using the seq_generation_config.yaml introduced above, you can run the program:
./seq_gen sequence_information.yaml
Once finished, there will be a .seq file containing all the information of the sequence.
Particle Swarm Template
We share the same pre-sampled PST as we used in our paper. Each PST is saved as an N×6 image and the N represents the number of particles. You can find the .tiff images in PST dicrectory, and please prelace the PST path in controller_config.yaml with your own path.
Running
To run the fusion code, you need to provide the camera_config.yaml, data_config.yaml and controller_config.yaml. We already share configuration files of many common datasets in ./camera_config, ./data_config, /controller_config. All the parameters of configuration can be modified as you want. With all the preparation done, you can run the code below:
./ROSEFsuion your_camera_config.yaml your_data_config.yaml your_controller_config.yaml
For a quick start, you can download and use a small size synthesis seq file and related configuration files. Here is a preview.
FastCaMo Dataset
We present the Fast Camera Motion dataset, which contains both synthesis and real captured sequences. You are welcome to download the sequences and take a try.
FastCaMo-Synth
With 10 diverse room-scale scenes from Replica Dataset, we render the color images and depth maps along the synthesis trajectories. The raw sequences are provided in FastCaMo-synth-data(raw).zip, and we also provide the FastCaMo-synth-data(noise).zip with synthesis noise. We use the same noise model as simkinect. For evaluation, you can download the ground truth trajectories.
FastCaMo-Real
There are 12 real captured RGB-D sequences with fast camera motions are released. Each sequence is recorded in a challenging scene like gym or stairwell by using Azure Kinect DK. We offer a full and dense reconstruction scanned using the high-end laser scanner, serving as ground truth. However, The original file is extremely large, we will share the dense reconstruction in another platform or release the sub-sampled version only.
Citation
If you find our work useful in your research, please consider citing:
@article {zhang_sig21,
title = {ROSEFusion: Random Optimization for Online Dense Reconstruction under Fast Camera Motion},
author = {Jiazhao Zhang and Chenyang Zhu and Lintao Zheng and Kai Xu},
journal = {ACM Transactions on Graphics (SIGGRAPH 2021)},
volume = {40},
number = {4},
year = {2021}
}
Acknowledgments
Our code is inspired by KinectFusionLib.
This is an open-source version of ROSEFusion, some functions have been rewritten to avoid certain license. It would not be expected to reproduce the result exactly, but the result is almost the same.
License
The source code is released under GPLv3 license.
Contact
If you have any questions, feel free to email Jiazhao Zhang at [email protected].
And the console warning "Frame could not be processed",
WHY??? I guess because of the camera is out of the reconstruction range. What is the unit of the "voxel_size" in dateset_config？Millimeter？
494 Jan 6, 2023
258 Jan 2, 2023
99 Dec 30, 2022
130 Jan 2, 2023
7 Jul 15, 2022
110 Dec 20, 2022
6 Dec 31, 2022
40 Dec 5, 2022
71 Dec 30, 2022
14 Nov 28, 2022
77 Dec 16, 2022
25 Dec 26, 2022
27 Jun 28, 2022
35 Dec 6, 2022
54 Dec 6, 2022
203 Jan 5, 2023
93 Dec 23, 2022
274 Jan 5, 2023