Hi, and thank you for making tis code available. I am building it on windows desktop, visual studio 2019, and after a day or so of tweaking, I have it compiled.

However, when I run main.exe -i=output p -useStereo The application does not start. ("output" is the dir containing the csv and video files data.csv, data.avi., data2.avi)

I think this is because i am not passing in parameters correctly.

I am confused as to how the parameter json is created. The system looks for: std::ifstream cmdParametersFile("../data/cmd.json");

But how do I create this file and pass in the camera intrinsics etc?

(I am using Zed2 data recorded with the zed capture application from your readme)

Thank you!

Hi, I am trying to compile and run HybVIO on a Raspberry Pi 4 with 4 GB of RAM running Debian Bullseye with no desktop environment (Raspberry Pi OS Lite 32-bit).

These are the steps I followed:

1. sudo apt update && sudo apt upgrade && sudo apt autoremove
2. sudo apt install clang libc++-dev libgtk2.0-dev libgstreamer1.0-dev libvtk6-dev libavresample-dev libglfw3-dev libglfw3 libglew-dev libxkbcommon-dev cmake git ffmpeg (I was unable to install glfw package, is it supported on Pi?)
3. git clone https://github.com/SpectacularAI/HybVIO.git --recursive
4. cd HybVIO/3rdparty/mobile-cv-suite
5. ./scripts/build.sh

And this is the received error:

CMake Warning (dev) at CMakeLists.txt:135 (if):
  Policy CMP0054 is not set: Only interpret if() arguments as variables or
  keywords when unquoted.  Run "cmake --help-policy CMP0054" for policy
  details.  Use the cmake_policy command to set the policy and suppress this
  warning.

  Quoted variables like "HASWELL" will no longer be dereferenced when the
  policy is set to NEW.  Since the policy is not set the OLD behavior will be
  used.
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Reading vars from /home/pi/HybVIO/3rdparty/mobile-cv-suite/OpenBLAS/kernel/arm/KERNEL...
-- Reading vars from /home/pi/HybVIO/3rdparty/mobile-cv-suite/OpenBLAS/kernel/arm/KERNEL.HASWELL...
CMake Error at cmake/utils.cmake:20 (file):
  file STRINGS file
  "/home/pi/HybVIO/3rdparty/mobile-cv-suite/OpenBLAS/kernel/arm/KERNEL.HASWELL"
  cannot be read.
Call Stack (most recent call first):
  kernel/CMakeLists.txt:16 (ParseMakefileVars)
  kernel/CMakeLists.txt:863 (build_core)

Is it something relative to a specific version of CMake that must be used to compile this project? And furthermore, is there a specific guide available on how to compile HybVIO on Raspberry and parameter optimizations to make it run in real time?

Thanks

Hi, I'm also stuck here, the same as closed issure #11 (on Ubuntu18.04). I tried to rm -rf target and using clang-10.0 but still doesn't work. May I ask is there any other way to solve this?

When running make -j6 here:

mkdir target
cd target
cmake -DBUILD_VISUALIZATIONS=ON -DUSE_SLAM=ON ..
# or if not using clang by default:
# CC=clang CXX=clang++ cmake ..
make -j6

I'm having this error:

/home/user/workspace/HybVIO/src/slam/orb_extractor.cpp: In member function ‘virtual void slam::{anonymous}::OrbExtractorImplementation::detectAndExtract(tracker::Image&, const tracker::Camera&, const std::vector<tracker::Feature>&, slam::KeyPointVector&, std::vector<int>&)’:
/home/user/workspace/HybVIO/src/slam/orb_extractor.cpp:110:18: warning: missing initializer for member ‘slam::KeyPoint::bearing’ [-Wmissing-field-initializers]
                 });
                  ^
/home/user/workspace/HybVIO/src/slam/orb_extractor.cpp:110:18: warning: missing initializer for member ‘slam::KeyPoint::descriptor’ [-Wmissing-field-initializers]
/home/user/workspace/HybVIO/src/slam/orb_extractor.cpp:161:18: error: no matching function for call to ‘std::vector<slam::KeyPoint, Eigen::aligned_allocator<slam::KeyPoint> >::push_back(<brace-enclosed initializer list>)’
                 });
                  ^
In file included from /usr/include/c++/7/vector:64:0,
                 from /home/user/workspace/HybVIO/src/slam/static_settings.hpp:4,
                 from /home/user/workspace/HybVIO/src/slam/orb_extractor.hpp:4,
                 from /home/user/workspace/HybVIO/src/slam/orb_extractor.cpp:40:
/usr/include/c++/7/bits/stl_vector.h:939:7: note: candidate: void std::vector<_Tp, _Alloc>::push_back(const value_type&) [with _Tp = slam::KeyPoint; _Alloc = Eigen::aligned_allocator<slam::KeyPoint>; std::vector<_Tp, _Alloc>::value_type = slam::KeyPoint]
       push_back(const value_type& __x)
       ^~~~~~~~~
/usr/include/c++/7/bits/stl_vector.h:939:7: note:   no known conversion for argument 1 from ‘<brace-enclosed initializer list>’ to ‘const value_type& {aka const slam::KeyPoint&}’
/usr/include/c++/7/bits/stl_vector.h:953:7: note: candidate: void std::vector<_Tp, _Alloc>::push_back(std::vector<_Tp, _Alloc>::value_type&&) [with _Tp = slam::KeyPoint; _Alloc = Eigen::aligned_allocator<slam::KeyPoint>; std::vector<_Tp, _Alloc>::value_type = slam::KeyPoint]
       push_back(value_type&& __x)
       ^~~~~~~~~
/usr/include/c++/7/bits/stl_vector.h:953:7: note:   no known conversion for argument 1 from ‘<brace-enclosed initializer list>’ to ‘std::vector<slam::KeyPoint, Eigen::aligned_allocator<slam::KeyPoint> >::value_type&& {aka slam::KeyPoint&&}’
src/slam/CMakeFiles/slam.dir/build.make:283: recipe for target 'src/slam/CMakeFiles/slam.dir/orb_extractor.cpp.o' failed
make[2]: *** [src/slam/CMakeFiles/slam.dir/orb_extractor.cpp.o] Error 1
CMakeFiles/Makefile2:340: recipe for target 'src/slam/CMakeFiles/slam.dir/all' failed
make[1]: *** [src/slam/CMakeFiles/slam.dir/all] Error 2
Makefile:94: recipe for target 'all' failed
make: *** [all] Error 2

Dear sir, Thanks for the excellent work! i have a question about the parameters "-maxVisualUpdates" and "-maxSuccessfulVisualUpdates",their default value are both 20.When i set them both to 30,the mono vio can also run in realtime in euroc, but the result seems to be the same as the 20's.Is that right?But I think that the more visual updates are done,the better result that we should get. Thanks again!

Dear Professor: Sorry for bothering you again. Are there some docs about the dy/dx and the dy/dq in the prediction function？It seems it is different from the EKF Based algorithm.

Love the paper, thank you so much for putting it and the code out there!

When I was trying to recreate the paper results, I noticed that my EUROC results match but TUM-VI did not. Looking at the paper, I found: Table B2 online-stereo-Normal SLAM has identical RMSE to postprocess (Table 4)

I suspect that this is just a typo although I could be wrong here.

Cheers and all the best!

hello Thank for you sharing so good job I build on ubuntu1804 system When I cmake the project after finished the build.sh file, I encoutered the problem so follow:

Hi, thanks for your beautiful work! I got a question about the jacobian at: https://github.com/SpectacularAI/HybVIO/blob/main/src/odometry/triangulation.cpp#L963

In my understanding, its about such a question: P_{point_in_camera} = R^{cam}{world} * P{point_in_world} and R = R(q). then we want to get the jacobian of d(P_{point_in_camera}) / d (q), why this jacobian is related to camera and imu baseline parameter?

I am trying to make equirectangular images + IMU available on your code, do you think this is a practicable idea?

Hello, thank you very much for opening up such great work. I'm looking at the details of the code recently. But at present, some problems have been bothering me. How do you deal with the rolling shutter problem? I don't see the relevant parameter settings in your code, such as "rolling_shutter_skew" or "rolling_shutter_readout_time". Thank you again, and look forward to your reply.

Dear Professor： Recently, I have read the paper 《HybVIO: Pushing the Limits of Real-time Visual-inertial Odometry》and its corresponding code. Thank you for your wonderful work that contributes to the robotics community. However, there are some troubles bothering me a lot. I have noticed that the visual landmark estimation part(https://github.com/SpectacularAI/HybVIO/blob/main/src/odometry/triangulation.cpp#L203) is different from the triangulation part in the original msckf. That's the beauty of the PIVO, your previous paper, Right？ However, I don't understand, in the landmark triangulation part, why could you estimate the landmark coordinate jacobian with respect to the camera pose in the trail. To my knowledge, in this part, we would only estimate the landmark position and should calculate the jacobian with the landmark coordinate, like this: More precisely, I don't understand how to derive the analytical-formula of the jacobian dE^TE in the triangulation part. I have read your paper《PIVO: Probabilistic Inertial-Visual Odometry for Occlusion-Robust Navigation》 again and again. However, the paper is concise and I am not clever enough to understand why to calculate the jacobian like this. Could you please give me some doc or clues about it? I'm really looking forward for your help, thank you very much. Yours Qi Wu

Platform

Macbook Air M2

OS

Ubuntu 20.04 Docker

Problem

I was trying to build dependencies by running

CC=clang-12 CXX=clang++-12 WITH_OPENGL=OFF BUILD_VISUALIZATIONS=OFF ./scripts/build.sh

and I got the following error:

-- Reading vars from /HybVIO/3rdparty/mobile-cv-suite/OpenBLAS/kernel/arm64/KERNEL.HASWELL...
CMake Error at cmake/utils.cmake:20 (file):
  file STRINGS file
  "/HybVIO/3rdparty/mobile-cv-suite/OpenBLAS/kernel/arm64/KERNEL.HASWELL"
  cannot be read.
Call Stack (most recent call first):
  kernel/CMakeLists.txt:16 (ParseMakefileVars)
  kernel/CMakeLists.txt:863 (build_core)

I have looked in the 3rdparty/mobile-cv-suite/OpenBLAS/kernel/arm64/ directory and there was no KERNEL.HASWELL in the directory.

Hello, I have a pair of stereo cameras from a VR headset that point at different angles. I've been able to get HybVIO to track stereo features only with -useRectification, but even then it doesn't seem possible for HybVIO to track features on the non-overlapping areas of the images.

1. Is there a way to avoid having to prerectify the images in this case?
2. Is it possible to track non-overlapping features in HybVIO?

Below is a stereo frame example obtained by running this command:

./main -i=../data/benchmark/ody-easy -p -c -s -windowResolution=640 -useSlam -useStereo -displayStereoMatching -useRectification

Over this HybVIO-formatted dataset. The original EuRoC-formatted dataset can be found here.