Hi, I had a problem when I ran the code "python stray_visualize.py --point-clouds".

When I run the code with example datasets "ZB2", I got an error:

Traceback (most recent call last): File "/home/user/Desktop/new1/sv5/stray_visualize.py", line 206, in main() File "/home/user/Desktop/new1/sv5/stray_visualize.py", line 190, in main data = read_data(flags) File "/home/user/Desktop/new1/sv5/stray_visualize.py", line 53, in read_data T_WC[:3, :3] = Rotation.from_quat(quaternion).as_matrix() File "rotation.pyx", line 624, in scipy.spatial.transform.rotation.Rotation.from_quat File "rotation.pyx", line 513, in scipy.spatial.transform.rotation.Rotation.init ValueError: Expected quat to have shape (4,) or (N x 4), got (2,).

I have no idea why this error occurred. Thank you for reading my issues, hope you reply.

Hello. Thank you for sharing.

I wonder what the timestamp of odometry.csv means.

I thought this was a unix timestamp, so I converted it to human date time, and although it was a timestamp obtained by the Stray scanner app on July 27, 2022, the time zone came out on January 1, 1970.

How do I get the current unix timestamp or date time in real-time through the Stray scanner app?

Hi Kenneth,

I have been looking for an app like this, to record RGB + depth images from the iPhone/iPad LiDAR sensor. Very cool!! :) Could I ask a few questions about the data acquisition process and data itself??

1) alignment (registration) between RGB and depth images It seems like the RGB, depth image resolutions are fixed as 1920x1440 / 256x192 in ARKit mode. (right?) If we downsample RGB as 1/7.5, then their sizes are exactly the same, showing good RGB and depth alignment.

I am very curious that the alignment (registration) between RGB and depth images is already done? (no need for camera extrinsic calibration parameters between the RGB and depth cameras?) RGB and depth images share the same camera intrinsic parameters stored in camera_matrix.csv? RGB's K will be the matrix in the camera_matrix.csv file, and depth's K will be RGB's K / 7.5,

Is this the correct understanding?

2) iPad Pro vs. iPhone 12 Pro Max It seems like the same RGB, depth image resolution is used in both iPad Pro and iPhone 12 Pro Max (AFAIK). Is there any difference in RGB, depth, and confidence data when recording datasets on both iPad Pro and iPhone 12 Pro Max?

3) adjustment of recoding FPS Is it possible to adjust the FPS setting of RGB, depth, and confidence images? It seems like the default FPS (60 FPS, right?) is too fast sometimes, so the data size grows too fast.... It would be really nice to have a setting that allows the user to set the FPS before data acquisition in the app.

4) camera_matrix.csv As far as I know, Apple ARKit estimates and provides K (camera matrix) for every frame. How is the K (camera matrix) determined? from the very first frame or the last frame??

5) depth image from depthMap As far as I know, we can access the depth map from the LiDAR sensor through didUpdate frame: ARFrame. (Swift) In this app, is the depth map to be saved with frame.sceneDepth?.depthMap or frame.smoothedSceneDepth?.depthMap ? Also, I am just curious what is the difference between them....

6) depth map format (npy) Is there any specific reason to save the depth map as .npy format? I think we can just directly save the depth map as .png format similar to confidence image..

I really appreciate your time and concerns for my several questions. Please let me know if you have any questions.

Thanks, Pyojin Kim.

I have a recording which has 631 frames (depth, confidence, odometry, etc), however the .mp4 of coloured images only has 627 frames encoded. It is relatively common for FFPMEG to drop frames when encoding to .mp4 when using the -r flag and not the -vf flag. I think that's what's happening here, and it's encoding with a variable frame-rate and dropping frames (this is in fact what is happening here). This makes the mp4 not very useful for aligning the rgb and depth.

One solution is to change the ffmpeg command used for encoding to RGB

A better solution is to output the frames as images rather than video already.

Finally last point, it would be MUCH more convenient if files from the app could be exported (as zipped) to the 'files' app on ios and then we could simply click the zip and airdrop it over to the computer and not have to deal with itunes.

Hope this feedback is helpful.

Cheers, Jono

Hi, I have recently used Stray Scanner. I found the resolution of depth image is 256*192. Is it right? I want to know the real resolution of iphone 12 pro's lidar, but I can't find the parameter through search engines and Apple offical documents. Can you tell me? Look forward to your apply.

Thanks so much for the great software of stray command toolkit, however I installed this tool using the following shell command "curl --proto '=https' --tlsv1.2 -sSf https://stray-builds.ams3.digitaloceanspaces.com/cli/install.sh | bash" After installed, nothing works, I can't use stray command, I don't know the reason and how to make it works. So I'm looking for your kind help. My system is Linux. Is it because that I don't have a STRAY_LICENSE_KEY=key? and how can I get this key? or some other reason else. I want use stray command toolkit to make a 3D reconstution pipeline based RGB-D sequences acquired by IPhone 12 pro device.

Hi, I have used Stray Scanner app and I'm really curious about the odometry information. I have tried to develop a small app to get quaternion (simd_quatf(frame.camera.transform)) and the position to get the extrinsic camera. But when I input the dataset from my app to Stray Visualizer, it's not show okay. Do I have to scale or apply anything to odometry? If I need to scale, What ratio I should to scale by? Could your share me how to calculate the odometry each frames. Thank you, hope you reply.

The first two items from odometry.csv should be discarded, since the file is organized as

timestamp, frame, x, y, z, qx, qy, qz, w

The indices of each camera position should be [2:5], and the indices of each quaternion should be [5:].

Signed-off-by: Gaoyang Zhang [email protected]

I have gotten quite poor results using the full Open3D reconstruction process, while using StrayVisualizer's --integrate works much better. The main difference is I am only using 1/10th of the frames for Open3D, since using all frames takes forever.

Have you been able to achieve good results with Open3D? I'm not sure if there are some parameters I need to adjust.

Hi,

there was a problem when I tried to create a video from the depth maps

as I am not using iphone 12 yet, I downloaded the example datasets which you uploaded (ZB1, ZB2, ZB3)

and when I tried to run the code "python make_video.py ", I ran into the error :

Traceback (most recent call last): File "make_video.py", line 39, in main() File "make_video.py", line 36, in main writer.close() File "/opt/anaconda3/envs/stray_scanner/lib/python3.7/site-packages/skvideo/io/abstract.py", line 474, in close if self._proc is None: # pragma: no cover AttributeError: 'FFmpegWriter' object has no attribute '_proc'

I have no idea what to do. I checked the version of scikit-video and tried it to be re-installed, but didn't work.

Hei,

I am try to create a depth video from my lidarscan. I am using python 3.7.7 (otherwise i can't install open3d) and numpy 1.20.3

If I use the make_video.py I get the error:

Traceback (most recent call last):
  File "make_video.py", line 39, in <module>
    main()
  File "make_video.py", line 22, in main
    depth = np.load(path)
  File "/usr/local/lib/python3.7/site-packages/numpy/lib/npyio.py", line 445, in load
    raise ValueError("Cannot load file containing pickled data "
ValueError: Cannot load file containing pickled data when allow_pickle=False

So if I ad depth = np.load(path, allow_pickle=True) there is a new Error i don't quit get.

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/numpy/lib/npyio.py", line 448, in load
    return pickle.load(fid, **pickle_kwargs)
_pickle.UnpicklingError: invalid load key, '\x00'.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "make_video.py", line 39, in <module>
    main()
  File "make_video.py", line 22, in main
    depth = np.load(path, allow_pickle=True)
  File "/usr/local/lib/python3.7/site-packages/numpy/lib/npyio.py", line 451, in load
    "Failed to interpret file %s as a pickle" % repr(file)) from e
OSError: Failed to interpret file '/Users/jonasvonarb/Downloads/8a42b3d040/depth/.DS_Store' as a pickle

Do you have an Idea how to solve this?

Hi, I am very new to open 3D and iOS programming. And I encounter a problem with the convert to open 3D program. I was able to run the program without any problem, but after it said done, I saw that that depth folder was empty. So I copied from the original raw data depth folder directly, but when I ran the 3D reconstruction, I was unable to run it. I show the following error:

I suspect the error is due to the size different between the image RGB and the depth data, but I am not sure? May I know many recommendations on the fix or other ways to import those data into open 3d?

OpenCV is detected. Using ORB + 5pt algorithm making fragments from RGBD sequence. Fragment 000 / 019 :: RGBD matching between frame : 0 and 1 [Open3D Error] (class std::shared_ptr __cdecl open3d::geometry::RGBDImage::CreateFromColorAndDepth(const class open3d::geometry::Image &,const class open3d::geometry::Image &,double,double,bool)) D:\a\Open3D\Open3D\cpp\open3d\geometry\RGBDImageFactory.cpp:40: Unsupported image format.

Traceback (most recent call last): File "C:\Users\elsto\Documents\GitHub\CityU-FYP-Metaversa-3DModel\Trial\Open3D-0.16.0\examples\python\reconstruction_system\run_system.py", line 130, in make_fragments.run(config) File "C:\Users\elsto\Documents\GitHub\CityU-FYP-Metaversa-3DModel\Trial\Open3D-0.16.0\examples\python\reconstruction_system\make_fragments.py", line 203, in run process_single_fragment(fragment_id, color_files, depth_files, File "C:\Users\elsto\Documents\GitHub\CityU-FYP-Metaversa-3DModel\Trial\Open3D-0.16.0\examples\python\reconstruction_system\make_fragments.py", line 174, in process_single_fragment make_posegraph_for_fragment(config["path_dataset"], sid, eid, color_files, File "C:\Users\elsto\Documents\GitHub\CityU-FYP-Metaversa-3DModel\Trial\Open3D-0.16.0\examples\python\reconstruction_system\make_fragments.py", line 94, in make_posegraph_for_fragment info] = register_one_rgbd_pair(s, t, color_files, depth_files, File "C:\Users\elsto\Documents\GitHub\CityU-FYP-Metaversa-3DModel\Trial\Open3D-0.16.0\examples\python\reconstruction_system\make_fragments.py", line 50, in register_one_rgbd_pair source_rgbd_image = read_rgbd_image(color_files[s], depth_files[s], True, File "C:\Users\elsto\Documents\GitHub\CityU-FYP-Metaversa-3DModel\Trial\Open3D-0.16.0\examples\python\open3d_example.py", line 213, in read_rgbd_image rgbd_image = o3d.geometry.RGBDImage.create_from_color_and_depth( RuntimeError: [Open3D Error] (class std::shared_ptr __cdecl open3d::geometry::RGBDImage::CreateFromColorAndDepth(const class open3d::geometry::Image &,const class open3d::geometry::Image &,double,double,bool)) D:\a\Open3D\Open3D\cpp\open3d\geometry\RGBDImageFactory.cpp:40: Unsupported image format.