Robust Consistent Video Depth Estimation

Related tags

Deep Learning robust_cvd
Overview

[CVPR 2021] Robust Consistent Video Depth Estimation

Open in Colab

This repository contains Python and C++ implementation of Robust Consistent Video Depth, as described in the paper

Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust Consistent Video Despth Estimation. CVPR 2021

Project | Paper | Video | Colab

We present an algorithm for estimating consistent dense depth maps and camera poses from a monocular video. We integrate a learning-based depth prior, in the form of a convolutional neural network trained for single-image depth estimation, with geometric optimization, to estimate a smooth camera trajectory as well as detailed and stable depth reconstruction.

teaser

Changelog

  • [June 2021] Released the companion Colab notebook.
  • [June 2021] Initial release of Robust CVD.

Installation

Please refer to the colab notebook for how to install the dependencies.

Running

Please refer to the colab notebook for how to run the cli tool for now.

Result Folder Structure

frames.txt              # meta data about number of frames, image resolution and timestamps for each frame
color_full/             # extracted frames in the original resolution
color_down/             # extracted frames in the resolution for disparity estimation 
color_down_png/      
color_flow/             # extracted frames in the resolution for flow estimation
flow_list.json          # indices of frame pairs to finetune the model with
flow/                   # optical flow 
mask/                   # mask of consistent flow estimation between frame pairs.
vis_flow/               # optical flow visualization. Green regions contain inconsistent flow. 
vis_flow_warped/        # visualzing flow accuracy by warping one frame to another using the estimated flow. e.g., frame_000000_000032_warped.png warps frame_000032 to frame_000000.
depth_${model_type}/    # initial disparity estimation using the original monocular depth model before test-time training
R_hierarchical2_${model_type}/ 
    flow_list_0.20.json                 # indices of frame pairs passing overlap ratio test of threshold 0.2. Same content as ../flow_list.json.
    videos/                             # video visualization of results 
    B0.1_R1.0_PL1-0_LR0.0004_BS4_Oadam/
        checkpoints/                    # checkpoint after each epoch
        depth/                          # final disparity map results after finishing test-time training
        eval/                           # intermediate losses and disparity maps after each epoch 
        tensorboard/                    # tensorboard log for the test-time training process

Citation

If you find our work useful in your research, please consider citing:

@inproceedings{kopf2021rcvd,
 title={Robust Consistent Video Depth Estimation},
 author={Kopf, Johannes and Rong, Xuejian and Huang, Jia-Bin},
 year={2021},
 booktitle=IEEE/CVF Conference on Computer Vision and Pattern Recognition
}

License

See the LICENSE for more details.

Issues & Help

For help or issues using Robust CVD, please submit a GitHub issue or a PR request.

Before you do this, make sure you have checked CODE_OF_CONDUCT, CONTRIBUTING, ISSUE_TEMPLATE, and PR_TEMPLATE.

Acknowledgements

Check our previous work on Consistent Video Depth Estimation.

We also thank the authors for releasing PyTorch, Ceres Solver, OpenCV, Eigen, MiDaS, RAFT, and detectron2.

Comments
  • while building make on Ubuntu 22.04 I am getting this error. can anyone please help me with this?

    while building make on Ubuntu 22.04 I am getting this error. can anyone please help me with this?

    /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:163:1: error: reference to ‘Vector3’ is ambiguous 163 | Vector3 obsToCamera( | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:175:1: error: reference to ‘Vector3’ is ambiguous 175 | Vector3 cameraToWorld( | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:196:1: error: reference to ‘Vector3’ is ambiguous 196 | Vector3 worldToCamera( | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp: In member function ‘bool facebook::cp::{anonymous}::SceneFlowSmoothnessLoss::operator()(const T* const*, T*) const’: /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:338:5: error: reference to ‘Vector3’ is ambiguous 338 | Vector3 pointCam0 = obsToCamera(obs0, params0); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:338:14: error: expected primary-expression before ‘>’ token 338 | Vector3 pointCam0 = obsToCamera(obs0, params0); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:338:16: error: ‘pointCam0’ was not declared in this scope 338 | Vector3 pointCam0 = obsToCamera(obs0, params0); | ^~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:341:5: error: reference to ‘Vector3’ is ambiguous 341 | Vector3 pointCam1 = obsToCamera(obs1, params1); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:341:14: error: expected primary-expression before ‘>’ token 341 | Vector3 pointCam1 = obsToCamera(obs1, params1); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:341:16: error: ‘pointCam1’ was not declared in this scope 341 | Vector3 pointCam1 = obsToCamera(obs1, params1); | ^~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:344:5: error: reference to ‘Vector3’ is ambiguous 344 | Vector3 pointCam2 = obsToCamera(obs2, params2); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:344:14: error: expected primary-expression before ‘>’ token 344 | Vector3 pointCam2 = obsToCamera(obs2, params2); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:344:16: error: ‘pointCam2’ was not declared in this scope 344 | Vector3 pointCam2 = obsToCamera(obs2, params2); | ^~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:347:5: error: reference to ‘Vector2’ is ambiguous 347 | Vector2 focal0, focal1, focal2; | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector2 = Eigen::Matrix<Type, 2, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:540:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 540 | EIGEN_MAKE_TYPEDEFS(2, 2) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:31:7: note: ‘template using Vector2 = Eigen::Matrix<Type, 2, 1>’ 31 | using Vector2 = Matrix<T, 2, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:347:14: error: expected primary-expression before ‘>’ token 347 | Vector2 focal0, focal1, focal2; | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:347:16: error: ‘focal0’ was not declared in this scope 347 | Vector2 focal0, focal1, focal2; | ^~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:347:24: error: ‘focal1’ was not declared in this scope 347 | Vector2 focal0, focal1, focal2; | ^~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:347:32: error: ‘focal2’ was not declared in this scope 347 | Vector2 focal0, focal1, focal2; | ^~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:362:18: error: template argument 1 is invalid 362 | Map<Vector3> smoothLoss(residuals); | ^~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:369:7: error: reference to ‘Vector3’ is ambiguous 369 | Vector3 pointWorld0 = cameraToWorld(pointCam0, focal0, params0.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:369:16: error: expected primary-expression before ‘>’ token 369 | Vector3 pointWorld0 = cameraToWorld(pointCam0, focal0, params0.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:369:18: error: ‘pointWorld0’ was not declared in this scope 369 | Vector3 pointWorld0 = cameraToWorld(pointCam0, focal0, params0.pose); | ^~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:370:7: error: reference to ‘Vector3’ is ambiguous 370 | Vector3 pointWorld1 = cameraToWorld(pointCam1, focal1, params1.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:370:16: error: expected primary-expression before ‘>’ token 370 | Vector3 pointWorld1 = cameraToWorld(pointCam1, focal1, params1.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:370:18: error: ‘pointWorld1’ was not declared in this scope 370 | Vector3 pointWorld1 = cameraToWorld(pointCam1, focal1, params1.pose); | ^~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:371:7: error: reference to ‘Vector3’ is ambiguous 371 | Vector3 pointWorld2 = cameraToWorld(pointCam2, focal2, params2.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:371:16: error: expected primary-expression before ‘>’ token 371 | Vector3 pointWorld2 = cameraToWorld(pointCam2, focal2, params2.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:371:18: error: ‘pointWorld2’ was not declared in this scope 371 | Vector3 pointWorld2 = cameraToWorld(pointCam2, focal2, params2.pose); | ^~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:375:7: error: reference to ‘Vector3’ is ambiguous 375 | Vector3 pointWorld0 = cameraToWorld(pointCam0, focal0, params0.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:375:16: error: expected primary-expression before ‘>’ token 375 | Vector3 pointWorld0 = cameraToWorld(pointCam0, focal0, params0.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:375:18: error: ‘pointWorld0’ was not declared in this scope 375 | Vector3 pointWorld0 = cameraToWorld(pointCam0, focal0, params0.pose); | ^~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:376:7: error: reference to ‘Vector3’ is ambiguous 376 | Vector3 pointWorld2 = cameraToWorld(pointCam2, focal2, params2.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:376:16: error: expected primary-expression before ‘>’ token 376 | Vector3 pointWorld2 = cameraToWorld(pointCam2, focal2, params2.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:376:18: error: ‘pointWorld2’ was not declared in this scope 376 | Vector3 pointWorld2 = cameraToWorld(pointCam2, focal2, params2.pose); | ^~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:378:7: error: reference to ‘Vector3’ is ambiguous 378 | Vector3 pointCam0To1 = worldToCamera(pointWorld0, focal1, params1.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:378:16: error: expected primary-expression before ‘>’ token 378 | Vector3 pointCam0To1 = worldToCamera(pointWorld0, focal1, params1.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:378:18: error: ‘pointCam0To1’ was not declared in this scope 378 | Vector3 pointCam0To1 = worldToCamera(pointWorld0, focal1, params1.pose); | ^~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:379:7: error: reference to ‘Vector3’ is ambiguous 379 | Vector3 pointCam2To1 = worldToCamera(pointWorld2, focal1, params1.pose); | ^~~~~~~ In file included from /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/Core:295, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/core/ParamsBase.h:8, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.h:7, from /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:3: /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:524:7: note: candidates are: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 524 | using Vector##SizeSuffix = Matrix<Type, Size, 1>;
    | ^~~~~~ /home/nadeem/anaconda3/envs/depth/include/eigen3/Eigen/src/Core/Matrix.h:541:1: note: in expansion of macro ‘EIGEN_MAKE_TYPEDEFS’ 541 | EIGEN_MAKE_TYPEDEFS(3, 3) | ^~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:34:7: note: ‘template using Vector3 = Eigen::Matrix<Type, 3, 1>’ 34 | using Vector3 = Matrix<T, 3, 1>; | ^~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:379:16: error: expected primary-expression before ‘>’ token 379 | Vector3 pointCam2To1 = worldToCamera(pointWorld2, focal1, params1.pose); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:379:18: error: ‘pointCam2To1’ was not declared in this scope 379 | Vector3 pointCam2To1 = worldToCamera(pointWorld2, focal1, params1.pose); | ^~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:381:18: error: request for member ‘x’ in ‘smoothLoss’, which is of non-class type ‘int’ 381 | smoothLoss.x() = | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:384:18: error: request for member ‘y’ in ‘smoothLoss’, which is of non-class type ‘int’ 384 | smoothLoss.y() = | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:394:20: error: request for member ‘z’ in ‘smoothLoss’, which is of non-class type ‘int’ 394 | smoothLoss.z() = repro0To1Disp + repro2To1Disp - disp1 * 2.0; | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:403:22: error: request for member ‘z’ in ‘smoothLoss’, which is of non-class type ‘int’ 403 | smoothLoss.z() = (maxDepth / minDepth - 1.0); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:405:22: error: request for member ‘z’ in ‘smoothLoss’, which is of non-class type ‘int’ 405 | smoothLoss.z() = log(minDepth / maxDepth); | ^ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp: In instantiation of ‘bool facebook::cp::{anonymous}::SceneFlowSmoothnessLoss::operator()(const T     const    , T    ) const [with T = double]’: /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:93:25: required from ‘bool ceres::DynamicAutoDiffCostFunction<CostFunctor, Stride>::Evaluate(const double const, double, double) const [with CostFunctor = facebook::cp::{anonymous}::SceneFlowSmoothnessLoss; int Stride = 4]’ /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:85:16: required from here /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:362:32: error: invalid conversion from ‘double’ to ‘int’ []8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-fpermissive-fpermissive]8;;] 362 | Map<Vector3> smoothLoss(residuals); | ^~~~~~~~~ | | | double* /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp: In instantiation of ‘bool facebook::cp::{anonymous}::SceneFlowSmoothnessLoss::operator()(const T* const*, T*) const [with T = ceres::Jet<double, 4>]’: /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:203:23: required from ‘bool ceres::DynamicAutoDiffCostFunction<CostFunctor, Stride>::Evaluate(const double* const*, double*, double**) const [with CostFunctor = facebook::cp::{anonymous}::SceneFlowSmoothnessLoss; int Stride = 4]’ /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:85:16: required from here /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:362:32: error: invalid conversion from ‘ceres::Jet<double, 4>’ to ‘int’ []8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-fpermissive-fpermissive]8;;] 362 | Map<Vector3> smoothLoss(residuals); | ^~~~~~~~~ | | | ceres::Jet<double, 4> /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp: In instantiation of ‘bool facebook::cp::{anonymous}::StaticSceneCost::operator()(const T* const*, T*) const [with T = double]’: /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:93:25: required from ‘bool ceres::DynamicAutoDiffCostFunction<CostFunctor, Stride>::Evaluate(const double* const*, double*, double**) const [with CostFunctor = facebook::cp::{anonymous}::StaticSceneCost; int Stride = 4]’ /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:85:16: required from here /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:259:39: error: ‘obsToCamera’ was not declared in this scope 259 | Vector3 pointCam0 = obsToCamera(obs0, params0); | ~~~~~~~~~~~^~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:261:22: error: ‘cameraToWorld’ was not declared in this scope 261 | cameraToWorld(pointCam0, focal0, params0.pose); | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:263:39: error: ‘obsToCamera’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation []8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-fpermissive-fpermissive]8;;] 263 | Vector3 pointCam1 = obsToCamera(obs1, params1); | ~~~~~~~~~~~^~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:270:45: error: ‘cameraToWorld’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation []8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-fpermissive-fpermissive]8;;] 270 | Vector3 pointWorld1 = cameraToWorld(pointCam1, focal1, params1.pose); | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:281:24: error: ‘worldToCamera’ was not declared in this scope 281 | worldToCamera(pointWorld0, focal1, params1.pose); | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp: In instantiation of ‘bool facebook::cp::{anonymous}::StaticSceneCost::operator()(const T const, T*) const [with T = ceres::Jet<double, 4>]’: /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:203:23: required from ‘bool ceres::DynamicAutoDiffCostFunction<CostFunctor, Stride>::Evaluate(const double* const*, double*, double**) const [with CostFunctor = facebook::cp::{anonymous}::StaticSceneCost; int Stride = 4]’ /home/nadeem/anaconda3/envs/depth/include/ceres/dynamic_autodiff_cost_function.h:85:16: required from here /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:259:39: error: ‘obsToCamera’ was not declared in this scope 259 | Vector3 pointCam0 = obsToCamera(*obs0, params0); | ~~~~~~~~~~~^~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:261:22: error: ‘cameraToWorld’ was not declared in this scope 261 | cameraToWorld(pointCam0, focal0, params0.pose); | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:263:39: error: ‘obsToCamera’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation []8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-fpermissive-fpermissive]8;;] 263 | Vector3 pointCam1 = obsToCamera(*obs1, params1); | ~~~~~~~~~~~^~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:270:45: error: ‘cameraToWorld’ was not declared in this scope, and no declarations were found by argument-dependent lookup at the point of instantiation []8;;https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-fpermissive-fpermissive]8;;] 270 | Vector3 pointWorld1 = cameraToWorld(pointCam1, focal1, params1.pose); | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/PoseOptimizer.cpp:281:24: error: ‘worldToCamera’ was not declared in this scope 281 | worldToCamera(pointWorld0, focal1, params1.pose); | ~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ make[2]: Leaving directory '/home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/build' make[1]: Leaving directory '/home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/build' make: Leaving directory '/home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/build' make[2]: *** [CMakeFiles/lib_cvd.dir/build.make:216: CMakeFiles/lib_cvd.dir/PoseOptimizer.o] Error 1 make[1]: *** [CMakeFiles/Makefile2:85: CMakeFiles/lib_cvd.dir/all] Error 2 make: *** [/home/nadeem/Downloads/robust_cvd/lib/cvd2/lib/build/Makefile:91: all] Error 2

    Process finished with exit code 2

    opened by muhammadnadeem23064 6
  • AttributeError: module 'cv2.cv2' has no attribute 'xfeatures2d'

    AttributeError: module 'cv2.cv2' has no attribute 'xfeatures2d' 

    ## Flow ##
Sampled 522 frame pairs.
Resizing flow to (224, 384)
Traceback (most recent call last):
  File "main.py", line 20, in <module>
    dp.process()
  File "/home/ubuntu/Downloads/cvd2/process.py", line 234, in process
    return self.pipeline()
  File "/home/ubuntu/Downloads/cvd2/process.py", line 171, in pipeline
    self.compute_flow()
  File "/home/ubuntu/Downloads/cvd2/process.py", line 135, in compute_flow
    self.flow.compute_flow(frame_pairs, self.params.flow_model)
  File "/home/ubuntu/Downloads/cvd2/flow.py", line 124, in compute_flow
    optical_flow_homography.process(args)
  File "/home/ubuntu/Downloads/cvd2/optical_flow_homography.py", line 269, in process
    flow = infer(args, OFlowModel, DEVICE, im1, im2)
  File "/home/ubuntu/Downloads/cvd2/optical_flow_homography.py", line 177, in infer
    img1, img2, img2_reg, H_BA = getimage(img1_name, img2_name)
  File "/home/ubuntu/Downloads/cvd2/optical_flow_homography.py", line 149, in getimage
    (kpsA, featuresA) = detectAndDescribe(frame1)
  File "/home/ubuntu/Downloads/cvd2/optical_flow_homography.py", line 69, in detectAndDescribe
    descriptor = cv2.xfeatures2d.SURF_create()
 AttributeError: module 'cv2.cv2' has no attribute 'xfeatures2d'

    >>> import cv2
>>> print(cv2.__version__)
4.5.2
>>> cv2.xfeatures2d.SURF_create()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'cv2.cv2' has no attribute 'xfeatures2d'
    opened by karliell 3
  • Is it possible?

    Is it possible?

    Is it possible to pause the Colab script execution after depth_midas2/depth get generated, substitute folder content by depth I've generated using Midas 3. And then continue processing? Or just upload my depth_midas2 folder and skip midas2 generation step? Is it possible to obtain a metric depth map using this network. Thank you in advance!

    opened by semel1 3
  • Main program running time is too long...

    Main program running time is too long...

    I try to run your code on google colab, but the running time for main.py is very long(more than 2 hours even using demo video), the program gets stuck in this position every time. Is this normal? I am not colab pro. image

    opened by zhuhu00 2
  • About downloading models

    About downloading models

    wget https://www.dropbox.com/s/v1oc39d8k5fa1zx/models.zip?dl=1 -O models.zip

    I tried many methods many times, such as wget and browser, but all failed to download the model. Is there any other download link? For example, Baidu Netdisk?

    opened by Minisal 2
  • Unable to download pretrained models

    Unable to download pretrained models

    Hi!

    The dropbox download link gives a 404 and the alternative download link needs a WeChat account for downloading the models.

    I kindly ask you to please share the models in a more straightforward way.

    Thank you and best regards!

    opened by AlverGant 1
  • How to get the true depth value for every pixel?

    How to get the true depth value for every pixel?

    I don't quite understand why you use

    https://github.com/facebookresearch/robust_cvd/blob/main/depth_fine_tuning.py#L269

    inv_depth_batch = 1.0 / depth_batch

            for i in range(batch_size):
                if batch_size > 1:
                    inv_depth = inv_depth_batch[i, :, :].squeeze()
                else:
                    inv_depth = inv_depth_batch
                frame_id = metadata["frame_id"][i]

                if frames is None or frame_id in frames:
                    image_io.save_raw_float32_image(
                        depth_fmt.format(frame_id) + ".raw", inv_depth
                    )

    why does inverse necessary? How to get true depth value? Thanks a lot for your help!!!!!!!!

    opened by karliell 1
  • Meaning of parameter `--frame_range 0-10`

    Meaning of parameter `--frame_range 0-10`

    Thanks for the amazing project! I'm wondering the meaning of this parameter: --frame_range 0-10

    In params.py file it says:

            self.parser.add_argument(
            "--frame_range", default="",
            type=frame_range.parse_frame_range,
            help="Range of depth to fine-tune, e.g., 0,2-10,21-40."
        )

    I set it to 10 and in the end I got 10 raw file in

    ./R0-10_hierarchical2_midas2/StD100.0_StR1.0_SmD0_SmR0.0/depth_e0000/e0000_filtered/depth/

    Does this mean it only processes the first 10 frames in the video? Is there any easy way to make it process the whole video (all frames) automatically? Thanks a lot!

    opened by karliell 1
  • Why images get downscaled?

    Why images get downscaled?

    Why all processed streams down scaled? Is there is a way to process and obtain depth in the same resolution as input source (without upscale processed images back to the original resolution)?

    opened by semel1 1
  • Problems in your previous version

    Problems in your previous version

    Hey there! I run your previous version of code and got half of the results and met this problem:

    https://github.com/facebookresearch/consistent_depth/issues/4#issuecomment-680177616

    May I ask could you please help me with this?

    opened by karliell 1
  • Adding Code of Conduct file

    Adding Code of Conduct file

    This is pull request was created automatically because we noticed your project was missing a Code of Conduct file.

    Code of Conduct files facilitate respectful and constructive communities by establishing expected behaviors for project contributors.

    This PR was crafted with love by Facebook's Open Source Team.

    CLA Signed 
    opened by facebook-github-bot 0
  • RTX Compatibility / PyTorch 1.8.0?

    RTX Compatibility / PyTorch 1.8.0?

    I'm having trouble getting RCVD to work with a RTX 3080 Ti card. It needs the PyTorch 1.8.0 in order to work but then RCVD doesn't work. Is there any way around this? Is it possible to use another version of PyTorch that works with RCVD and RTX 30 cards?

    opened by toooft 0
  • contrast loss

    contrast loss

    Hello I have noticed that there is an additional loss used by default in your algorithm, yet it wasn't described in the paper. It is defined in loss/contrast_loss.py andit can be disabled by setting "--lambda_contrast_loss 0". If I disable this loss, the final result appears to be extremely blurry. Could someone explain where this loss comes from and what it does? Was it taken from some other project? Thank you

    opened by lkosh 0
  • No Module Named 'lib_python' Error In Colab.

    No Module Named 'lib_python' Error In Colab.

    I managed to download Detectron2 using other methods than the ones provided and switched the OpenCV version from 3.4.2.16 to 3.4.2.17 since you can no longer install 16. Now I get no errors while installing everything, except that when I go to run the Robust CVD code, it tells me that lib_python is not installed. Here is the code: /content/cvd2 ['/content/cvd2', '/env/python', '/usr/lib/python37.zip', '/usr/lib/python3.7', '/usr/lib/python3.7/lib-dynload', '/usr/local/lib/python3.7/dist-packages', '/content/detectron2_repo', '/usr/lib/python3/dist-packages', '/content/cvd2/main.py', 'lib/build'] Traceback (most recent call last): File "main.py", line 12, in <module> from params import Video3dParamsParser File "/content/cvd2/params.py", line 8, in <module> from depth_fine_tuning import DepthFineTuningParams File "/content/cvd2/depth_fine_tuning.py", line 30, in <module> from loaders.video_dataset import VideoDataset, VideoFrameDataset File "/content/cvd2/loaders/video_dataset.py", line 16, in <module> from lib_python import (DepthVideo, ValueXformType, DepthXformType, SpatialXformType) ModuleNotFoundError: No module named 'lib_python'

    opened by FoxLynx64 4
  • How to use .raw files?

    How to use .raw files?

    Such a fantastic library, thanks a lot for your work!

    I need help with two things:

    1. I'm struggling to figure out how to open the .raw files and would like some help - I'm sure it's easy but I can't load them into any image software I own. How do I convert them to .png for instance?

    2. In the colab, what script language is the Result Visualization part made in? I can't seem to make it work on Ubuntu 20.04 LTS but I'm sure it's easy as well.

    opened by toooft 3
  • is it possible to run on mobile devices?

    is it possible to run on mobile devices?

    Dear all: I was shocked by the consistent/robust video depth estimation series work! it's so promising. But i also realised that the efficiency may be a strong obstable to overcome. anyone tested on this work give some advice about its efficiency? is it possible to run on mobile devices? thank you very much.

    opened by dragen1860 0
Owner
Facebook Research
Facebook Research
GitHub https://robust-cvd.github.io/
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Junjie Hu 13 Dec 10, 2022
The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

null 32 Jun 14, 2022
Code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in Video".

Consistent Depth of Moving Objects in Video This repository contains training code for the SIGGRAPH 2021 paper "Consistent Depth of Moving Objects in

Google 203 Jan 5, 2023
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

Minsoo Song 205 Dec 30, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 138 Dec 28, 2022
Official implementation of the network presented in the paper "M4Depth: A motion-based approach for monocular depth estimation on video sequences"

M4Depth This is the reference TensorFlow implementation for training and testing depth estimation models using the method described in M4Depth: A moti

Michaël Fonder 76 Jan 3, 2023
Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021) Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma. We address the pr

Kranti Kumar Parida 33 Jun 27, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

ONNX msg_chn_wacv20 depth completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20 model in

Ibai Gorordo 19 Oct 22, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

Ibai Gorordo 2 Oct 4, 2021
Data-depth-inference - Data depth inference with python

Welcome! This readme will guide you through the use of the code in this reposito

Marco 3 Feb 8, 2022
Re-implementation of the Noise Contrastive Estimation algorithm for pyTorch, following "Noise-contrastive estimation: A new estimation principle for unnormalized statistical models." (Gutmann and Hyvarinen, AISTATS 2010)

Noise Contrastive Estimation for pyTorch Overview This repository contains a re-implementation of the Noise Contrastive Estimation algorithm, implemen

Denis Emelin 42 Nov 24, 2022
Official Pytorch implementation of "Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video", CVPR 2021

TCMR: Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video Qualtitative result Paper teaser video Introduction This r

Hongsuk Choi 215 Jan 6, 2023
Layered Neural Atlases for Consistent Video Editing

Layered Neural Atlases for Consistent Video Editing Project Page | Paper This repository contains an implementation for the SIGGRAPH Asia 2021 paper L

Yoni Kasten 353 Dec 27, 2022
Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

CorDA Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation Prerequisite Please create and activate the follo

Qin Wang 60 Nov 30, 2022
Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

Computational Photography Lab @ SFU 1.1k Jan 2, 2023
[CVPR 2021] Monocular depth estimation using wavelets for efficiency

Single Image Depth Prediction with Wavelet Decomposition Michaël Ramamonjisoa, Michael Firman, Jamie Watson, Vincent Lepetit and Daniyar Turmukhambeto

Niantic Labs 205 Jan 2, 2023
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 70 Dec 22, 2022
ESTDepth: Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks (CVPR 2021)

ESTDepth: Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks (CVPR 2021) Project Page | Video | Paper | Data We present a novel metho

null 65 Nov 28, 2022
Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation

UniFuse (RAL+ICRA2021) Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation, arXiv, Demo Preparation I

Alibaba 47 Dec 26, 2022