Current state of supervised and unsupervised depth completion methods

Overview

Awesome Depth Completion

Table of Contents

About Sparse-to-Dense Depth Completion

In the sparse-to-dense depth completion problem, one wants to infer the dense depth map of a 3-D scene given an RGB image and its corresponding sparse reconstruction in the form of a sparse depth map obtained either from computational methods such as SfM (Strcuture-from-Motion) or active sensors such as lidar or structured light sensors.

Example 1: VOID dataset (indoor VIO)

Input RGB image Sparse point cloud Output point cloud from KBNet

Example 2: KITTI dataset (outdoor lidar)

Input RGB image Output point cloud from ScaffNet

Current State of Depth Completion Methods

Here we compile both unsupervised/self-supervised (monocular and stereo) and supervised methods published in recent conferences and journals on the VOID (Wong et. al., 2020) and KITTI (Uhrig et. al., 2017) depth completion benchmarks. Our ranking considers all four metrics rather than just RMSE.

Quick Links

Unsupervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 39.80 95.86 21.16 49.72
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 60.68 122.01 35.24 67.34
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 85.05 169.79 48.92 104.02
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 151.86 222.36 74.59 112.36
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 178.85 243.84 80.12 107.69

Supervised VOID Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 59.40 181.42 19.37 46.56

Unsupervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Unsupervised Depth Completion with Calibrated Backprojection Layers ICCV 2021 PyTorch 256.76 1069.47 1.02 2.95
Learning Topology from Synthetic Data for Unsupervised Depth Completion RA-L & ICRA 2021 Tensorflow 280.76 1121.93 1.15 3.30
Project to Adapt: Domain Adaptation for Depth Completion from Noisy and Sparse Sensor Data ACCV 2020 PyTorch 280.42 1095.26 1.19 3.53
Unsupervised Depth Completion from Visual Inertial Odometry RA-L & ICRA 2020 Tensorflow 299.41 1169.97 1.20 3.56
A Surface Geometry Model for LiDAR Depth Completion RA-L & ICRA 2021 Tensorflow 298.3 1239.84 1.21 3.76
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 343.46 1263.19 1.32 3.58
DFuseNet: Deep Fusion of RGB and Sparse Depth Information for Image Guided Dense Depth Completion ITSC 2019 PyTorch 429.93 1206.66 1.79 3.62
In Defense of Classical Image Processing: Fast Depth Completion on the CPU CRV 2018 Python 302.60 1288.46 1.29 3.78
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 350.32 1299.85 1.57 4.07
Semantically Guided Depth Upsampling GCPR 2016 N/A 605.47 2312.57 2.05 7.38

Supervised KITTI Depth Completion Benchmark

Paper Publication Code MAE RMSE iMAE iRMSE
Non-Local Spatial Propagation Network for Depth Completion ECCV 2020 PyTorch 199.5 741.68 0.84 1.99
CSPN++: Learning Context and Resource Aware Convolutional Spatial Propagation Networks for Depth Completion AAAI 2020 N/A 209.28 743.69 0.90 2.07
Dense depth posterior (ddp) from single image and sparse range CVPR 2019 Tensorflow 203.96 832.94 0.85 2.10
Adaptive context-aware multi-modal network for depth completion TIP 2021 PyTorch 206.80 732.99 0.90 2.08
PENet: Towards Precise and Efficient Image Guided Depth Completion ICRA 2021 PyTorch 210.55 730.08 0.94 2.17
FCFR-Net: Feature Fusion based Coarse- to-Fine Residual Learning for Depth Completion AAAI 2021 N/A 217.15 735.81 0.98 2.20
Learning Guided Convolutional Network for Depth Completion TIP 2020 PyTorch 218.83 736.24 0.99 2.25
DenseLiDAR: A Real-Time Pseudo Dense Depth Guided Depth Completion Network ICRA 2021 N/A 214.13 755.41 0.96 2.25
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 PyTorch 220.41 762.19 0.98 2.30
Sparse and noisy LiDAR completion with RGB guidance and uncertainty MVA 2019 PyTorch 215.02 772.87 0.93 2.19
A Multi-Scale Guided Cascade Hourglass Network for Depth Completion WACV 2020 N/A 220.41 762.19 0.98 2.30
Learning Joint 2D-3D Representations for Depth Completion ICCV 2019 N/A 221.19 752.88 1.14 2.34
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image CVPR 2019 PyTorch 226.50 758.38 1.15 2.56
Depth Completion from Sparse LiDAR Data with Depth-Normal Constraints ICCV 2019 N/A 235.17 777.05 1.13 2.42
Scanline Resolution-Invariant Depth Completion Using a Single Image and Sparse LiDAR Point Cloud RA-L & IROS 2021 N/A 233.34 809.09 1.06 2.57
Confidence propagation through cnns for guided sparse depth regression PAMI 2019 PyTorch 233.26 829.98 1.03 2.60
Self-supervised Sparse-to-Dense: Self- supervised Depth Completion from LiDAR and Monocular Camera ICRA 2019 PyTorch 249.95 814.73 1.21 2.80
Uncertainty-Aware CNNs for Depth Completion: Uncertainty from Beginning to End CVPR 2020 PyTorch 251.77 960.05 1.05 3.37
Sparse and Dense Data with CNNs: Depth Completion and Semantic Segmentation 3DV 2019 N/A 234.81 917.64 0.95 2.17
Depth coefficients for depth completion CVPR 2019 N/A 252.21 988.38 1.13 2.87
Depth estimation via affinity learned with convolutional spatial propagation network ECCV 2018 N/A 279.46 1019.64 1.15 2.93
Learning morphological operators for depth completion ACIVS 2019 N/A 310.49 1045.45 1.57 3.84
Sparsity Invariant CNNs 3DV 2017 Tensorflow 416.14 1419.75 1.29 3.25
Deep Convolutional Compressed Sensing for LiDAR Depth Completion ACCV 2018 Tensorflow 439.48 1325.37 3.19 59.39
You might also like...
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Data-depth-inference - Data depth inference with python
Data-depth-inference - Data depth inference with python

Welcome! This readme will guide you through the use of the code in this reposito

(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

[CVPR 2021] Unsupervised 3D Shape Completion through GAN Inversion
[CVPR 2021] Unsupervised 3D Shape Completion through GAN Inversion

ShapeInversion Paper Junzhe Zhang, Xinyi Chen, Zhongang Cai, Liang Pan, Haiyu Zhao, Shuai Yi, Chai Kiat Yeo, Bo Dai, Chen Change Loy "Unsupervised 3D

SimpleDepthEstimation - An unified codebase for NN-based monocular depth estimation methods
SimpleDepthEstimation - An unified codebase for NN-based monocular depth estimation methods

SimpleDepthEstimation Introduction This is an unified codebase for NN-based monocular depth estimation methods, the framework is based on detectron2 (

A PyTorch-based open-source framework that provides methods for improving the weakly annotated data and allows researchers to efficiently develop and compare their own methods.
A PyTorch-based open-source framework that provides methods for improving the weakly annotated data and allows researchers to efficiently develop and compare their own methods.

Knodle (Knowledge-supervised Deep Learning Framework) - a new framework for weak supervision with neural networks. It provides a modularization for se

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning
UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning

UniMoCo: Unsupervised, Semi-Supervised and Full-Supervised Visual Representation Learning This is the official PyTorch implementation for UniMoCo pape

aka
aka "Bayesian Methods for Hackers": An introduction to Bayesian methods + probabilistic programming with a computation/understanding-first, mathematics-second point of view. All in pure Python ;)

Bayesian Methods for Hackers Using Python and PyMC The Bayesian method is the natural approach to inference, yet it is hidden from readers behind chap

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Implementation of temporal pooling methods studied in [ICIP'20] A Comparative Evaluation Of Temporal Pooling Methods For Blind Video Quality Assessment

Owner
I am a post-doctoral researcher at the UCLA Vision Lab under the supervision of Professor Stefano Soatto.
null
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

ONNX msg_chn_wacv20 depth completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20 model in

Ibai Gorordo 19 Oct 22, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

Ibai Gorordo 2 Oct 4, 2021
ICRA 2021 "Towards Precise and Efficient Image Guided Depth Completion"

PENet: Precise and Efficient Depth Completion This repo is the PyTorch implementation of our paper to appear in ICRA2021 on "Towards Precise and Effic

null 232 Dec 25, 2022
Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation (RA-L/ICRA 2020)

Aerial Depth Completion This work is described in the letter "Aerial Single-View Depth Completion with Image-Guided Uncertainty Estimation", by Lucas

ETHZ V4RL 70 Dec 22, 2022
RGB-D Local Implicit Function for Depth Completion of Transparent Objects

RGB-D Local Implicit Function for Depth Completion of Transparent Objects [Project Page] [Paper] Overview This repository maintains the official imple

NVIDIA Research Projects 43 Dec 12, 2022
Official page of Struct-MDC (RA-L'22 with IROS'22 option); Depth completion from Visual-SLAM using point & line features

Struct-MDC (click the above buttons for redirection!) Official page of "Struct-MDC: Mesh-Refined Unsupervised Depth Completion Leveraging Structural R

Urban Robotics Lab. @ KAIST 37 Dec 22, 2022
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

Pranav 39 Nov 21, 2022
The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

null 32 Jun 14, 2022
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

Minsoo Song 205 Dec 30, 2022
Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021) Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma. We address the pr

Kranti Kumar Parida 33 Jun 27, 2022