Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

Related tags

Deep Learning open-source machine-learning youtube deep-learning camera images academic image-processing prediction torch python3 pytorch neural-networks mit-license average depth-estimation depth-prediction torchvision average-calculator
Overview

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute depth estimation.

Quick Start

First, download the pretrained models using the download_models script.

Next, run the infer script - this will run on all images in test/input and save the results to test/output.

python3 -m pip install -r requirements.txt
python3 -m merged_depth.utils.download_models
python3 -m merged_depth.infer

The results include (1) a _depth.npy file that you can load (see load_and_display_depth.py), (2) a _stacked.png file that shows the original and colorized depth images.

To run the predictor on a single input, use infer_single.py

python3 -m merged_depth.infer_single ~/foo/bar/test.png

Sample Output
Comments
  • Can't download models

    Can't download models

    When I try python3 -m merged_depth.utils.download_models, following errors occurred

    Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/me/merged_depth/merged_depth/utils/download_models.py", line 65, in <module>
    main()
  File "/home/me/merged_depth/merged_depth/utils/download_models.py", line 62, in main
    extract_models(pretrained_models)
  File "/home/me/merged_depth/merged_depth/utils/download_models.py", line 56, in extract_models
    with zipfile.ZipFile(file, 'r') as zip_ref:
  File "/usr/lib/python3.8/zipfile.py", line 1269, in __init__
    self._RealGetContents()
  File "/usr/lib/python3.8/zipfile.py", line 1336, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

    The content of pretrained/pretrained.zip is

    <HTML>
<HEAD>
<TITLE>Not Found</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Not Found</H1>
<H2>Error 404</H2>
</BODY>
</HTML>

    How can I get the model? Thanks!

    opened by eppppi 7
  • Relative depth to absolute

    Relative depth to absolute

    In one of your response, related relative to absolute depth conversion, you mentioned : " I'm using some linear parameters to scale these to absolute depth". Could you please point to the particular code block where this transformation implemented. Thank you.

    opened by semel1 4
  • Error downloading files.

    Error downloading files. 

    merged_depth >python utils\download_models.py
Traceback (most recent call last):
  File "utils\download_models.py", line 65, in <module>
    main()
  File "utils\download_models.py", line 62, in main
    extract_models(pretrained_models)
  File "utils\download_models.py", line 56, in extract_models
    with zipfile.ZipFile(file, 'r') as zip_ref:
  File "c:\g\vr\lw\python\lib\zipfile.py", line 1258, in __init__
    self._RealGetContents()
  File "c:\g\vr\lw\python\lib\zipfile.py", line 1325, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
    opened by apiszcz 4
  • clarification about output

    clarification about output

    Hii @p-ranav thank so much for sharing your amazing work i wanted to know what output image represents, is it representing absolute depth or anything proportional to absolute depth? what are the values in average_depth in infer.py represent? if i want to get the absolute depth how can i get that

    Thank you

    opened by nithinvenny07 1
  • Update facebookresearch/WSL-Images torchhub repository

    Update facebookresearch/WSL-Images torchhub repository

    This change resolves a 404 error from facebookresearch WSL-Images

    Downloading: "https://github.com/facebookresearch/WSL-Images/archive/master.zip" to /home/mike/.cache/torch/hub/master.zip
Traceback (most recent call last):
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/infer_single.py", line 36, in <module>
    main(args.path)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/infer_single.py", line 13, in main
    engine = InferenceEngine()
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/infer.py", line 83, in __init__
    self.midas_model = MidasNet(self.midas_model_path, non_negative=True)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/nets/MiDaS/midas/midas_net.py", line 30, in __init__
    self.pretrained, self.scratch = _make_encoder(backbone="resnext101_wsl", features=features, use_pretrained=use_pretrained)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/nets/MiDaS/midas/blocks.py", line 7, in _make_encoder
    pretrained = _make_pretrained_resnext101_wsl(use_pretrained)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/nets/MiDaS/midas/blocks.py", line 85, in _make_pretrained_resnext101_wsl
    resnet = torch.hub.load("facebookresearch/WSL-Images", "resnext101_32x8d_wsl")
  File "/home/mike/.pyenv/versions/merged_depth/lib/python3.8/site-packages/torch/hub.py", line 337, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "/home/mike/.pyenv/versions/merged_depth/lib/python3.8/site-packages/torch/hub.py", line 144, in _get_cache_or_reload
    download_url_to_file(url, cached_file, progress=False)
  File "/home/mike/.pyenv/versions/merged_depth/lib/python3.8/site-packages/torch/hub.py", line 394, in download_url_to_file
    u = urlopen(req)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
    opened by mmurray 0
  • Upgrade to Midas V.3

    Upgrade to Midas V.3

    It looks like you use Midas v.2.1. Any chance to upgrade to Midas v.3.0 with dpt_large-midas-2f21e586.pt model ? I replaced the "core" Midas to Midas V.3 in the ...\merged_depth\nets and everything works just fine. However attempt to change the model self.midas_model_path = "./pretrained/MiDaS_f6b98070.pt" to self.midas_model_path = "./pretrained/dpt_large-midas-2f21e586.pt" throws multiple errors upon execution. My great appreciation for your suggestion.

    opened by semel1 0
  • GPU utilization

    GPU utilization

    Running on GTX 1060 6Gb. GPU utilization ` 0 -1%. Processed by CPU only that terribly slow. "Standalone" Midas3 and monodepth2 utilize GPU just perfectly. Solved by downgrading to torch from 1.9 to 1.8. This issue could be closed

    opened by semel1 0
  • SGDepth Predictions Inverted

    SGDepth Predictions Inverted

    It seems to me that the predicted values from SGDepth are inverted compared to other predictions.

    (Also thank you for having this repository public! Really nice for comparing different depth networks!)

    opened by richardmarcus 0
  • Issue with installation - bad zip file

    Issue with installation - bad zip file

    Error received:

    Traceback (most recent call last):
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/ashok/Desktop/wsnr/merged_depth/merged_depth/utils/download_models.py", line 65, in <module>
    main()
  File "/Users/ashok/Desktop/wsnr/merged_depth/merged_depth/utils/download_models.py", line 62, in main
    extract_models(pretrained_models)
  File "/Users/ashok/Desktop/wsnr/merged_depth/merged_depth/utils/download_models.py", line 56, in extract_models
    with zipfile.ZipFile(file, 'r') as zip_ref:
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/zipfile.py", line 1266, in __init__
    self._RealGetContents()
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/zipfile.py", line 1333, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
    opened by ashok-arora 0
Owner
Pranav
Pranav
GitHub
Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

Computational Photography Lab @ SFU 1.1k Jan 2, 2023
A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

Justin 1.1k Dec 24, 2022
Code for Transformers Solve Limited Receptive Field for Monocular Depth Prediction

Official PyTorch code for Transformers Solve Limited Receptive Field for Monocular Depth Prediction. Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe,

stanley 152 Dec 16, 2022
AdelaiDepth is an open source toolbox for monocular depth prediction.

AdelaiDepth is an open source toolbox for monocular depth prediction.

Adelaide Intelligent Machines (AIM) Group 743 Jan 1, 2023
Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction This is the code for the paper Combining E

Robotics and Perception Group 69 Dec 26, 2022
We envision models that are pre-trained on a vast range of domain-relevant tasks to become key for molecule property prediction

We envision models that are pre-trained on a vast range of domain-relevant tasks to become key for molecule property prediction. This repository aims to give easy access to state-of-the-art pre-trained models.

GMUM 90 Jan 8, 2023
Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts

t5-japanese Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts. The following is a list of models that

Kimio Kuramitsu 1 Dec 13, 2021
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Junjie Hu 13 Dec 10, 2022
OHLC Average Prediction of Apple Inc. Using LSTM Recurrent Neural Network

Stock Price Prediction of Apple Inc. Using Recurrent Neural Network OHLC Average Prediction of Apple Inc. Using LSTM Recurrent Neural Network Dataset:

Nouroz Rahman 410 Jan 5, 2023
Official implementation of the network presented in the paper "M4Depth: A motion-based approach for monocular depth estimation on video sequences"

M4Depth This is the reference TensorFlow implementation for training and testing depth estimation models using the method described in M4Depth: A moti

Michaël Fonder 76 Jan 3, 2023
[CVPR 2021] Monocular depth estimation using wavelets for efficiency

Single Image Depth Prediction with Wavelet Decomposition Michaël Ramamonjisoa, Michael Firman, Jamie Watson, Vincent Lepetit and Daniyar Turmukhambeto

Niantic Labs 205 Jan 2, 2023
[ICCV 2021] Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation

EPCDepth EPCDepth is a self-supervised monocular depth estimation model, whose supervision is coming from the other image in a stereo pair. Details ar

Rui Peng 110 Dec 23, 2022
This repo is for Self-Supervised Monocular Depth Estimation with Internal Feature Fusion(arXiv), BMVC2021

DIFFNet This repo is for Self-Supervised Monocular Depth Estimation with Internal Feature Fusion(arXiv), BMVC2021 A new backbone for self-supervised d

Hang 3 Oct 22, 2021
the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

G2S This is the official code for ICRA 2021 Paper: Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation by Hemang

NeurAI 4 Jul 27, 2022
SimpleDepthEstimation - An unified codebase for NN-based monocular depth estimation methods

SimpleDepthEstimation Introduction This is an unified codebase for NN-based monocular depth estimation methods, the framework is based on detectron2 (

null 8 Dec 13, 2022
ONNX-GLPDepth - Python scripts for performing monocular depth estimation using the GLPDepth model in ONNX

ONNX-GLPDepth - Python scripts for performing monocular depth estimation using the GLPDepth model in ONNX

Ibai Gorordo 18 Nov 6, 2022
ONNX-PackNet-SfM: Python scripts for performing monocular depth estimation using the PackNet-SfM model in ONNX

Python scripts for performing monocular depth estimation using the PackNet-SfM model in ONNX

Ibai Gorordo 14 Dec 9, 2022
Repository for "Toward Practical Monocular Indoor Depth Estimation" (CVPR 2022)

Toward Practical Monocular Indoor Depth Estimation Cho-Ying Wu, Jialiang Wang, Michael Hall, Ulrich Neumann, Shuochen Su [arXiv] [project site] DistDe

Meta Research 122 Dec 13, 2022
Multiple-criteria decision-making (MCDM) with Electre, Promethee, Weighted Sum and Pareto

EasyMCDM - Quick Installation methods Install with PyPI Once you have created your Python environment (Python 3.6+) you can simply type: pip3 install

Labrak Yanis 6 Nov 22, 2022