Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

Last update: Nov 21, 2022

Overview

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute depth estimation.

Quick Start

First, download the pretrained models using the download_models script.

Next, run the infer script - this will run on all images in test/input and save the results to test/output.

python3 -m pip install -r requirements.txt
python3 -m merged_depth.utils.download_models
python3 -m merged_depth.infer

The results include (1) a _depth.npy file that you can load (see load_and_display_depth.py), (2) a _stacked.png file that shows the original and colorized depth images.

To run the predictor on a single input, use infer_single.py

python3 -m merged_depth.infer_single ~/foo/bar/test.png

Sample Output

Comments

Can't download models

When I try python3 -m merged_depth.utils.download_models, following errors occurred

Traceback (most recent call last):
  File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/me/merged_depth/merged_depth/utils/download_models.py", line 65, in <module>
    main()
  File "/home/me/merged_depth/merged_depth/utils/download_models.py", line 62, in main
    extract_models(pretrained_models)
  File "/home/me/merged_depth/merged_depth/utils/download_models.py", line 56, in extract_models
    with zipfile.ZipFile(file, 'r') as zip_ref:
  File "/usr/lib/python3.8/zipfile.py", line 1269, in __init__
    self._RealGetContents()
  File "/usr/lib/python3.8/zipfile.py", line 1336, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

The content of pretrained/pretrained.zip is

<HTML>
<HEAD>
<TITLE>Not Found</TITLE>
</HEAD>
<BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Not Found</H1>
<H2>Error 404</H2>
</BODY>
</HTML>

How can I get the model? Thanks!

opened by eppppi 7

Relative depth to absolute

In one of your response, related relative to absolute depth conversion, you mentioned : " I'm using some linear parameters to scale these to absolute depth". Could you please point to the particular code block where this transformation implemented. Thank you.

opened by semel1 4

Error downloading files.

merged_depth >python utils\download_models.py
Traceback (most recent call last):
  File "utils\download_models.py", line 65, in <module>
    main()
  File "utils\download_models.py", line 62, in main
    extract_models(pretrained_models)
  File "utils\download_models.py", line 56, in extract_models
    with zipfile.ZipFile(file, 'r') as zip_ref:
  File "c:\g\vr\lw\python\lib\zipfile.py", line 1258, in __init__
    self._RealGetContents()
  File "c:\g\vr\lw\python\lib\zipfile.py", line 1325, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

opened by apiszcz 4

clarification about output

Hii @p-ranav thank so much for sharing your amazing work i wanted to know what output image represents, is it representing absolute depth or anything proportional to absolute depth? what are the values in average_depth in infer.py represent? if i want to get the absolute depth how can i get that

Thank you

opened by nithinvenny07 1

Update facebookresearch/WSL-Images torchhub repository

This change resolves a 404 error from facebookresearch WSL-Images

Downloading: "https://github.com/facebookresearch/WSL-Images/archive/master.zip" to /home/mike/.cache/torch/hub/master.zip
Traceback (most recent call last):
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/infer_single.py", line 36, in <module>
    main(args.path)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/infer_single.py", line 13, in main
    engine = InferenceEngine()
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/infer.py", line 83, in __init__
    self.midas_model = MidasNet(self.midas_model_path, non_negative=True)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/nets/MiDaS/midas/midas_net.py", line 30, in __init__
    self.pretrained, self.scratch = _make_encoder(backbone="resnext101_wsl", features=features, use_pretrained=use_pretrained)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/nets/MiDaS/midas/blocks.py", line 7, in _make_encoder
    pretrained = _make_pretrained_resnext101_wsl(use_pretrained)
  File "/home/mike/Workspaces/enli/merged_depth/merged_depth/nets/MiDaS/midas/blocks.py", line 85, in _make_pretrained_resnext101_wsl
    resnet = torch.hub.load("facebookresearch/WSL-Images", "resnext101_32x8d_wsl")
  File "/home/mike/.pyenv/versions/merged_depth/lib/python3.8/site-packages/torch/hub.py", line 337, in load
    repo_or_dir = _get_cache_or_reload(repo_or_dir, force_reload, verbose)
  File "/home/mike/.pyenv/versions/merged_depth/lib/python3.8/site-packages/torch/hub.py", line 144, in _get_cache_or_reload
    download_url_to_file(url, cached_file, progress=False)
  File "/home/mike/.pyenv/versions/merged_depth/lib/python3.8/site-packages/torch/hub.py", line 394, in download_url_to_file
    u = urlopen(req)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 563, in error
    result = self._call_chain(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 755, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 640, in http_response
    response = self.parent.error(
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 502, in _call_chain
    result = func(*args)
  File "/home/mike/.pyenv/versions/3.8.11/lib/python3.8/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found

opened by mmurray 0

Upgrade to Midas V.3

It looks like you use Midas v.2.1. Any chance to upgrade to Midas v.3.0 with dpt_large-midas-2f21e586.pt model ? I replaced the "core" Midas to Midas V.3 in the ...\merged_depth\nets and everything works just fine. However attempt to change the model self.midas_model_path = "./pretrained/MiDaS_f6b98070.pt" to self.midas_model_path = "./pretrained/dpt_large-midas-2f21e586.pt" throws multiple errors upon execution. My great appreciation for your suggestion.

opened by semel1 0
GPU utilization

Running on GTX 1060 6Gb. GPU utilization ` 0 -1%. Processed by CPU only that terribly slow. "Standalone" Midas3 and monodepth2 utilize GPU just perfectly. Solved by downgrading to torch from 1.9 to 1.8. This issue could be closed

opened by semel1 0
SGDepth Predictions Inverted

It seems to me that the predicted values from SGDepth are inverted compared to other predictions.

(Also thank you for having this repository public! Really nice for comparing different depth networks!)

opened by richardmarcus 0

Issue with installation - bad zip file

Error received:

Traceback (most recent call last):
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/ashok/Desktop/wsnr/merged_depth/merged_depth/utils/download_models.py", line 65, in <module>
    main()
  File "/Users/ashok/Desktop/wsnr/merged_depth/merged_depth/utils/download_models.py", line 62, in main
    extract_models(pretrained_models)
  File "/Users/ashok/Desktop/wsnr/merged_depth/merged_depth/utils/download_models.py", line 56, in extract_models
    with zipfile.ZipFile(file, 'r') as zip_ref:
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/zipfile.py", line 1266, in __init__
    self._RealGetContents()
  File "/Users/ashok/opt/anaconda3/envs/wsn/lib/python3.9/zipfile.py", line 1333, in _RealGetContents
    raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file

opened by ashok-arora 0

Owner

Pranav

GitHub

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging

Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging This repository contains an implementation

1.1k Jan 2, 2023

A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

1.1k Dec 24, 2022

Code for Transformers Solve Limited Receptive Field for Monocular Depth Prediction

Official PyTorch code for Transformers Solve Limited Receptive Field for Monocular Depth Prediction. Guanglei Yang, Hao Tang, Mingli Ding, Nicu Sebe,

152 Dec 16, 2022

AdelaiDepth is an open source toolbox for monocular depth prediction.

Adelaide Intelligent Machines (AIM) Group

743 Jan 1, 2023

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction This is the code for the paper Combining E

69 Dec 26, 2022

We envision models that are pre-trained on a vast range of domain-relevant tasks to become key for molecule property prediction

We envision models that are pre-trained on a vast range of domain-relevant tasks to become key for molecule property prediction. This repository aims to give easy access to state-of-the-art pre-trained models.

90 Jan 8, 2023

Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts

t5-japanese Codes to pre-train T5 (Text-to-Text Transfer Transformer) models pre-trained on Japanese web texts. The following is a list of models that

1 Dec 13, 2021

Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

13 Dec 10, 2022

OHLC Average Prediction of Apple Inc. Using LSTM Recurrent Neural Network

Stock Price Prediction of Apple Inc. Using Recurrent Neural Network OHLC Average Prediction of Apple Inc. Using LSTM Recurrent Neural Network Dataset:

410 Jan 5, 2023

Official implementation of the network presented in the paper "M4Depth: A motion-based approach for monocular depth estimation on video sequences"

M4Depth This is the reference TensorFlow implementation for training and testing depth estimation models using the method described in M4Depth: A moti

76 Jan 3, 2023

[CVPR 2021] Monocular depth estimation using wavelets for efficiency

Single Image Depth Prediction with Wavelet Decomposition Michaël Ramamonjisoa, Michael Firman, Jamie Watson, Vincent Lepetit and Daniyar Turmukhambeto

205 Jan 2, 2023

[ICCV 2021] Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation

EPCDepth EPCDepth is a self-supervised monocular depth estimation model, whose supervision is coming from the other image in a stereo pair. Details ar

110 Dec 23, 2022

This repo is for Self-Supervised Monocular Depth Estimation with Internal Feature Fusion(arXiv), BMVC2021

DIFFNet This repo is for Self-Supervised Monocular Depth Estimation with Internal Feature Fusion(arXiv), BMVC2021 A new backbone for self-supervised d

3 Oct 22, 2021

the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

G2S This is the official code for ICRA 2021 Paper: Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation by Hemang

4 Jul 27, 2022

Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

Related tags