Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation

Related tags

Deep Learning UniFuse-Unidirectional-Fusion
Overview

UniFuse (RAL+ICRA2021)

Office source code of paper UniFuse: Unidirectional Fusion for 360$^\circ$ Panorama Depth Estimation, arXiv, Demo

Preparation

Installation

Environments

  • python 3.6
  • Pytorch >= 1.0.0
  • CUDA >= 9.0

Install requirements

pip install -r requirements.txt

Datasets

Please download the preferred datasets, i.e., Matterport3D, Stanford2D3D, 3D60 and PanoSUNCG. For Matterport3D, please preprocess it following M3D-README.md.

Training

UniFuse on Matterport3D

python train.py --data_path $DATA_PATH \
-dataset matterport3d \
--model_name Matterport3D_UniFuse \
--batch_size 6 \
--num_epochs 100 \
--height 512 \
--width 1024 \
--imagenet_pretrained \
--net UniFuse

Equirectangular baseline on Matterport3D

python train.py --data_path $DATA_PATH \
-dataset matterport3d \
--model_name Matterport3D_Equi \
--batch_size 6 \
--num_epochs 100 \
--height 512 \
--width 1024 \
--imagenet_pretrained \
--net Equi

It is similar for other datasets.

Evaluation

Pre-trained models

The pre-trained models of UniFuse for 4 datasets are available, Matterport3D, Stanford2D3D, 3D60 and PanoSUNCG.

Test on a pre-trained model

python evaluate.py  --data_path $DATA_PATH --dataset matterport3d --load_weights_folder $MODEL_PATH

Citation

Please cite our paper if you find our work useful in your research.

@article{jiang2021unifuse,
      title={UniFuse: Unidirectional Fusion for 360$^{\circ}$ Panorama Depth Estimation}, 
      author={Hualie Jiang and Zhe Sheng and Siyu Zhu and Zilong Dong and Rui Huang},
	  journal={IEEE Robotics and Automation Letters},
	  year={2021},
	  publisher={IEEE}
}
Comments
  • Is there anything about Matterport Preprocessing

    Is there anything about Matterport Preprocessing

    Download PanoBasic

    Copy stitching_Matterport3D.m to PanoBasic

    Modify the directories in stitching_Matterport3D.m, i.e., changing source_dir to the folder of Matterport3D and changing target_dir as the output folder of panorama images and depth maps.

    Execute stitching_Matterport3D.m using Matlab.

    It seems that stiching_Matterport3D uses stich.m, while stich.m is missing in PanoBasic.

    @robertsheng

    opened by chenerg 6
  • A question about metrics.py

    A question about metrics.py

    Thank you very much for your work, but there is a piece of code that makes me a little confused. The location is Line18-Line21 in metrics.py I understand that it means that gt and pred are limited to the range of 0.1-10 before calculating the evaluation metrics. However, the logic of these four lines only limits gt to 0.1 to 10, while pred will have an area greater than 0.1. I hope you can remove my confusion.

    opened by huangyuan2020 2
  • About 'cube_inputs' for unifuse.py.

    About 'cube_inputs' for unifuse.py.

    [Question] cube_inputs = torch.cat(torch.split(input_cube_image, self.cube_h, dim=-1), dim=0) in unifuse.py. this code seemingly cat six cube face on dim0. i.e., (batch_size, 3, h, 6h) --> (batch_size6, 3, h, h). I think that the cube encoder can not ultize feature of different minibatch. Intuitively, catting these face on dim of channel are responable. But, it can't use the imagenet-pretrained network. Did I understand correctly?

    opened by LZleejean 2
  • How to deal with Matterport3D images

    How to deal with Matterport3D images

    Hi,

    Can you please tell us how to handle Matterport3D images? I mean, once you downloded the dataset from the official website, how did you create the equirectangolar RGB images and depth maps?

    Thanks in advance, Valerio

    opened by valeriomieuli 2
  • The problem about testing results using the 3D60 dataset

    The problem about testing results using the 3D60 dataset

    Hello, I encounter a problem similar to zcq15 about 3D60 dataset test. I have tryed to retrained the network on 3D60, but the results seemed to poor compared with the results in paper. I just used your codes directly and didn't do any data preprocessing on 3D60. The results with the given parameters are my I have also tested the 3D60 dataset with the given model parameters, but it seem to worse than the results in paper too. the results are below pre Is there any additional operations on 3D60 dataset just like Stanford2D3D? Thanks for your help. Looking forward for the reply!

    opened by sone1208 1
  • evaluate.py

    evaluate.py

    Hi nice work. Coud u tell me pls. Whats wrong with this, after run python evaluate.py I have error.

    python evaluate.py --data_path /Users/kgkozlov/Downloads/img_4.png --dataset matterport3d --load_weights_dir /Users/UniFuse-Unidirectional-Fusion/PretrainedModels/

    File "/UniFuse-Unidirectional-Fusion/UniFuse/datasets/matterport3d.py", line 83, in __getitem__
    rgb = cv2.cvtColor(rgb, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(4.5.3) /private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/pip-req-build-z9mn802i/opencv/modules/imgproc/src/color.cpp:182: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

    fpath = os.path.join(os.path.dirname(__file__), "datasets", "{}_{}.txt")

    But you add txt file with RGB and DEPTH, But shouldn't we just add one RGB image ?

    opened by KirillHiddleston 2
Owner
Alibaba
Alibaba Open Source
Alibaba
GitHub
Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal, multi-exposure and multi-focus image fusion.

U2Fusion Code of U2Fusion: a unified unsupervised image fusion network for multiple image fusion tasks, including multi-modal (VIS-IR, medical), multi

Han Xu 129 Dec 11, 2022
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

Pranav 39 Nov 21, 2022
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Junjie Hu 13 Dec 10, 2022
This repo is for Self-Supervised Monocular Depth Estimation with Internal Feature Fusion(arXiv), BMVC2021

DIFFNet This repo is for Self-Supervised Monocular Depth Estimation with Internal Feature Fusion(arXiv), BMVC2021 A new backbone for self-supervised d

Hang 3 Oct 22, 2021
Indoor Panorama Planar 3D Reconstruction via Divide and Conquer

HV-plane reconstruction from a single 360 image Code for our paper in CVPR 2021: Indoor Panorama Planar 3D Reconstruction via Divide and Conquer (pape

sunset 36 Jan 3, 2023
Image-Stitching - Panorama composition using SIFT Features and a custom implementaion of RANSAC algorithm

About The Project Panorama composition using SIFT Features and a custom implementaion of RANSAC algorithm (Random Sample Consensus). Author: Andreas P

Andreas Panayiotou 3 Jan 3, 2023
The implemention of Video Depth Estimation by Fusing Flow-to-Depth Proposals

Flow-to-depth (FDNet) video-depth-estimation This is the implementation of paper Video Depth Estimation by Fusing Flow-to-Depth Proposals Jiaxin Xie,

null 32 Jun 14, 2022
Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals

LapDepth-release This repository is a Pytorch implementation of the paper "Monocular Depth Estimation Using Laplacian Pyramid-Based Depth Residuals" M

Minsoo Song 205 Dec 30, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 138 Dec 28, 2022
Source code for "OmniPhotos: Casual 360° VR Photography"

OmniPhotos: Casual 360° VR Photography Project Page | Video | Paper | Demo | Data This repository contains the source code for creating and viewing Om

Christian Richardt 144 Dec 30, 2022
TorchDistiller - a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

This project is a collection of the open source pytorch code for knowledge distillation, especially for the perception tasks, including semantic segmentation, depth estimation, object detection and instance segmentation.

yifan liu 147 Dec 3, 2022
Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments Paper: arXiv (ICRA 2021) Video : https://youtu.be/CC

Sachini Herath 68 Jan 3, 2023
Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral)

DSA^2 F: Deep RGB-D Saliency Detection with Depth-Sensitive Attention and Automatic Multi-Modal Fusion (CVPR'2021, Oral) This repo is the official imp

如今我已剑指天涯 46 Dec 21, 2022
Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation

CorDA Code for our paper Domain Adaptive Semantic Segmentation with Self-Supervised Depth Estimation Prerequisite Please create and activate the follo

Qin Wang 60 Nov 30, 2022
the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

G2S This is the official code for ICRA 2021 Paper: Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation by Hemang

NeurAI 4 Jul 27, 2022
Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021)

Beyond Image to Depth: Improving Depth Prediction using Echoes (CVPR 2021) Kranti Kumar Parida, Siddharth Srivastava, Gaurav Sharma. We address the pr

Kranti Kumar Parida 33 Jun 27, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in ONNX

ONNX msg_chn_wacv20 depth completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20 model in

Ibai Gorordo 19 Oct 22, 2022
Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model in Tensorflow Lite.

TFLite-msg_chn_wacv20-depth-completion Python script for performing depth completion from sparse depth and rgb images using the msg_chn_wacv20. model

Ibai Gorordo 2 Oct 4, 2021
Data-depth-inference - Data depth inference with python

Welcome! This readme will guide you through the use of the code in this reposito

Marco 3 Feb 8, 2022