Python3 / PyTorch implementation of the following paper: Fine-grained Semantics-aware Representation Enhancement for Self-supervisedMonocular Depth Estimation. ICCV 2021 (oral)

Overview

FSRE-Depth

This is a Python3 / PyTorch implementation of FSRE-Depth, as described in the following paper:

Fine-grained Semantics-aware Representation Enhancement for Self-supervisedMonocular Depth Estimation overview Hyunyoung Jung, Eunhyeok Park and Sungjoo Yoo

ICCV 2021 (oral)

arXiv pdf

The code was implemented based on Monodepth2.

Setup

This code was implemented under torch==1.3.0 and torchvision==0.4.1, using two NVIDIA TITAN Xp gpus with distrutibted training. Different version may produce different results.

pip install -r requirements.txt

Dataset

KITTI Raw Data and pre-computed segmentation images are required for training.

KITTI/
    ├── 2011_09_26/             
    ├── 2011_09_28/                    
    ├── 2011_09_29/
    ├── 2011_09_30/
    ├── 2011_10_03/
    └── segmentation/   # download and unzip "segmentation.zip" 

Training

For training the full model, run the command as below:

CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.launch --nproc_per_node 2 --master_port YOUR_PORT_NUMBER train_ddp.py --dapa_path YOUR_KITTI_DATA_PATH

Evaluation

Download Models

will be uploaded soon

Reference

Please use the following citation when referencing our work:

@InProceedings{Jung_2021_ICCV,
    author    = {Jung, Hyunyoung and Park, Eunhyeok and Yoo, Sungjoo},
    title     = {Fine-Grained Semantics-Aware Representation Enhancement for Self-Supervised Monocular Depth Estimation},
    booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    month     = {October},
    year      = {2021},
    pages     = {12642-12652}
}
Comments
  • A question about the paper

    A question about the paper

    Hi,

    Thank you. I have a question on equation 5 and 6 of the paper. In those equations, since both depths of i and j are normalized, they both become 1 or not ?

    opened by shuuchen 6
  • What GT depth is used for evaluation?

    What GT depth is used for evaluation?

    Hi @hyBlue ,

    Thank you for your great work. I just have one question regarding to the GT depth used for evaluation. In your dataloader, you load gt_depth from file. (https://github.com/hyBlue/FSRE-Depth/blob/d746866965204c37a072f3518dfd873df7ed2f1a/datasets/kitti_dataset.py#L84). However, in your evaluation code, gt_depth is loaded from an npz file. (https://github.com/hyBlue/FSRE-Depth/blob/d746866965204c37a072f3518dfd873df7ed2f1a/evaluate_depth.py#L148) And the gt_depth from dataloader is not used?

    Is this true?

    Also, for the gt_depth from npz, this gt_depth is corresponding to the groundtruth set of KITTI Depth Prediction or velodyne_raw set of KITTI Depth Prediction?

    Thank you in advance.

    Best,

    opened by SwagJ 5
  • About Semantic Segmentation Tags

    About Semantic Segmentation Tags

    Hi, the labels I got using NIVDA's segmentation network are pseudo-labels for 19 categories, which are not the same as the pre-computed segmentation you uploaded

    Could you please share your steps and practices?

    Thank you very much!

    opened by joyqoo 4
  • Inference image failed

    Inference image failed

    Hi, I used your architecture to train my own dataset, but the results are not as expected, the evaluation results and images are quite poor.

    Illogical oddities occur when inferring images. I see that you do not have the inference part.

    Could you please give me some advice? Thank you !

    166509721-86023852-608a-45e8-97b0-667fdd45d27e 166509320-e34d449b-b34a-4b7b-984d-aa5c4cf23210
    opened by joyqoo 3
  • error in code (train and test)

    error in code (train and test)

    Hi,

    I try to train and test your model. I found some error in your code.

    1. '--data_path' option -> when I set --data_path for my own dataset path, the path that your code modified was pointed to wrong directory
    2. when I run 'evaluate_depth.py' for tersting, 2-1. '--fix_K' option does not exist in KittiDataset class 2-2. in 111 line, output = models['depth'](features, mode='depth'), return error => forward() got an unexpected keyword argument 'mode'

    I hope to use your official evaluate code / script.

    Thanks!

    opened by jjinCode 2
  • Question About Error Map Generation in the paper

    Question About Error Map Generation in the paper

    Hi @hyBlue,

    What method did you use to handle discoutinuity of GT depth when plotting the error map in your paper? Did you use interpolation? Since GT depth map inevitably has empty pixel, I am just wondering about this. In addition, I think there is a bug in your evaluation code in your current implementation.

    https://github.com/hyBlue/FSRE-Depth/blob/d746866965204c37a072f3518dfd873df7ed2f1a/evaluate_depth.py#L68 the dataset doesn't have input: fix_K.

    So, I am looking forward to your reply on error map part. Thank you in advance.

    Best,

    opened by SwagJ 2
  • UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.

    UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum.

    when i run evaluate_depth /home/shs/miniconda3/envs/shs/lib/python3.7/site-packages/torchvision/transforms/transforms.py:334: UserWarning: Argument interpolation should be of type InterpolationMode instead of int. Please, use InterpolationMode enum. "Argument interpolation should be of type InterpolationMode instead of int. "

    opened by lonelysheeep 1
  • Explanation of the sgementation dataset

    Explanation of the sgementation dataset

    Hi Thank you for a great job. Could you tell me how the sgementation dataset is obtained? Is it used to train semantic segmentation subnetworks ? Thank you very much.

    opened by CUMTchenlv 1
  • Question about Cross-task Similarity

    Question about Cross-task Similarity

    Hi, thanks for your good work. I have some confusion, hope for your answer. when calculating Cross-task Similarity, you use att = (q_out * k_out).sum(dim=2) / np.sqrt(self.out_channel) tensor mul is that (B, Head, C, H, W) * (B, Head, C, H, W) = B, Head, C, H, W I think Cross-task Similarity is like cross-task self-attention, is it right? Usually, self-attention is commmon that (B, Head, C, HW) * (B, Head, HW, C) = B, Head, HW* HW my qusetion: what is the difference between Cross-task Similarity and self-attention? Or Cross-task Similarity (B, Head, C, H, W) * (B, Head, C, H, W) = B, Head, C, H, W is a special way to calculate self-attention?

    opened by fanrz 0
Owner
null
Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras (ICCV 2021)

N-ImageNet: Towards Robust, Fine-Grained Object Recognition with Event Cameras Official PyTorch implementation of N-ImageNet: Towards Robust, Fine-Gra

null 32 Dec 26, 2022
(CVPR 2022 - oral) Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry

Multi-View Depth Estimation by Fusing Single-View Depth Probability with Multi-View Geometry Official implementation of the paper Multi-View Depth Est

Bae, Gwangbin 138 Dec 28, 2022
Official pytorch code for SSC-GAN: Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation(ICCV 2021)

SSC-GAN_repo Pytorch implementation for 'Semi-Supervised Single-Stage Controllable GANs for Conditional Fine-Grained Image Generation'.PDF SSC-GAN:Sem

tyty 4 Aug 28, 2022
Implementation for paper "STAR: A Structure-aware Lightweight Transformer for Real-time Image Enhancement" (ICCV 2021).

STAR-pytorch Implementation for paper "STAR: A Structure-aware Lightweight Transformer for Real-time Image Enhancement" (ICCV 2021). CVF (pdf) STAR-DC

null 43 Dec 21, 2022
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

Pranav 39 Nov 21, 2022
Light-weight network, depth estimation, knowledge distillation, real-time depth estimation, auxiliary data.

light-weight-depth-estimation Boosting Light-Weight Depth Estimation Via Knowledge Distillation, https://arxiv.org/abs/2105.06143 Junjie Hu, Chenyou F

Junjie Hu 13 Dec 10, 2022
Code and data of the Fine-Grained R2R Dataset proposed in paper Sub-Instruction Aware Vision-and-Language Navigation

Fine-Grained R2R Code and data of the Fine-Grained R2R Dataset proposed in the EMNLP2020 paper Sub-Instruction Aware Vision-and-Language Navigation. C

YicongHong 34 Nov 15, 2022
[ICCV 2021] Counterfactual Attention Learning for Fine-Grained Visual Categorization and Re-identification

Counterfactual Attention Learning Created by Yongming Rao*, Guangyi Chen*, Jiwen Lu, Jie Zhou This repository contains PyTorch implementation for ICCV

Yongming Rao 90 Dec 31, 2022
[ICCV 2021] Excavating the Potential Capacity of Self-Supervised Monocular Depth Estimation

EPCDepth EPCDepth is a self-supervised monocular depth estimation model, whose supervision is coming from the other image in a stereo pair. Details ar

Rui Peng 110 Dec 23, 2022
The official implementation of ICCV paper "Box-Aware Feature Enhancement for Single Object Tracking on Point Clouds".

Box-Aware Tracker (BAT) Pytorch-Lightning implementation of the Box-Aware Tracker. Box-Aware Feature Enhancement for Single Object Tracking on Point C

Kangel Zenn 5 Mar 26, 2022
PyTorch code for the paper "Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval".

Complementarity is the King: Multi-modal and Multi-grained Hierarchical Semantic Enhancement Network for Cross-modal Retrieval (M2HSE) PyTorch code fo

Xinlei-Pei 6 Dec 23, 2022
FIRA: Fine-Grained Graph-Based Code Change Representation for Automated Commit Message Generation

FIRA is a learning-based commit message generation approach, which first represents code changes via fine-grained graphs and then learns to generate commit messages automatically.

Van 21 Dec 30, 2022
WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose

WHENet: Real-time Fine-Grained Estimation for Wide Range Head Pose Yijun Zhou and James Gregson - BMVC2020 Abstract: We present an end-to-end head-pos

null 368 Dec 26, 2022
This is the official PyTorch implementation of the paper "TransFG: A Transformer Architecture for Fine-grained Recognition" (Ju He, Jie-Neng Chen, Shuai Liu, Adam Kortylewski, Cheng Yang, Yutong Bai, Changhu Wang, Alan Yuille).

TransFG: A Transformer Architecture for Fine-grained Recognition Official PyTorch code for the paper: TransFG: A Transformer Architecture for Fine-gra

Ju He 307 Jan 3, 2023
PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimation

StructDepth PyTorch implementation of our ICCV2021 paper: StructDepth: Leveraging the structural regularities for self-supervised indoor depth estimat

SJTU-ViSYS 112 Nov 28, 2022
The implementation of CVPR2021 paper Temporal Query Networks for Fine-grained Video Understanding, by Chuhan Zhang, Ankush Gupta and Andrew Zisserman.

Temporal Query Networks for Fine-grained Video Understanding ?? This repository contains the implementation of CVPR2021 paper Temporal_Query_Networks

null 55 Dec 21, 2022
the official code for ICRA 2021 Paper: "Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation"

G2S This is the official code for ICRA 2021 Paper: Multimodal Scale Consistency and Awareness for Monocular Self-Supervised Depth Estimation by Hemang

NeurAI 4 Jul 27, 2022
PyTorch implementation for Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuous Sign Language Recognition.

Stochastic CSLR This is the PyTorch implementation for the ECCV 2020 paper: Stochastic Fine-grained Labeling of Multi-state Sign Glosses for Continuou

Zhe Niu 28 Dec 19, 2022
PyTorch implementation of Weak-shot Fine-grained Classification via Similarity Transfer

SimTrans-Weak-Shot-Classification This repository contains the official PyTorch implementation of the following paper: Weak-shot Fine-grained Classifi

BCMI 60 Dec 2, 2022