Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Overview

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

This is the code for the paper Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction by Daniel Gehrig*, Michelle Rüegg*, Mathias Gehrig, Javier Hidalgo-Carrió, and Davide Scaramuzza:

You can find a pdf of the paper here and the project homepage here. If you use this work in an academic context, please cite the following publication:

@Article{RAL21Gehrig,
  author        = {Daniel Gehrig, Michelle Rüegg, Mathias Gehrig, Javier Hidalgo-Carrio and Davide Scaramuzza},
  title         = {Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction},
  journal       = {{IEEE} Robotic and Automation Letters. (RA-L)},
  url           = {http://rpg.ifi.uzh.ch/docs/RAL21_Gehrig.pdf},
  year          = 2021
}

If you use the event-camera plugin go to CARLA, please cite the following publication:

@Article{Hidalgo20threedv,
  author        = {Javier Hidalgo-Carrio, Daniel Gehrig and Davide Scaramuzza},
  title         = {Learning Monocular Dense Depth from Events},
  journal       = {{IEEE} International Conference on 3D Vision.(3DV)},
  url           = {http://rpg.ifi.uzh.ch/docs/3DV20_Hidalgo.pdf},
  year          = 2020
}

Install with Anaconda

The installation requires Anaconda3. You can create a new Anaconda environment with the required dependencies as follows (make sure to adapt the CUDA toolkit version according to your setup):

conda create --name RAMNET python=3.7
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
pip install tb-nightly kornia scikit-learn scikit-image opencv-python

Branches

To run experiments on Event Scape plese switch to the main branch

git checkout main

To run experiments on real data from MVSEC, switch to asynchronous_irregular_real_data.

git checkout asynchronous_irregular_real_data

Checkpoints

The checkpoints for RAM-Net can be found here:

EventScape

This work uses the EventScape dataset which can be downloaded here:

Video to Events

Qualitative results on MVSEC

Here the qualitative results of RAM-Net against state-of-the-art is shown. The video shows MegaDepth, E2Depth and RAM-Net in the upper row, image and event inputs and depth ground truth in the lower row.

Video to Events

Using RAM-Net

A detailed description on how to run the code can be found in the README in the folder /RAM_Net. Another README can be found in /RAM_Net/configs, it describes the meaning of the different parameters in the configs.

Comments
  • How to train recurrent model on multiple GPUs ?

    How to train recurrent model on multiple GPUs ?

    Hi, Thank you for your hard work on this project!

    I want to train the model with recurrent structures on multiple GPUs. I used the troch.nn.DataParaller() to wrap the model, but I got error at ConvLSTM Layer. How can I train the model on multiple GPUs?

    I'd appreciate it if you could give me a tip! Thanks!

    opened by luoxijing 0
  • Can I use RAMNET with custom data?

    Can I use RAMNET with custom data?

    Hi, Thanks for sharing great work!
    I made a custom dataset using ESIM.

    I want to use RAMNET using my dataset, but how do I do that?
    I'd appreciate it if you could give me a tip.

    opened by Taeyoung96 0
  • Unable to run pretrained models

    Unable to run pretrained models

    I am not able to run the pretrained models. What config file do I need to run with the provided checkpoints? I get this error when the checkpoint is loaded: RuntimeError: Error(s) in loading state_dict for ERGB2DepthRecurrent: Missing key(s) in state_dict: "statenetphasedrecurrent.state_combination_images.0.recurrent_block.Gates.weight", "statenetphasedrecurrent.state_combination_images.0.recurrent_block.Gates.bias", "statenetphasedrecurrent.state_combination_images.1.recurrent_block.Gates.weight", "statenetphasedrecurrent.state_combination_images.1.recurrent_block.Gates.bias", "statenetphasedrecurrent.state_combination_images.2.recurrent_block.Gates.weight", "statenetphasedrecurrent.state_combination_images.2.recurrent_block.Gates.bias". Unexpected key(s) in state_dict: "statenetphasedrecurrent.head_events.conv2d.weight", "statenetphasedrecurrent.head_events.conv2d.bias", "statenetphasedrecurrent.encoders_events.0.conv2d.weight", "statenetphasedrecurrent.encoders_events.0.conv2d.bias", "statenetphasedrecurrent.encoders_events.1.conv2d.weight", ... etc.

    opened by ricshaw 0
  • Only

    Only "train_e2depth_si_grad_loss_statenet_baseline_e.json" does not have the parameter reg_factor

    Hi! This paper is really a breathtaking work.

    Could you explain why only "train_e2depth_si_grad_loss_statenet_baseline_e.json" does not have the parameter reg_factor?

    Moreover, there will be two warnings because of the absence of reg_factor in the json file mentioned above
    "RuntimeWarning: divide by zero encountered in divide frame = 1.0 + np.log(frame) / reg_factor" "RuntimeWarning: invalid value encountered in divide frame = 1.0 + np.log(frame) / reg_factor"

    And followed by an error "return forward_call(*input, **kwargs) TypeError: forward() missing 1 required positional argument: 'prev_states_lstm' "

    I am looking forward to your reply. Thanks!

    opened by Andybrizt 0
  • MVSEC

    MVSEC

    Could you share how do you unpack the available online data in ROS bag format and pack the stream of events in voxel grid synchronized with the ground truth depth ? I don't know how to process the MVSEC dataset

    opened by huacong 0
  • MVSEC

    MVSEC

    Could you please describe the structure of the mvsec dataset file? I have download the mvsec dataset, hdf5 file, however, I don't know how to process them. I hope you could give me a hand. Thanks!

    opened by huacong 0
  • how to train rpg_ramnet on eventscape

    how to train rpg_ramnet on eventscape

    Hello, thanks for this great work, but I worked for 2 weeks still couldn't make this code work on the Eventscape dataset.

    I tried to change the JSON, the dataloader , still did not work.

    For example, the eventscape's events are .npz files, but in dataloader they only read .npy files. And some other questions either.

    Really want to follow this good work. Hope for your help. THanks.

    opened by tlwzzy 0
Owner
Robotics and Perception Group
Robotics and Perception Group
Deep Learning Datasets Maker is a QGIS plugin to make datasets creation easier for raster and vector data.

Deep Learning Dataset Maker Deep Learning Datasets Maker is a QGIS plugin to make datasets creation easier for raster and vector data. How to use Down

deepbands 25 Dec 15, 2022
Cl datasets - PyTorch image dataloaders and utility functions to load datasets for supervised continual learning

Continual learning datasets Introduction This repository contains PyTorch image

berjaoui 5 Aug 28, 2022
Source code, datasets and trained models for the paper Learning Advanced Mathematical Computations from Examples (ICLR 2021), by François Charton, Amaury Hayat (ENPC-Rutgers) and Guillaume Lample

Maths from examples - Learning advanced mathematical computations from examples This is the source code and data sets relevant to the paper Learning a

Facebook Research 171 Nov 23, 2022
EMNLP 2021 paper Models and Datasets for Cross-Lingual Summarisation.

This repository contains data and code for our EMNLP 2021 paper Models and Datasets for Cross-Lingual Summarisation. Please contact me at [email protected]

null 9 Oct 28, 2022
A Pytorch implementation of CVPR 2021 paper "RSG: A Simple but Effective Module for Learning Imbalanced Datasets"

RSG: A Simple but Effective Module for Learning Imbalanced Datasets (CVPR 2021) A Pytorch implementation of our CVPR 2021 paper "RSG: A Simple but Eff

null 120 Dec 12, 2022
Code and datasets for the paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction"

KnowPrompt Code and datasets for our paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction" Requireme

ZJUNLP 137 Dec 31, 2022
Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision https://arxiv.org/abs/2003.00393 Abstract Active learning (AL) aims to min

Denis 29 Nov 21, 2022
NeurIPS 2021 Datasets and Benchmarks Track

AP-10K: A Benchmark for Animal Pose Estimation in the Wild Introduction | Updates | Overview | Download | Training Code | Key Questions | License Intr

AP-10K 82 Dec 11, 2022
Official PyTorch implementation of "Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets" (ICLR 2021)

Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets This is the official PyTorch implementation for the paper Rapid Neural A

null 48 Dec 26, 2022
[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets

[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets Introduction This repo contains the source code accompanying the paper: Well-tuned Sim

null 52 Jan 4, 2023
A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''.

P-tuning A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''. How to use our code We have released the code

THUDM 562 Dec 27, 2022
Preprocessed Datasets for our Multimodal NER paper

Unified Multimodal Transformer (UMT) for Multimodal Named Entity Recognition (MNER) Two MNER Datasets and Codes for our ACL'2020 paper: Improving Mult

null 76 Dec 21, 2022
Datasets accompanying the paper ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers.

ConditionalQA Datasets accompanying the paper ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers. Disclaimer This dataset

null 2 Oct 14, 2021
The implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets.

Joint t-sne This is the implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets. abstract: We present Jo

IDEAS Lab 7 Dec 18, 2022
The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

This repository contains the software implementation of most algorithms used or developed in my research. The LaTeX and Python code for generating the

João Fonseca 3 Jan 3, 2023
Code, Models and Datasets for OpenViDial Dataset

OpenViDial This repo contains downloading instructions for the OpenViDial dataset in 《OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Vis

null 119 Dec 8, 2022
Final project code: Implementing MAE with downscaled encoders and datasets, for ESE546 FA21 at University of Pennsylvania

546 Final Project: Masked Autoencoder Haoran Tang, Qirui Wu 1. Training To train the network, please run mae_pretraining.py. Please modify folder path

Haoran Tang 0 Apr 22, 2022
Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.

NeuralTextures This is repository with inference code for paper "StylePeople: A Generative Model of Fullbody Human Avatars" (CVPR21). This code is for

Visual Understanding Lab @ Samsung AI Center Moscow 18 Oct 6, 2022