Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Robotics and Perception Group

Last update: Dec 26, 2022

Related tags

Deep Learning rpg_ramnet

Overview

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

This is the code for the paper Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction by Daniel Gehrig*, Michelle Rüegg*, Mathias Gehrig, Javier Hidalgo-Carrió, and Davide Scaramuzza:

You can find a pdf of the paper here and the project homepage here. If you use this work in an academic context, please cite the following publication:

@Article{RAL21Gehrig,
  author        = {Daniel Gehrig, Michelle Rüegg, Mathias Gehrig, Javier Hidalgo-Carrio and Davide Scaramuzza},
  title         = {Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction},
  journal       = {{IEEE} Robotic and Automation Letters. (RA-L)},
  url           = {http://rpg.ifi.uzh.ch/docs/RAL21_Gehrig.pdf},
  year          = 2021
}

If you use the event-camera plugin go to CARLA, please cite the following publication:

@Article{Hidalgo20threedv,
  author        = {Javier Hidalgo-Carrio, Daniel Gehrig and Davide Scaramuzza},
  title         = {Learning Monocular Dense Depth from Events},
  journal       = {{IEEE} International Conference on 3D Vision.(3DV)},
  url           = {http://rpg.ifi.uzh.ch/docs/3DV20_Hidalgo.pdf},
  year          = 2020
}

Install with Anaconda

The installation requires Anaconda3. You can create a new Anaconda environment with the required dependencies as follows (make sure to adapt the CUDA toolkit version according to your setup):

conda create --name RAMNET python=3.7
conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch
pip install tb-nightly kornia scikit-learn scikit-image opencv-python

Branches

To run experiments on Event Scape plese switch to the main branch

git checkout main

To run experiments on real data from MVSEC, switch to asynchronous_irregular_real_data.

git checkout asynchronous_irregular_real_data

Checkpoints

The checkpoints for RAM-Net can be found here:

EventScape

This work uses the EventScape dataset which can be downloaded here:

Qualitative results on MVSEC

Here the qualitative results of RAM-Net against state-of-the-art is shown. The video shows MegaDepth, E2Depth and RAM-Net in the upper row, image and event inputs and depth ground truth in the lower row.

Using RAM-Net

A detailed description on how to run the code can be found in the README in the folder /RAM_Net. Another README can be found in /RAM_Net/configs, it describes the meaning of the different parameters in the configs.

Comments

How to train recurrent model on multiple GPUs ?

Hi, Thank you for your hard work on this project!

I want to train the model with recurrent structures on multiple GPUs. I used the troch.nn.DataParaller() to wrap the model, but I got error at ConvLSTM Layer. How can I train the model on multiple GPUs?

I'd appreciate it if you could give me a tip! Thanks!

opened by luoxijing 0
Can I use RAMNET with custom data?

Hi, Thanks for sharing great work!
I made a custom dataset using ESIM.

I want to use RAMNET using my dataset, but how do I do that?
I'd appreciate it if you could give me a tip.

opened by Taeyoung96 0
Unable to run pretrained models

I am not able to run the pretrained models. What config file do I need to run with the provided checkpoints? I get this error when the checkpoint is loaded: RuntimeError: Error(s) in loading state_dict for ERGB2DepthRecurrent: Missing key(s) in state_dict: "statenetphasedrecurrent.state_combination_images.0.recurrent_block.Gates.weight", "statenetphasedrecurrent.state_combination_images.0.recurrent_block.Gates.bias", "statenetphasedrecurrent.state_combination_images.1.recurrent_block.Gates.weight", "statenetphasedrecurrent.state_combination_images.1.recurrent_block.Gates.bias", "statenetphasedrecurrent.state_combination_images.2.recurrent_block.Gates.weight", "statenetphasedrecurrent.state_combination_images.2.recurrent_block.Gates.bias". Unexpected key(s) in state_dict: "statenetphasedrecurrent.head_events.conv2d.weight", "statenetphasedrecurrent.head_events.conv2d.bias", "statenetphasedrecurrent.encoders_events.0.conv2d.weight", "statenetphasedrecurrent.encoders_events.0.conv2d.bias", "statenetphasedrecurrent.encoders_events.1.conv2d.weight", ... etc.

opened by ricshaw 0
Only "train_e2depth_si_grad_loss_statenet_baseline_e.json" does not have the parameter reg_factor

Hi! This paper is really a breathtaking work.

Could you explain why only "train_e2depth_si_grad_loss_statenet_baseline_e.json" does not have the parameter reg_factor?

Moreover, there will be two warnings because of the absence of reg_factor in the json file mentioned above
"RuntimeWarning: divide by zero encountered in divide frame = 1.0 + np.log(frame) / reg_factor" "RuntimeWarning: invalid value encountered in divide frame = 1.0 + np.log(frame) / reg_factor"

And followed by an error "return forward_call(*input, **kwargs) TypeError: forward() missing 1 required positional argument: 'prev_states_lstm' "

I am looking forward to your reply. Thanks!

opened by Andybrizt 0
MVSEC

Could you share how do you unpack the available online data in ROS bag format and pack the stream of events in voxel grid synchronized with the ground truth depth ? I don't know how to process the MVSEC dataset

opened by huacong 0
MVSEC

Could you please describe the structure of the mvsec dataset file? I have download the mvsec dataset, hdf5 file, however, I don't know how to process them. I hope you could give me a hand. Thanks!

opened by huacong 0
how to train rpg_ramnet on eventscape

Hello, thanks for this great work, but I worked for 2 weeks still couldn't make this code work on the Eventscape dataset.

I tried to change the JSON, the dataloader , still did not work.

For example, the eventscape's events are .npz files, but in dataloader they only read .npy files. And some other questions either.

Really want to follow this good work. Hope for your help. THanks.

opened by tlwzzy 0

Code and datasets for the paper "Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction" (RA-L, 2021)

Related tags

Overview

Combining Events and Frames using Recurrent Asynchronous Multimodal Networks for Monocular Depth Prediction

Install with Anaconda

Branches

Checkpoints

EventScape

Qualitative results on MVSEC

Using RAM-Net

Comments

How to train recurrent model on multiple GPUs ?

Can I use RAMNET with custom data?

Unable to run pretrained models

Only "train_e2depth_si_grad_loss_statenet_baseline_e.json" does not have the parameter reg_factor

MVSEC

MVSEC

how to train rpg_ramnet on eventscape

Owner

Robotics and Perception Group

Deep Learning Datasets Maker is a QGIS plugin to make datasets creation easier for raster and vector data.

Cl datasets - PyTorch image dataloaders and utility functions to load datasets for supervised continual learning

Source code, datasets and trained models for the paper Learning Advanced Mathematical Computations from Examples (ICLR 2021), by François Charton, Amaury Hayat (ENPC-Rutgers) and Guillaume Lample

EMNLP 2021 paper Models and Datasets for Cross-Lingual Summarisation.

A Pytorch implementation of CVPR 2021 paper "RSG: A Simple but Effective Module for Learning Imbalanced Datasets"

Code and datasets for the paper "KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction"

Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

NeurIPS 2021 Datasets and Benchmarks Track

Official PyTorch implementation of "Rapid Neural Architecture Search by Learning to Generate Graphs from Datasets" (ICLR 2021)

[NeurIPS 2021] Well-tuned Simple Nets Excel on Tabular Datasets

A novel method to tune language models. Codes and datasets for paper ``GPT understands, too''.

The official repository for our paper "The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers". We significantly improve the systematic generalization of transformer models on a variety of datasets using simple tricks and careful considerations.

Preprocessed Datasets for our Multimodal NER paper

Datasets accompanying the paper ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers.

The implementation for paper Joint t-SNE for Comparable Projections of Multiple High-Dimensional Datasets.

The LaTeX and Python code for generating the paper, experiments' results and visualizations reported in each paper is available (whenever possible) in the paper's directory

Code, Models and Datasets for OpenViDial Dataset

Final project code: Implementing MAE with downscaled encoders and datasets, for ESE546 FA21 at University of Pennsylvania

Inference code for "StylePeople: A Generative Model of Fullbody Human Avatars" paper. This code is for the part of the paper describing video-based avatars.