Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks

Related tags

Deep Learning mono-semantic-maps
Overview

Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks

This is the code associated with the paper Predicting Semantic Map Representations from Images with Pyramid Occupancy Networks, published at CVPR 2020.

Pyramid Occupancy Network architecture

Data generation

In our work we report results on two large-scale autonomous driving datasets: NuScenes and Argoverse. The birds-eye-view ground truth labels we use to train and evaluate our networks are generated by combining map information provided by the two datasets with 3D bounding box annotations, which we rasterise to produce a set of one-hot binary labels. We also make use of LiDAR point clouds to infer regions of the birds-eye-view which are completely occluded by buildings or other objects.

NuScenes

To train our method on NuScenes you will first need to

  1. Download the NuScenes dataset which can be found at https://www.nuscenes.org/download. Only the metadata, keyframe and lidar blobs are necessary.
  2. Download the map expansion pack. Note that to replicate our original results you should use the original version of the expansion (v1.0). The later versions fixed some bugs with the original maps so we would expect even better performance!
  3. Install the NuScenes devkit from https://github.com/nutonomy/nuscenes-devkit
  4. Cd to mono-semantic-maps
  5. Edit the configs/datasets/nuscenes.yml file, setting the dataroot and label_root entries to the location of the NuScenes dataset and the desired ground truth folder respectively.
  6. Run our data generation script: python scripts/make_nuscenes_labels.py. Bewarned there's a lot of data so this will take a few hours to run!

Argoverse

To train on the Argoverse dataset:

  1. Download the Argoverse tracking data from https://www.argoverse.org/data.html#tracking-link. Our models were trained on version 1.1, you will need to download the four training blobs, validation blob, and the HD map data.
  2. Install the Argoverse devkit from https://github.com/argoai/argoverse-api
  3. Cd to mono-semantic-maps
  4. Edit the configs/datasets/argoverse.yml file, setting the dataroot and label_root entries to the location of the install Argoverse data and the desired ground truth folder respectively.
  5. Run our data generation script: python scripts/make_argoverse_labels.py. This script will also take a while to run!

Training

Once ground truth labels have been generated, you can train our method by running the train.py script in the root directory:

python train.py --dataset nuscenes --model pyramid

The --dataset flag allows you to specify the dataset to train on, either 'argoverse' or 'nuscenes'. The model flag allows training of the proposed method 'pyramid', or one of the baseline methods ('vpn' or 'ved'). Additional command line options can be specified by passing a list of key-value pairs to the --options flag. The full list of configurable options can be found in the configs/defaults.yml file.

Comments
  • code & dataset release?

    code & dataset release?

    Hi Tom,

    That's great work!
    I would love to play around with the data and models :)

    Do you plan to release anytime soon the following?

    1. the dataset-creation pipeline from ArgoVerse and NuScenes
    2. the created dataset (ground-truth maps etc.)
    3. your networks and models.

    I think all of the above would be of great value to the community. Are you planning to do that soon? Or is it dependent on the paper first being published?

    Thanks a lot! Z.

    opened by zoharbarzelay 7
  • Small fixes to generate the dataset labels and train the models

    Small fixes to generate the dataset labels and train the models

    This PR fixes a few issue when generating the dataset labels and training the models.

    In render_polygon, sometimes the list of polygon is empty, in that case nothing should be done right?

    opened by anthonyhu 3
  • Model overfitting and not learning

    Model overfitting and not learning

    I used your script and generated the ground truth labels and then used your code to train the model. However, the model is only overfitting. Train IoUs increase consistently but validation IoUs do not increase.

    Screenshot from 2022-09-22 16-06-03 Screenshot from 2022-09-22 16-06-21

    opened by sadrasafa 1
  • Dear author,Exception: Error: You are using an outdated map version (%s)! Please go to https://www.nuscenes.org/download to download the latest map!

    Dear author,Exception: Error: You are using an outdated map version (%s)! Please go to https://www.nuscenes.org/download to download the latest map!

    Exception: Error: You are using an outdated map version (%s)! Please go to https://www.nuscenes.org/download to download the latest map!

    I would like to ask if it is not possible to use the v1.0 map expansion?

    Thx

    opened by emilyemliyM 2
  • KeyError when train argoverse dataset

    KeyError when train argoverse dataset

    Hi, thanks for the great work! I want to train this model on argoverse dataset, got following error:

    === Beginning epoch 1 of 200 ===
  0%|                                                                                                                                                | 0/50000 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/media/new_data3/mono-semantic-maps-master/train.py", line 340, in <module>
    main()
  File "/media/new_data3/mono-semantic-maps-master/train.py", line 316, in main
    train(train_loader, model, criterion, optimiser, summary, config, epoch)
  File "/media/new_data3/mono-semantic-maps-master/train.py", line 34, in train
    for i, batch in enumerate(tqdm(dataloader)):
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/tqdm/std.py", line 1180, in __iter__
    for obj in iterable:
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
KeyError: Caught KeyError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/media/new_data3/anaconda3/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/media/new_data3/mono-semantic-maps-master/src/data/augmentation.py", line 14, in __getitem__
    image, calib, labels, mask = self.dataset[index]
  File "/media/new_data3/mono-semantic-maps-master/src/data/argoverse/dataset.py", line 59, in __getitem__
    split, log, camera = self.examples[timestamp]
KeyError: 30314

    It seems like examples are indexed by timestamp inside argoverse, which can not index by normal numbers. How can i fix this issue? Thanks.

    My env is:

    python 3.9.7
1.10.2+cu113
    opened by XinnWang 3
Owner
Thomas Roddick
Thomas Roddick
GitHub
ICCV2021 Oral SA-ConvONet: Sign-Agnostic Optimization of Convolutional Occupancy Networks

Sign-Agnostic Convolutional Occupancy Networks Paper | Supplementary | Video | Teaser Video | Project Page This repository contains the implementation

null 64 Jan 5, 2023
An image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testingAn image base contains 490 images for learning (400 cars and 90 boats), and another 21 images for testing

SVM Données Une base d’images contient 490 images pour l’apprentissage (400 voitures et 90 bateaux), et encore 21 images pour fait des tests. Prétrait

Achraf Rahouti 3 Nov 30, 2021
Repository for the paper "Online Domain Adaptation for Occupancy Mapping", RSS 2020

RSS 2020 - Online Domain Adaptation for Occupancy Mapping Repository for the paper "Online Domain Adaptation for Occupancy Mapping", Robotics: Science

Anthony 26 Sep 22, 2022
PyTorch implementation of SMODICE: Versatile Offline Imitation Learning via State Occupancy Matching

SMODICE: Versatile Offline Imitation Learning via State Occupancy Matching This is the official PyTorch implementation of SMODICE: Versatile Offline I

Jason Ma 14 Aug 30, 2022
Nest Protect integration for Home Assistant. This will allow you to integrate your smoke, heat, co and occupancy status real-time in HA.

Nest Protect integration for Home Assistant Custom component for Home Assistant to interact with Nest Protect devices via an undocumented and unoffici

Mick Vleeshouwer 175 Dec 29, 2022
[CVPR'22] COAP: Learning Compositional Occupancy of People

COAP: Compositional Articulated Occupancy of People Paper | Video | Project Page This is the official implementation of the CVPR 2022 paper COAP: Lear

Marko Mihajlovic 111 Dec 11, 2022
Project for tracking occupancy in Tel-Aviv parking lots.

Ahuzat Dibuk - Tracking occupancy in Tel-Aviv parking lots main.py This module was set-up to be executed on Google Cloud Platform. I run it every 15 m

Geva Kipper 35 Nov 22, 2022
Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019)

Adaptive Pyramid Context Network for Semantic Segmentation (APCNet CVPR'2019) Introduction Official implementation of Adaptive Pyramid Context Network

null 21 Nov 9, 2022
Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation.

Unified-EPT Code for the ICCV 2021 Workshop paper: A Unified Efficient Pyramid Transformer for Semantic Segmentation. Installation Linux, CUDA>=10.0,

null 29 Aug 23, 2022
TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation, CVPR2022

TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation Paper Links: TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentati

Hust Visual Learning Team 253 Dec 21, 2022
SkipGNN: Predicting Molecular Interactions with Skip-Graph Networks (Scientific Reports)

SkipGNN: Predicting Molecular Interactions with Skip-Graph Networks Molecular interaction networks are powerful resources for the discovery. While dee

Kexin Huang 49 Oct 15, 2022
Codes for TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization.

TS-CAM: Token Semantic Coupled Attention Map for Weakly SupervisedObject Localization This is the official implementaion of paper TS-CAM: Token Semant

vasgaowei 112 Jan 2, 2023
Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localization and Semantic Segmentation (CVPR 2022)

CCAM (Unsupervised) Code repository for our paper "CCAM: Contrastive learning of Class-agnostic Activation Map for Weakly Supervised Object Localizati

Computer Vision Insitute, SZU 113 Dec 27, 2022
An Implementation of SiameseRPN with Feature Pyramid Networks

SiameseRPN with FPN This project is mainly based on HelloRicky123/Siamese-RPN. What I've done is just add a Feature Pyramid Network method to the orig

null 3 Apr 16, 2022
The official implementation of paper Siamese Transformer Pyramid Networks for Real-Time UAV Tracking, accepted by WACV22

SiamTPN Introduction This is the official implementation of the SiamTPN (WACV2022). The tracker intergrates pyramid feature network and transformer in

Robotics and Intelligent Systems Control @ NYUAD 28 Nov 25, 2022
A tool for making map images from OpenTTD save games

OpenTTD Surveyor A tool for making map images from OpenTTD save games. This is not part of the main OpenTTD codebase, nor is it ever intended to be pa

Aidan Randle-Conde 9 Feb 15, 2022
A large dataset of 100k Google Satellite and matching Map images, resembling pix2pix's Google Maps dataset.

Larger Google Sat2Map dataset This dataset extends the aerial ⟷ Maps dataset used in pix2pix (Isola et al., CVPR17). The provide script download_sat2m

null 34 Dec 28, 2022
Honours project, on creating a depth estimation map from two stereo images of featureless regions

image-processing This module generates depth maps for shape-blocked-out images Install If working with anaconda, then from the root directory: conda e

null 2 Oct 17, 2022
RetinaNet-PyTorch - A RetinaNet Pytorch Implementation on remote sensing images and has the similar mAP result with RetinaNet in MMdetection

?? RetinaNet Horizontal Detector Based PyTorch This is a horizontal detector Ret

Fang Zhonghao 13 Nov 19, 2022