Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals
This repository contains code for "Multimodal trajectory prediction conditioned on lane-graph traversals" by Nachiket Deo, Eric M. Wolff and Oscar Beijbom, presented at CoRL 2021.
@inproceedings{deo2021multimodal,
title={Multimodal Trajectory Prediction Conditioned on Lane-Graph Traversals},
author={Deo, Nachiket and Wolff, Eric and Beijbom, Oscar},
booktitle={5th Annual Conference on Robot Learning},
year={2021}
}
Note: While I'm one of the authors of the paper, this is an independent re-implementation of the original code developed during an internship at Motional. The code follows the implementation details in the paper. Hope this helps! -Nachiket
Installation
-
Clone this repository
-
Set up a new conda environment
conda create --name pgp python=3.7
- Install dependencies
conda activate pgp
# nuScenes devkit
pip install nuscenes-devkit
# Pytorch: The code has been tested with Pytorch 1.7.1, CUDA 10.1, but should work with newer versions
conda install pytorch==1.7.1 torchvision==0.8.2 torchaudio==0.7.2 cudatoolkit=10.1 -c pytorch
# Additional utilities
pip install ray
pip install psutil
pip install positional-encodings
pip install imageio
pip install tensorboard
Dataset
-
Download the nuScenes dataset. For this project we just need the following.
- Metadata for the Trainval split (v1.0)
- Map expansion pack (v1.3)
-
Organize the nuScenes root directory as follows
└── nuScenes/
├── maps/
| ├── basemaps/
| ├── expansion/
| ├── prediction/
| ├── 36092f0b03a857c6a3403e25b4b7aab3.png
| ├── 37819e65e09e5547b8a3ceaefba56bb2.png
| ├── 53992ee3023e5494b90c316c183be829.png
| └── 93406b464a165eaba6d9de76ca09f5da.png
└── v1.0-trainval
├── attribute.json
├── calibrated_sensor.json
...
└── visibility.json
- Run the following script to extract pre-processed data. This speeds up training significantly.
python preprocess.py -c configs/preprocess_nuscenes.yml -r path/to/nuScenes/root/directory -d path/to/directory/with/preprocessed/data
Inference
You can download the trained model weights using this link.
To evaluate on the nuScenes val set run the following script. This will generate a text file with evaluation metrics at the specified output directory. The results should match the benchmark entry on Eval.ai.
python evaluate.py -c configs/pgp_gatx2_lvm_traversal.yml -r path/to/nuScenes/root/directory -d path/to/directory/with/preprocessed/data -o path/to/output/directory -w path/to/trained/weights
To visualize predictions run the following script. This will generate gifs for a set of instance tokens (track ids) from nuScenes val at the specified output directory.
python visualize.py -c configs/pgp_gatx2_lvm_traversal.yml -r path/to/nuScenes/root/directory -d path/to/directory/with/preprocessed/data -o path/to/output/directory -w path/to/trained/weights
Training
To train the model from scratch, run
python train.py -c configs/pgp_gatx2_lvm_traversal.yml -r path/to/nuScenes/root/directory -d path/to/directory/with/preprocessed/data -o path/to/output/directory -n 150
The training script will save training checkpoints and tensorboard logs in the output directory.
To launch tensorboard, run
tensorboard --logdir=path/to/output/directory/tensorboard_logs
but the code implementation is different
5 Jun 28, 2022
520 Dec 31, 2022
57 Nov 8, 2022
88 Dec 26, 2022
2 Oct 22, 2022
21 Mar 30, 2022
6 Nov 16, 2022
8 Jan 30, 2022
8 May 25, 2022
5 Mar 1, 2022
32 Nov 21, 2022
195 Jan 4, 2023
8.2k Jan 5, 2023
43 Nov 12, 2022
32 Dec 25, 2022
199 Dec 27, 2022
24 Jul 19, 2022
16.3k Dec 31, 2022