mmTransformer
Introduction
-
This repo is official implementation for mmTransformer in pytorch. Currently, the core code of mmTransformer is implemented in the commercial project, we provide inference code of model with six trajectory propopals for your reference.
-
For other information, please refer to our paper Multimodal Motion Prediction with Stacked Transformers. (CVPR 2021) [Paper] [Webpage]
Set up your virtual environment
-
Initialize virtual environment:
conda create -n mmTrans python=3.7 -
Install agoverse api. Please refer to this page.
-
Install the pytorch. The latest codes are tested on Ubuntu 16.04, CUDA11.1, PyTorch 1.8 and Python 3.7: (Note that we require the version of torch >= 1.5.0 for testing with pretrained model)
pip install torch==1.8.0+cu111\ torchvision==0.9.0+cu111\ torchaudio==0.8.0 -f https://download.pytorch.org/whl/torch_stable.html -
For other requirement, please install with following command:
pip install -r requirement.txt
Preparation
Download the code, model and data
-
Clone this repo from the GitHub.
git clone https://github.com/decisionforce/mmTransformer.git -
Download the pretrained model and data [here] (map.pkl for Python 3.7 is available [here]) and save it to
./modelsand./interm_data.cd mmTransformer mkdir models mkdir interm_data -
Finally, your directory structure should look something like this:
mmTransformer └── models └── demo.pt └── interm_data └── argoverse_info_val.pkl └── map.pkl
Preprocess the dataset
Alternatively, you can process the data from scratch using following commands.
-
Download Argoverse dataset and create a symbolic link to
./datafolder or use following commands.cd path/to/mmtransformer/root mkdir data cd data wget https://s3.amazonaws.com/argoai-argoverse/forecasting_val_v1.1.tar.gz tar -zxvf forecasting_val_v1.1.tar.gz -
Then extract the agent and map information from raw data via Argoverse API:
python -m lib.dataset.argoverse_convertor ./config/demo.py -
Finally, your directory structure should look something like above illustrated.
Format of processed data in ‘argoverse_info_val.pkl’:
Format of map information in ‘map.pkl’:
Run the mmTransformer
For testing:
python Evaluation.py ./config/demo.py --model-name demo
Results
Here we showcase the expected results on validation set:
| Model | Expected results | Results in paper |
|---|---|---|
| minADE | 0.709 | 0.713 |
| minFDE | 1.081 | 1.153 |
| MR (K=6) | 10.2 | 10.6 |
TODO
- We are going to open source our visualization tools and a demo result. (TBD)
Contact us
If you have any issues with the code, please contact to this email: [email protected]
Citation
If you find our work useful for your research, please consider citing the paper
@article{liu2021multimodal,
title={Multimodal Motion Prediction with Stacked Transformers},
author={Liu, Yicheng and Zhang, Jinghuai and Fang, Liangji and Jiang, Qinhong and Zhou, Bolei},
journal={Computer Vision and Pattern Recognition},
year={2021}
}
Without filtering:
138 Dec 9, 2022
44 Dec 12, 2022
308 Jan 9, 2023
78 Nov 19, 2022
37 Dec 22, 2022
184 Dec 11, 2022
68 Dec 6, 2022
28 Dec 30, 2022
109 Dec 23, 2022
215 Jan 6, 2023
334 Dec 23, 2022
55 Nov 14, 2022
76 Jan 2, 2023
157 Dec 16, 2022
149 Jan 8, 2023
116 Dec 14, 2022
128 Dec 8, 2022