Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision
Project | PDF | Poster
Fangyu Li, N. Dinesh Reddy, Xudong Chen and Srinivasa G. Narasimhan
Proceedings of IEEE Intelligent Vehicles Symposium (IV'21)
Best Paper Award
Following instructions below, the user will get keypoints, trajectory reconstruction and vehicular activity clustering results like
Set up
The set up process can be skipped if using docker. Please check "Docker" section.
Python
Python version 3.6.9 is used. Python packages are in requirements.txt .
git clone https://github.com/Emrys-Lee/Traffic4D-Release.git
sudo apt-get install python3.6
sudo apt-get install python3-pip
cd Traffic4D-Release
pip3 install -r requirements.txt
C++
Traffic4D uses C++ libraries ceres and pybind for efficient optimization. pybind needs clang compiler, so Traffic4D uses clang compiler.
Install clang compiler
sudo apt-get install clang++-6.0
Install prerequisites for ceres
# CMake
sudo apt-get install cmake
# google-glog + gflags
sudo apt-get install libgoogle-glog-dev libgflags-dev
# BLAS & LAPACK
sudo apt-get install libatlas-base-dev
# Eigen3
sudo apt-get install libeigen3-dev
# SuiteSparse and CXSparse (optional)
sudo apt-get install libsuitesparse-dev
Download and install ceres
wget https://github.com/ceres-solver/ceres-solver/archive/1.12.0.zip
unzip 1.12.0.zip
cd ceres-solver-1.12.0/
mkdir build
cd build
cmake ..
make
sudo make install
Download and install pybind
git clone https://github.com/pybind/pybind11
cd pybind11
cmake .
make
sudo make install
Build Traffic4D optimization library
cd Traffic4D-Release/src/ceres
make
ceres_reconstruct.so and ceres_spline.so are generated under path Traffic4D-Release/src/ceres/.
Dataset
Download dataset and pre-generated results from here, and put it under Traffic4D-Release/.
cd Traffic4D-Release
mv Data-Traffic4D.zip ./
unzip Data-Traffic4D.zip
The directory should be like
Traffic4D-Release/
Data-Traffic4D/
└───fifth_morewood/
└───fifth_morewood_init.vd
└───top_view.png
└───images/
00001.jpg
00002.jpg
...
06288.jpg
└───arterial_kennedy/
└───arterial_kennedy_init.vd
└───top_view.png
└───images/
<put AI City Challenge frames here>
...
The input and output paths can be modified in config/*.yml.
Explanation
1. Input videos
Sample videos in Traffic4D are provided. Note arterial_kennedy and dodge_century are from Nvidia AI City Challenge City-Scale Multi-Camera Vehicle Tracking Challenge Track. Please request the access to the dataset here. Once get the data, run
ffmpeg -i <mtmc-dir>/train/S01/c001/vdo.avi Traffic4D-Release/Data-Traffic4D/arterial_kennedy/images/%05d.jpg
ffmpeg -i <mtmc-dir>/test/S02/c007/vdo.avi Traffic4D-Release/Data-Traffic4D/dodge_century/images/%05d.jpg
to extract frames into images/.
2. Pre-Generated 2D results
Detected 2D bounding boxes, keypoints and tracking IDs are stored in *_init.vd. Check Occlusionnet implementation for detecting keypoints; V-IOU for multi-object tracking.
3. Output folder
Folder Traffic4D-Release/Result/ will be created by default.
Experiments
Run python exp/traffic4d.py config/<intersection_name>.yml <action>. Here YML configuration files for multiple intersections are provided under config/ folder. <action> shoulbe be reconstruction or clustering to perform longitudinal reconstruction and activity clustering sequentially. For example, below runs Fifth and Morewood intersection.
cd Traffic4D-Release
python3 exp/traffic4d.py config/fifth_morewood.yml reconstruction
python3 exp/traffic4d.py config/fifth_morewood.yml clustering
Results
Find these results in the output folder:
- 2D keypoints: If 3D reconstruction is done, 2D reprojected keypoints will be plotted in
Traffic4D-Release/Result/<intersection_name>_keypoints/. - 3D reconstructed trajectories and clusters: The clustered 3D trajectories are plotted on the top view map as
Traffic4D-Release/Result/<intersection_name>_top_view.jpg.
Docker
We provide docker image with dependencies already set up. The steps in "Set up" can be skipped if you use docker image. You still need to clone the repo and download the dataset and put it in under Traffic4D-Release/.
git clone https://github.com/Emrys-Lee/Traffic4D-Release.git
Pull Traffic4D docker image.
docker pull emrysli/traffic4d-release:latest
Then create a container and map the git repo into docker container to access the dataset. For example, if the cloned repo locates at host directory /home/xxx/Traffic4D-Release, <path_to_repo> should be /home/xxx. If <path_in_container> is /home/yyy, then /home/xxx/Traffic4D-Release will be mapped as /home/yyy/Traffic4D-Release inside the container.
docker run -it -v <path_to_repo>/Traffic4D-Release:<path_in_container>/Traffic4D-Release emrysli/traffic4d-release:latest /bin/bash
Inside container compile Traffic4D again.
# inside container
cd <path_in_container>/Traffic4D-Release/src/ceres
make
Run experiments.
cd <path_in_container>/Traffic4D-Release
python3 exp/traffic4d.py config/fifth_morewood.yml reconstruction
python3 exp/traffic4d.py config/fifth_morewood.yml clustering
Trouble Shooting
tkintermodule is missing
File "/usr/local/lib/python3.6/dist-packages/matplotlib/backends/_backend_tk.py", line 5, in <module>
import tkinter as Tk
ModuleNotFoundError: No module named 'tkinter'
Solution: install tkinter.
sudo apt-get install python3-tk
opencvimport error such as
File "/usr/local/lib/python3.6/dist-packages/cv2/__init__.py", line 3, in <module>
from .cv2 import *
ImportError: libSM.so.6: cannot open shared object file: No such file or directory
Solution: install the missing libraries.
sudo apt-get install libsm6 libxrender1 libfontconfig1 libxext6
Citation
Traffic4D
@conference{Li-2021-127410,
author = {Fangyu Li and N. Dinesh Reddy and Xudong Chen and Srinivasa G. Narasimhan},
title = {Traffic4D: Single View Reconstruction of Repetitious Activity Using Longitudinal Self-Supervision},
booktitle = {Proceedings of IEEE Intelligent Vehicles Symposium (IV '21)},
year = {2021},
month = {July},
publisher = {IEEE},
keywords = {Self-Supervision, vehicle Detection, 4D Reconstruction, 3D reconstuction, Pose Estimation.},
}
Occlusion-Net
@inproceedings{onet_cvpr19,
author = {Reddy, N. Dinesh and Vo, Minh and Narasimhan, Srinivasa G.},
title = {Occlusion-Net: 2D/3D Occluded Keypoint Localization Using Graph Networks},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
pages = {7326--7335},
year = {2019}
}
63 Dec 16, 2022
296 Dec 29, 2022
![[ICCV'21] UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction](https://github.com/autonomousvision/unisurf/raw/main/media/unisurf_teaser.gif)
331 Dec 28, 2022
148 Nov 20, 2022
29 Nov 21, 2022
38 Sep 16, 2022
![[CVPR 2022] PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision (Oral)](https://github.com/Garfield-kh/PoseTriplet/raw/main/assets/dual-loop-detail-v3.jpg)
256 Dec 28, 2022
80 Dec 25, 2022
332 Jan 3, 2023
42 Dec 5, 2022
118 Jan 6, 2023
193 Dec 15, 2022
11 Feb 8, 2022
304 Jan 3, 2023
103 Dec 17, 2022
117 Dec 28, 2022
274 Jan 5, 2023
1 Nov 15, 2021