MultiNet
MultiNet is able to jointly perform road segmentation, car detection and street classification. The model achieves real-time speed and state-of-the-art performance in segmentation. Check out our paper for a detailed model description.
MultiNet is optimized to perform well at a real-time speed. It has two components: KittiSeg, which sets a new state-of-the art in road segmentation; and KittiBox, which improves over the baseline Faster-RCNN in both inference speed and detection performance.
The model is designed as an encoder-decoder architecture. It utilizes one VGG encoder and several independent decoders for each task. This repository contains generic code that combines several tensorflow models in one network. The code for the individual tasks is provided by the KittiSeg, KittiBox, and KittiClass repositories. These repositories are utilized as submodules in this project. This project is built to be compatible with the TensorVision back end, which allows for organizing experiments in a very clean way.
Requirements
The code requires Python 2.7, Tensorflow 1.0, as well as the following python libraries:
- matplotlib
- numpy
- Pillow
- scipy
- runcython
- commentjson
Those modules can be installed using: pip install numpy scipy pillow matplotlib runcython commentjson or pip install -r requirements.txt.
Setup
- Clone this repository:
https://github.com/MarvinTeichmann/MultiNet.git - Initialize all submodules:
git submodule update --init --recursive cd submodules/KittiBox/submodules/utils/ && maketo build cython code- [Optional] Download Kitti Road Data:
- Retrieve kitti data url here: http://www.cvlibs.net/download.php?file=data_road.zip
- Call
python download_data.py --kitti_url URL_YOU_RETRIEVED
- [Optional] Run
cd submodules/KittiBox/submodules/KittiObjective2/ && maketo build the Kitti evaluation code (see submodules/KittiBox/submodules/KittiObjective2/README.md for more information)
Running the model using demo.py only requires you to perform step 1-3. Step 4 and 5 is only required if you want to train your own model using train.py. Note that I recommend using download_data.py instead of downloading the data yourself. The script will also extract and prepare the data. See Section Manage data storage if you like to control where the data is stored.
To update MultiNet do:
- Pull all patches:
git pull - Update all submodules:
git submodule update --init --recursive
If you forget the second step you might end up with an inconstant repository state. You will already have the new code for MultiNet but run it old submodule versions code. This can work, but I do not run any tests to verify this.
Tutorial
Getting started
Run: python demo.py --gpus 0 --input data/demo/um_000005.png to obtain a prediction using demo.png as input.
Run: python evaluate.py to evaluate a trained model.
Run: python train.py --hypes hypes/multinet2.json to train a multinet2
If you like to understand the code, I would recommend looking at demo.py first. I have documented each step as thoroughly as possible in this file.
Only training of MultiNet3 (joint detection and segmentation) is supported out of the box. The data to train the classification model is not public an those cannot be used to train the full MultiNet3 (detection, segmentation and classification). The full code is given here, so you can still train MultiNet3 if you have your own data.
Manage Data Storage
MultiNet allows to separate data storage from code. This is very useful in many server environments. By default, the data is stored in the folder MultiNet/DATA and the output of runs in MultiNet/RUNS. This behaviour can be changed by setting the bash environment variables: $TV_DIR_DATA and $TV_DIR_RUNS.
Include export TV_DIR_DATA="/MY/LARGE/HDD/DATA" in your .profile and the all data will be downloaded to /MY/LARGE/HDD/DATA/. Include export TV_DIR_RUNS="/MY/LARGE/HDD/RUNS" in your .profile and all runs will be saved to /MY/LARGE/HDD/RUNS/MultiNet
Modifying Model & Train on your own data
The model is controlled by the file hypes/multinet3.json. This file points the code to the implementation of the submodels. The MultiNet code then loads all models provided and integrates the decoders into one neural network. To train on your own data, it should be enough to modify the hype files of the submodels. A good start will be the KittiSeg model, which is very well documented.
"models": {
"segmentation" : "../submodules/KittiSeg/hypes/KittiSeg.json",
"detection" : "../submodules/KittiBox/hypes/kittiBox.json",
"road" : "../submodules/KittiClass/hypes/KittiClass.json"
},
RUNDIR and Experiment Organization
MultiNet helps you to organize a large number of experiments. To do so, the output of each run is stored in its own rundir. Each rundir contains:
output.loga copy of the training output which was printed to your screentensorflow eventstensorboard can be run in rundirtensorflow checkpointsthe trained model can be loaded from rundir[dir] imagesa folder containing example output images.image_itercontrols how often the whole validation set is dumped[dir] model_filesA copy of all source code need to build the model. This can be very useful of you have many versions of the model.
To keep track of all the experiments, you can give each rundir a unique name with the --name flag. The --project flag will store the run in a separate subfolder allowing to run different series of experiments. As an example, python train.py --project batch_size_bench --name size_5 will use the following dir as rundir: $TV_DIR_RUNS/KittiSeg/batch_size_bench/size_5_KittiSeg_2017_02_08_13.12.
The flag --nosave is very useful to not spam your rundir.
Useful Flags & Variabels
Here are some Flags which will be useful when working with KittiSeg and TensorVision. All flags are available across all scripts.
--hypes : specify which hype-file to use
--logdir : specify which logdir to use
--gpus : specify on which GPUs to run the code
--name : assign a name to the run
--project : assign a project to the run
--nosave : debug run, logdir will be set to debug
In addition the following TensorVision environment Variables will be useful:
$TV_DIR_DATA: specify meta directory for data
$TV_DIR_RUNS: specify meta directory for output
$TV_USE_GPUS: specify default GPU behaviour.
On a cluster it is useful to set $TV_USE_GPUS=force. This will make the flag --gpus mandatory and ensure, that run will be executed on the right GPU.
Citation
If you benefit from this code, please cite our paper:
@article{teichmann2016multinet,
title={MultiNet: Real-time Joint Semantic Reasoning for Autonomous Driving},
author={Teichmann, Marvin and Weber, Michael and Zoellner, Marius and Cipolla, Roberto and Urtasun, Raquel},
journal={arXiv preprint arXiv:1612.07695},
year={2016}
}
![error in train.py ResourceExhaustedError OOM when allocating tensor with shape [1,256,92,309]](https://avatars.githubusercontent.com/u/15725187?v=4)
![ResourceExhaustedError (see above for traceback): OOM when allocating tensor of shape [7,7,512,4096] and type float [[Node: fc6/weights/Adam/Initializer/zeros = Const[dtype=DT_FLOAT, value=Tensor<type: float shape: [7,7,512,4 096] values: [[[0 0 0]]]...>, _device=](https://avatars.githubusercontent.com/u/37062395?v=4)
271 Nov 29, 2022
16 Dec 14, 2022
308 Jan 4, 2023
24 Dec 12, 2022
51 Nov 17, 2022
305 Dec 19, 2022
96 Dec 29, 2022
695 Jan 5, 2023
148 Dec 28, 2022
4 Sep 4, 2021
101 Dec 5, 2022
6 Oct 27, 2022
300 Dec 15, 2022
1 Feb 15, 2022
83 Jan 1, 2023
477 Jan 6, 2023
35 Oct 26, 2022
62 Dec 5, 2022
