Use tensorflow to implement a Deep Neural Network for real time lane detection

Overview

LaneNet-Lane-Detection

Use tensorflow to implement a Deep Neural Network for real time lane detection mainly based on the IEEE IV conference paper "Towards End-to-End Lane Detection: an Instance Segmentation Approach".You can refer to their paper for details https://arxiv.org/abs/1802.05591. This model consists of a encoder-decoder stage, binary semantic segmentation stage and instance semantic segmentation using discriminative loss function for real time lane detection task.

The main network architecture is as follows:

Network Architecture NetWork_Architecture

Installation

This software has only been tested on ubuntu 16.04(x64), python3.5, cuda-9.0, cudnn-7.0 with a GTX-1070 GPU. To install this software you need tensorflow 1.12.0 and other version of tensorflow has not been tested but I think it will be able to work properly in tensorflow above version 1.12. Other required package you may install them by

pip3 install -r requirements.txt

Test model

In this repo I uploaded a model trained on tusimple lane dataset Tusimple_Lane_Detection. The deep neural network inference part can achieve around a 50fps which is similar to the description in the paper. But the input pipeline I implemented now need to be improved to achieve a real time lane detection system.

The trained lanenet model weights files are stored in lanenet_pretrained_model. You can download the model and put them in folder model/tusimple_lanenet/

You can test a single image on the trained model as follows

python tools/test_lanenet.py --weights_path /PATH/TO/YOUR/CKPT_FILE_PATH 
--image_path ./data/tusimple_test_image/0.jpg

The results are as follows:

Test Input Image

Test Input

Test Lane Mask Image

Test Lane_Mask

Test Lane Binary Segmentation Image

Test Lane_Binary_Seg

Test Lane Instance Segmentation Image

Test Lane_Instance_Seg

If you want to evaluate the model on the whole tusimple test dataset you may call

python tools/evaluate_lanenet_on_tusimple.py 
--image_dir ROOT_DIR/TUSIMPLE_DATASET/test_set/clips 
--weights_path /PATH/TO/YOUR/CKPT_FILE_PATH 
--save_dir ROOT_DIR/TUSIMPLE_DATASET/test_set/test_output

If you set the save_dir argument the result will be saved in that folder or the result will not be saved but be displayed during the inference process holding on 3 seconds per image. I test the model on the whole tusimple lane detection dataset and make it a video. You may catch a glimpse of it bellow.

Tusimple test dataset gif tusimple_batch_test_gif

Train your own model

Data Preparation

Firstly you need to organize your training data refer to the data/training_data_example folder structure. And you need to generate a train.txt and a val.txt to record the data used for training the model.

The training samples consist of three components, a binary segmentation label file, a instance segmentation label file and the original image. The binary segmentation uses 255 to represent the lane field and 0 for the rest. The instance use different pixel value to represent different lane field and 0 for the rest.

All your training image will be scaled into the same scale according to the config file.

Use the script here to generate the tensorflow records file

python tools/make_tusimple_tfrecords.py 

Train model

In my experiment the training epochs are 80010, batch size is 4, initialized learning rate is 0.001 and use polynomial decay with power 0.9. About training parameters you can check the global_configuration/config.py for details. You can switch --net argument to change the base encoder stage. If you choose --net vgg then the vgg16 will be used as the base encoder stage and a pretrained parameters will be loaded. And you can modified the training script to load your own pretrained parameters or you can implement your own base encoder stage. You may call the following script to train your own model

python tools/train_lanenet_tusimple.py 

You may monitor the training process using tensorboard tools

During my experiment the Total loss drops as follows:
Training loss

The Binary Segmentation loss drops as follows:
Training binary_seg_loss

The Instance Segmentation loss drops as follows:
Training instance_seg_loss

Experiment

The accuracy during training process rises as follows: Training accuracy

Please cite my repo lanenet-lane-detection if you use it.

Recently updates 2018.11.10

Adjust some basic cnn op according to the new tensorflow api. Use the traditional SGD optimizer to optimize the whole model instead of the origin Adam optimizer used in the origin paper. I have found that the SGD optimizer will lead to more stable training process and will not easily stuck into nan loss which may often happen when using the origin code.

Recently updates 2018.12.13

Since a lot of user want a automatic tools to generate the training samples from the Tusimple Dataset. I upload the tools I use to generate the training samples. You need to firstly download the Tusimple dataset and unzip the file to your local disk. Then run the following command to generate the training samples and the train.txt file.

python tools/generate_tusimple_dataset.py --src_dir path/to/your/unzipped/file

The script will make the train folder and the test folder. The training samples of origin rgb image, binary label image, instance label image will be automatically generated in the training/gt_image, training/gt_binary_image, training/gt_instance_image folder.You may check it yourself before start the training process.

Pay attention that the script only process the training samples and you need to select several lines from the train.txt to generate your own val.txt file. In order to obtain the test images you can modify the script on your own.

Recently updates 2020.06.12

Add real-time segmentation model BiseNetV2 as lanenet backbone. You may modify the config/tusimple_lanenet.yaml config file to choose the front-end of lanenet model.

New lanenet model trainned based on BiseNetV2 can be found here

The new model can reach 78 fps in single image inference process.

MNN Project

Add tools to convert lanenet tensorflow ckpt model into mnn model and deploy the model on mobile device

Freeze your tensorflow ckpt model weights file

cd LANENET_PROJECT_ROOT_DIR
python mnn_project/freeze_lanenet_model.py -w lanenet.ckpt -s lanenet.pb

Convert pb model into mnn model

cd MNN_PROJECT_ROOT_DIR/tools/converter/build
./MNNConver -f TF --modelFile lanenet.pb --MNNModel lanenet.mnn --bizCode MNN

Add lanenet source code into MNN project

Add lanenet source code into MNN project and modified CMakeList.txt to compile the executable binary file.

TODO

  • Add a embedding visualization tools to visualize the embedding feature map
  • Add detailed explanation of training the components of lanenet separately.
  • Training the model on different dataset
  • [ ] Adjust the lanenet hnet model and merge the hnet model to the main lanenet model
  • [ ] Change the normalization function from BN to GN

Acknowledgement

The lanenet project refers to the following projects:

Contact

Scan the following QR to disscuss :) qr

Comments
  • Embedding branch 问题?

    Embedding branch 问题?

    用的数据是 tusimple lane dataset 训练出来的embedding.png 显示出来的照片一直都是 雪花点,不是正常的图片 论文上 Embedding branch。您这块代码是不是有问题,或者没有这块代码?

    进行聚合的时候 出现nan的问题?,计算loss的时候 被除数肯定是0的情况下? 我看了所有的图片,二值图都是正常的,实例图也是没有问题。。

    opened by SentimentSongs 76
  • Retraining model on new dataset

    Retraining model on new dataset

    i was successful in testing the trained model by getting the trained weights you uploaded in Dropbox. However, I want to retrain the model on new training data.

    I added one new image for the existing training data of five images following the instructions in the repo and added new images in the image, gt_image_instance, and gt_image_binary folders, but i get errors. I enter this line from your repo in bash:

    python tools/train_lanenet.py --net vgg --dataset_dir data/training_data_example/

    The errors I get are:

    cv2.error: OpenCV(3.4.2) /Users/travis/build/skvark/opencv-python/opencv/modules/imgproc/src/resize.cpp:4044: error: (-215:Assertion failed) !ssize.empty() in function 'resize'

    and sometimes i get this error:

    ValueError: Variable lanenet_loss/inference/encode/conv1_1/conv/W already exists

    I already modified the train.txt and val.txt and changed the file paths for the images found locally on my machine.

    How to fix this?

    opened by dscha09 69
  • 训练过程还是出现nan

    训练过程还是出现nan

    你的做法和我的一樣,我也是把學習率降低之後就可以訓練了,但是當epoch跑到一定次數之後,仍會有nan的情形出現. 而且accuracy只有在0.5~0.6之間,請問你有這樣的情形嗎? 謝謝!!

    你好,我遇到的问题和你的一模一样,请问你解决了吗?是怎么解决的啊?

    Originally posted by @fayechou in https://github.com/MaybeShewill-CV/lanenet-lane-detection/issue_comments#issuecomment-445708580

    help wanted 
    opened by fayechou 62
  • custom data training

    custom data training

    Hi, I'm trying to train with my own dataset and got this error, Any suggestion would be helpful

    Traceback (most recent call last): File "tools/train_lanenet.py", line 719, in train_lanenet(args.dataset_dir, args.weights_path, net_flag=args.net_flag) File "tools/train_lanenet.py", line 442, in train_lanenet train_images, train_binary_labels, train_instance_labels]) File "/home/pcific/anaconda3/envs/tf_gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run run_metadata_ptr) File "/home/pcific/anaconda3/envs/tf_gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run feed_dict_tensor, options, run_metadata) File "/home/pcific/anaconda3/envs/tf_gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run run_metadata) File "/home/pcific/anaconda3/envs/tf_gpu/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 470016 values, but the requested shape has 156672 [[{{node Reshape_2}}]] [[train_IteratorGetNext]]

    opened by aacmi 34
  • I use the model to train the CULane dataset, but...

    I use the model to train the CULane dataset, but...

    I use the model to train the CULane dataset, but there are questions: 1 the GPU memory is used 107M, seems that the gpu didn't work. 2 the training stops at the 3840th epoch, it is very strange, why?

    @MaybeShewill-CV need your help, thanks!

    opened by passion3394 25
  • ValueError: Cannot feed value of shape

    ValueError: Cannot feed value of shape

    Thanks for your share,I want to use your model to train my data,but raise some error: ValueError: Cannot feed value of shape (1, 256, 512, 3) for Tensor 'instance_input_label:0', which has shape '(?, 256, 512)',my works include:

    • draw different color on black background for instance segmentation(three channel),for example(10,10,10),(20,20,20),(30,30,30)...
    • draw white line on lane otherwise black(0,0,0),(255,255,255),(I try to use json file to draw line and then use cv2.imwrite to save,when i read the image,inspecte it's shape,it's channel is 3 even if the black backgroud is zeros(720,1280),3 channel image data is ok?).
    • generator data list for example:/xxx/image /xxx/gt_gray/gt_instance in train.txt and val.txt when i try to train from scrach,the code raise the error,if you can you tell me which step is error,i will appreciate for you help,thanks for your share too. @MaybeShewill-CV
    opened by bleedingfight 22
  • what will happen if I modify some config

    what will happen if I modify some config

    In config/global_config.py, I find the following config

    Set the shadownet validation batch size

    __C.TRAIN.VAL_BATCH_SIZE = 8

    Set the learning rate decay steps

    __C.TRAIN.LR_DECAY_STEPS = 410000

    Set the learning rate decay rate

    __C.TRAIN.LR_DECAY_RATE = 0.1

    Set the class numbers

    __C.TRAIN.CLASSES_NUMS = 2

    Set the image height

    __C.TRAIN.IMG_HEIGHT = 256

    Set the image width

    __C.TRAIN.IMG_WIDTH = 512

    what will happen if I change the IMG_HEIGHT and IMG_WIDTH to a bigger value? thanks.

    opened by passion3394 21
  • Testing LANENET

    Testing LANENET

    Hi ,

    Its working good for lane detection for one of my dataset. Im having problem like, after testing the model, when marking the detected lanes, each and everytime the colour is changing for the same line which is detected in previous frame, for example as shown in below two figure

    the color representation is changed in immediate frame. Kindly can you give suggestions to maintain same color representation for same lane in next frame if it got detected.

    Thanks for your valuable time, look forward for response.

    Regards, Anil.

    opened by chokkarapu 20
  • A question about the test result

    A question about the test result

    @MaybeShewill-CV Hi, I have trained on CULane dataset, and this is a strange test result, have you ever met this? And when two lanes are close, they are easy to become wider and be connected together, you can see it in the middle of the image. result_2

    opened by khw321 20
  • 使用dareset数据集里边的lanenet_hnet_data_processor.py报错啊

    使用dareset数据集里边的lanenet_hnet_data_processor.py报错啊

    改了 json_file_list = glob.glob('{:s}/*.json'.format('G:/dareset/dareset/label_data_0313.json' 'G:/dareset/training')) 两个路径 python 3.6 tensorflow 1.8.0 File "G:/lanenet-lane-detection/dareset/dareset/lanenet_hnet_data_processor.py", line 93, in _random_dataset random_idx = np.random.permutation(self._label_image_path.shape[0])

    File "mtrand.pyx", line 4907, in mtrand.RandomState.permutation

    File "mtrand.pyx", line 4841, in mtrand.RandomState.shuffle

    RecursionError: maximum recursion depth exceeded in comparison

    opened by baihualinxin 19
  • Train Loss and Val Loss as nan

    Train Loss and Val Loss as nan

    I am training the model with the BDD-100K dataset for a single class with an input dim of 256 x 512. I have resized the images from BDD original dim of 720 x1280 to 256 x 512 before adding them to the TFrecords.

    I am getting an error specific to train and val loss being nan. I have already removed images with no lane markings from the dataset. Still, I am getting the same error.

    I ran an experiment for 10 epochs which results are attached below Screenshot 2020-11-10 at 11 46 48 AM

    opened by letdivedeep 18
  • Val Loss Nan

    Val Loss Nan

    Training on Custom Data->taining data no. - ~10000 , Val data - ~ 500 -> Label data - binary 255 for lane and 0 for background and instance 1,2 3, 4, 5,6,... for lane and 0 for background Validity Loss coming Nan..
    But when training on small data around 100 and val data - 5 ...val loss is coming Please help sir which parameter should I check

    opened by pankaj251003 0
Owner
MaybeShewill-CV
Computer Vision R&D
MaybeShewill-CV
Find-Lane-Line - Use openCV library and Python to detect the road-lane-line

Find-Lane-Line This project is to use openCV library and Python to detect the road-lane-line. Data Pipeline Step one : Color Selection Step two : Cann

Kenny Cheng 3 Aug 17, 2022
LaneDetectionAndLaneKeeping - Lane Detection And Lane Keeping

LaneDetectionAndLaneKeeping This project is part of my bachelor's thesis. The go

null 5 Jun 27, 2022
Lane assist for ETS2, built with the ultra-fast-lane-detection model.

Euro-Truck-Simulator-2-Lane-Assist Lane assist for ETS2, built with the ultra-fast-lane-detection model. This project was made possible by the amazing

null 36 Jan 5, 2023
Lane follower: Lane-detector (OpenCV) + Object-detector (YOLO5) + CAN-bus

Lane Follower This code is for the lane follower, including perception and control, as shown below. Environment Hardware Industrial Camera Intel-NUC(1

Siqi Fan 3 Jul 7, 2022
A lane detection integrated Real-time Instance Segmentation based on YOLACT (You Only Look At CoefficienTs)

Real-time Instance Segmentation and Lane Detection This is a lane detection integrated Real-time Instance Segmentation based on YOLACT (You Only Look

Jin 4 Dec 30, 2022
Example scripts for the detection of lanes using the ultra fast lane detection model in Tensorflow Lite.

TFlite Ultra Fast Lane Detection Inference Example scripts for the detection of lanes using the ultra fast lane detection model in Tensorflow Lite. So

Ibai Gorordo 12 Aug 27, 2022
A Data Annotation Tool for Semantic Segmentation, Object Detection and Lane Line Detection.(In Development Stage)

Data-Annotation-Tool How to Run this Tool? To run this software, follow the steps: git clone https://github.com/Autonomous-Car-Project/Data-Annotation

TiVRA AI 13 Aug 18, 2022
Example scripts for the detection of lanes using the ultra fast lane detection model in ONNX.

Example scripts for the detection of lanes using the ultra fast lane detection model in ONNX.

Ibai Gorordo 35 Sep 7, 2022
TCNN Temporal convolutional neural network for real-time speech enhancement in the time domain

TCNN Pandey A, Wang D L. TCNN: Temporal convolutional neural network for real-time speech enhancement in the time domain[C]//ICASSP 2019-2019 IEEE Int

凌逆战 16 Dec 30, 2022
implement of SwiftNet:Real-time Video Object Segmentation

SwiftNet The official PyTorch implementation of SwiftNet:Real-time Video Object Segmentation, which has been accepted by CVPR2021. Requirements Python

haochen wang 64 Dec 14, 2022
Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs.

Lunar Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs. About Lunar can be modified to work

Zeyad Mansour 276 Jan 7, 2023
Real-Time-Student-Attendence-System - Real Time Student Attendence System

Real-Time-Student-Attendence-System The Student Attendance Management System Pro

Rounak Das 1 Feb 15, 2022
Official Tensorflow implementation of "M-LSD: Towards Light-weight and Real-time Line Segment Detection"

M-LSD: Towards Light-weight and Real-time Line Segment Detection Official Tensorflow implementation of "M-LSD: Towards Light-weight and Real-time Line

NAVER/LINE Vision 357 Jan 4, 2023
Implementation of our paper 'RESA: Recurrent Feature-Shift Aggregator for Lane Detection' in AAAI2021.

RESA PyTorch implementation of the paper "RESA: Recurrent Feature-Shift Aggregator for Lane Detection". Our paper has been accepted by AAAI2021. Intro

null 137 Jan 2, 2023
LaneAF: Robust Multi-Lane Detection with Affinity Fields

LaneAF: Robust Multi-Lane Detection with Affinity Fields This repository contains Pytorch code for training and testing LaneAF lane detection models i

null 155 Dec 17, 2022
CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution

CondLaneNet: a Top-to-down Lane Detection Framework Based on Conditional Convolution This is the official implementation code of the paper "CondLaneNe

Alibaba Cloud 311 Dec 30, 2022
VIL-100: A New Dataset and A Baseline Model for Video Instance Lane Detection (ICCV 2021)

Preparation Please see dataset/README.md to get more details about our datasets-VIL100 Please see INSTALL.md to install environment and evaluation too

null 82 Dec 15, 2022
Code for the IJCAI 2021 paper "Structure Guided Lane Detection"

SGNet Project for the IJCAI 2021 paper "Structure Guided Lane Detection" Abstract Recently, lane detection has made great progress with the rapid deve

Jinming Su 27 Dec 8, 2022
Python scripts for performing lane detection using the LSTR model in ONNX

ONNX LSTR Lane Detection Python scripts for performing lane detection using the Lane Shape Prediction with Transformers (LSTR) model in ONNX. Requirem

Ibai Gorordo 29 Aug 30, 2022