ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation

Overview

ENet in Caffe

Alt text

Execution times and hardware requirements

Network 1024x512 1280x720 Parameters Model size (fp32)
ENet 20.4 ms 32.9 ms 0.36 M 1.5 MB
SegNet 66.5 ms 114.3 ms 29.4 M 117.8 MB

A comparison of computational time, number of parameters and model size required for ENet and SegNet. The caffe time command was used to compute time requirement averaged over 100 iterations. Hardware setup: Intel Xeon E5-1620v3, Titan X Pascal with cuDNN v5.1.

Tutorial

For a detailed introduction on how to train and test ENet please see the tutorial.

Publication

This work has been published in arXiv: ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation.

ModelDepot

Also available on ModelDepot.

License

This software is released under a creative commons license which allows for personal and research use only. You can view a license summary here: http://creativecommons.org/licenses/by-nc/4.0/

Comments
  • The difference between running the code in c ++ and python

    The difference between running the code in c ++ and python

    Hi, I tried to run an example on the python and everything turned out fine. But when I run an example on c ++, the result does not match the declared one. original res

    The console displays: Processing time = 361 ms Output_blob (n, c, h, w) = 1, 19, 512, 1024 Processing time on the python is 1800 ms.

    If I use weights with Segnet, then I get a fine result, but the speed has not changed. ressegnet

    opened by sawakk 10
  • ImportError: No module named spatial dropout

    ImportError: No module named spatial dropout

    I0626 19:33:00.715394 10046 net.cpp:100] Creating Layer bn1_0_2 I0626 19:33:00.715397 10046 net.cpp:434] bn1_0_2 <- conv1_0_2 I0626 19:33:00.715414 10046 net.cpp:408] bn1_0_2 -> bn1_0_2 I0626 19:33:00.715584 10046 net.cpp:150] Setting up bn1_0_2 I0626 19:33:00.715590 10046 net.cpp:157] Top shape: 4 64 128 256 (8388608) I0626 19:33:00.715593 10046 net.cpp:165] Memory required for data: 394657792 I0626 19:33:00.715602 10046 layer_factory.hpp:77] Creating layer drop1_0_3 ImportError: No module named spatial_dropout *** Aborted at 1498473180 (unix time) try "date -d @1498473180" if you are using GNU date *** PC: @ 0x7f5b31ba7a95 caffe::Blob<>::~Blob() *** SIGSEGV (@0x7f5b321d02b8) received by PID 10046 (TID 0x7f5b3239ca40) from PID 840762040; stack trace: *** @ 0x7f5b2fc1acb0 (unknown) @ 0x7f5b31ba7a95 caffe::Blob<>::~Blob() @ 0x7f5b31c3fed8 caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x7f5b31c797d4 caffe::DenseImageDataLayer<>::~DenseImageDataLayer() @ 0x7f5b31c79909 caffe::DenseImageDataLayer<>::~DenseImageDataLayer() @ 0x40e226 std::vector<>::~vector() @ 0x7f5b31c1f098 caffe::Net<>::Net() @ 0x7f5b31bb7cea caffe::Solver<>::InitTrainNet() @ 0x7f5b31bb8efc caffe::Solver<>::Init() @ 0x7f5b31bb922a caffe::Solver<>::Solver() @ 0x7f5b31bc2b33 caffe::Creator_AdamSolver<>() @ 0x411ff6 caffe::SolverRegistry<>::CreateSolver() @ 0x40af72 train() @ 0x4089ac main @ 0x7f5b2fc05f45 (unknown) @ 0x4092b3 (unknown) @ 0x0 (unknown) Segmentation fault (core dumped)

    opened by chamcham115 6
  • poor inference results

    poor inference results

    Thank you for publishing your code. But why does the algorithm work much better on the cityscape test set images than on random traffic images from the internet? Is there some preprocessing necessary?

    opened by manuel-88 4
  • Spattial_Dropout error wbhhile training

    Spattial_Dropout error wbhhile training

    Hai,

    During training of an encoder iam met with Import error:no module named spatial_dropout. Kindly suggest me how to solve this issue since iam under job pressure to get some decent results. Kindly do the needful.

    Regards Kumar

    opened by ChidanandKumarKS 4
  • [Q] When I train on ENet, I met the error

    [Q] When I train on ENet, I met the error

    Hello, I have a question.

    When I train, I met the error.


    I0626 10:41:47.785637 4806 layer_factory.hpp:77] Creating layer data I0626 10:41:47.786315 4806 net.cpp:100] Creating Layer data I0626 10:41:47.786324 4806 net.cpp:408] data -> data I0626 10:41:47.786337 4806 net.cpp:408] data -> label I0626 10:41:47.786352 4806 dense_image_data_layer.cpp:50] Opening file dataset/train_fine_cityscapes.txt I0626 10:41:47.788602 4806 dense_image_data_layer.cpp:60] Shuffling data I0626 10:41:47.789371 4806 dense_image_data_layer.cpp:65] A total of 2975 examples. I0626 10:41:47.857427 4806 dense_image_data_layer.cpp:118] output data size top[0]: 3,3,512,1024 I0626 10:41:47.857450 4806 dense_image_data_layer.cpp:121] output data size top[1]: 3,1,64,128 I0626 10:41:47.862567 4806 net.cpp:150] Setting up data I0626 10:41:47.862602 4806 net.cpp:157] Top shape: 3 3 512 1024 (4718592) I0626 10:41:47.862608 4806 net.cpp:157] Top shape: 3 1 64 128 (24576) I0626 10:41:47.862612 4806 net.cpp:165] Memory required for data: 18972672 I0626 10:41:47.862619 4806 layer_factory.hpp:77] Creating layer data_data_0_split I0626 10:41:47.862988 4806 net.cpp:100] Creating Layer data_data_0_split I0626 10:41:47.862996 4806 net.cpp:434] data_data_0_split <- data I0626 10:41:47.863008 4806 net.cpp:408] data_data_0_split -> data_data_0_split_0 I0626 10:41:47.863015 4806 net.cpp:408] data_data_0_split -> data_data_0_split_1 I0626 10:41:47.863376 4806 net.cpp:150] Setting up data_data_0_split I0626 10:41:47.863386 4806 net.cpp:157] Top shape: 3 3 512 1024 (4718592) I0626 10:41:47.863391 4806 net.cpp:157] Top shape: 3 3 512 1024 (4718592) I0626 10:41:47.863394 4806 net.cpp:165] Memory required for data: 56721408 I0626 10:41:47.863415 4806 layer_factory.hpp:77] Creating layer label_data_1_split I0626 10:41:47.863423 4806 net.cpp:100] Creating Layer label_data_1_split I0626 10:41:47.863430 4806 net.cpp:434] label_data_1_split <- label I0626 10:41:47.863436 4806 net.cpp:408] label_data_1_split -> label_data_1_split_0 I0626 10:41:47.863445 4806 net.cpp:408] label_data_1_split -> label_data_1_split_1 I0626 10:41:47.863474 4806 net.cpp:150] Setting up label_data_1_split I0626 10:41:47.863483 4806 net.cpp:157] Top shape: 3 1 64 128 (24576) I0626 10:41:47.863488 4806 net.cpp:157] Top shape: 3 1 64 128 (24576) I0626 10:41:47.863493 4806 net.cpp:165] Memory required for data: 56918016 I0626 10:41:47.863497 4806 layer_factory.hpp:77] Creating layer conv0_1 I0626 10:41:47.863513 4806 net.cpp:100] Creating Layer conv0_1 I0626 10:41:47.863519 4806 net.cpp:434] conv0_1 <- data_data_0_split_0 I0626 10:41:47.863526 4806 net.cpp:408] conv0_1 -> conv0_1 F0626 10:41:48.349544 4806 cudnn.hpp:113] Check failed: status == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM *** Check failure stack trace: *** @ 0x7f0005e46daa (unknown) @ 0x7f0005e46ce4 (unknown) @ 0x7f0005e466e6 (unknown) @ 0x7f0005e49687 (unknown) @ 0x7f00064a3dc6 caffe::CuDNNConvolutionLayer<>::Reshape() @ 0x7f0006433045 caffe::Net<>::Init() @ 0x7f0006433ed5 caffe::Net<>::Net() @ 0x7f00063cccea caffe::Solver<>::InitTrainNet() @ 0x7f00063cdefc caffe::Solver<>::Init() @ 0x7f00063ce22a caffe::Solver<>::Solver() @ 0x7f00063d7b33 caffe::Creator_AdamSolver<>() @ 0x411ff6 caffe::SolverRegistry<>::CreateSolver() @ 0x40af72 train() @ 0x4089ac main @ 0x7f000441af45 (unknown) @ 0x4092b3 (unknown) @ (nil) (unknown) Aborted (core dumped)

    my cudnn version is 5.1.0

    opened by chamcham115 4
  • Really bad performance on TX2

    Really bad performance on TX2

    The ENet paper claims to have 21fps on TX1 at the resolution 480*320 but I am getting 2fps (400ms forward pass time). Is there some kind of optimization that needs to be done.

    opened by IAmSaad 3
  • PythonLayer is not imlemented in Multi-GPU training

    PythonLayer is not imlemented in Multi-GPU training

    Hi, I have try to training using multi-gpu . I got this error

    [python_layer.hpp:25] PythonLayer is not implemented in Multi-GPU training

    So, does any one have experience on change the code spatial_dropout.py to use multi-gpu ?

    opened by An-Pan 3
  • caffe train -solver error: Check failed: bottom[i]->shape() == bottom[0]->shape()

    caffe train -solver error: Check failed: bottom[i]->shape() == bottom[0]->shape()

    Hi Timo,

    Got this when was running this command: /root/ENet/caffe-enet/build/tools/caffe train -solver /root/ENet/prototxts/enet_solver_encoder.prototxt

    I0711 14:14:57.111349 209 net.cpp:100] Creating Layer eltwise2_0_4 I0711 14:14:57.111353 209 net.cpp:434] eltwise2_0_4 <- drop2_0_3 I0711 14:14:57.111371 209 net.cpp:434] eltwise2_0_4 <- bn2_0_4 I0711 14:14:57.111377 209 net.cpp:408] eltwise2_0_4 -> eltwise2_0_4 F0711 14:14:57.111387 209 eltwise_layer.cpp:34] Check failed: bottom[i]->shape() == bottom[0]->shape() *** Check failure stack trace: *** @ 0x7f6f05d78daa (unknown) @ 0x7f6f05d78ce4 (unknown) @ 0x7f6f05d786e6 (unknown) @ 0x7f6f05d7b687 (unknown) @ 0x7f6f0644fbbd caffe::EltwiseLayer<>::Reshape() @ 0x7f6f06317835 caffe::Net<>::Init() @ 0x7f6f063186c5 caffe::Net<>::Net() @ 0x7f6f063296ca caffe::Solver<>::InitTrainNet() @ 0x7f6f0632a8dc caffe::Solver<>::Init() @ 0x7f6f0632ac0a caffe::Solver<>::Solver() @ 0x7f6f062eab23 caffe::Creator_AdamSolver<>() @ 0x411fd6 caffe::SolverRegistry<>::CreateSolver() @ 0x40af52 train() @ 0x40898c main @ 0x7f6f0456ef45 (unknown) @ 0x409293 (unknown) @ (nil) (unknown) Aborted (core dumped)

    What could be wrong?

    opened by SergeMv 2
  • GPU size required to train ENet

    GPU size required to train ENet

    Iam training with NVIDIA Geforce GTX titanZ. While training, i got stuck with check failed : cudaSuccess(2 Vs 0) out of memory.

    Kindly suggest what is the size of GPU needed to train ENet?

    Regards Kumar

    opened by ChidanandKumarKS 2
  • Is it Done?

    Is it Done?

    I0627 03:28:07.408565 10388 solver.cpp:244] Train net output #20: per_class_accuracy = 0.037037 I0627 03:28:07.408572 10388 sgd_solver.cpp:106] Iteration 74980, lr = 5e-06 I0627 03:28:14.539170 10388 solver.cpp:454] Snapshotting to binary proto file /home/chamcham/Dev/ENet/weights/snapshots_encoder/enet_iter_75000.caffemodel I0627 03:28:14.571485 10388 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/chamcham/Dev/ENet/weights/snapshots_encoder/enet_iter_75000.solverstate I0627 03:28:14.706820 10388 solver.cpp:317] Iteration 75000, loss = 0.411978 I0627 03:28:14.706852 10388 solver.cpp:322] Optimization Done. I0627 03:28:14.706856 10388 caffe.cpp:254] Optimization Done. *** Aborted at 1498501694 (unix time) try "date -d @1498501694" if you are using GNU date *** PC: @ 0x7fba3dfb813c (unknown) *** SIGSEGV (@0x0) received by PID 10388 (TID 0x7fba406eda40) from PID 0; stack trace: *** @ 0x7fba3df6bcb0 (unknown) @ 0x7fba3dfb813c (unknown) @ 0x7fba3fef8a87 caffe::Blob<>::~Blob() @ 0x7fba3ff90ecc caffe::BasePrefetchingDataLayer<>::~BasePrefetchingDataLayer() @ 0x7fba3ffca7d4 caffe::DenseImageDataLayer<>::~DenseImageDataLayer() @ 0x7fba3ffca909 caffe::DenseImageDataLayer<>::~DenseImageDataLayer() @ 0x40e226 std::vector<>::~vector() @ 0x40ec10 caffe::Net<>::~Net() @ 0x40edc9 caffe::Net<>::~Net() @ 0x7fba3ff12690 caffe::AdamSolver<>::~AdamSolver() @ 0x40d2fe boost::detail::sp_counted_base::release() @ 0x40b0a7 train() @ 0x4089ac main @ 0x7fba3df56f45 (unknown) @ 0x4092b3 (unknown) @ 0x0 (unknown) Segmentation fault (core dumped)

    opened by chamcham115 2
  • calculate_class_weighting.py

    calculate_class_weighting.py

    Hi I am following your tutorial to train on the Cityscape dataset.

    I am running the following code for the ENet predictions.

    python calculate_class_weighting.py --source ENet/dataset/train_fine_cityscapes.txt --num_classes 19 
    

    but i get:

    Traceback (most recent call last):
      File "calculate_class_weighting.py", line 48, in <module>
        if ((labels == i).sum()) == 0:
    AttributeError: 'bool' object has no attribute 'sum'
    

    thank you

    opened by ThomasLengeling 2
  • Question about the params and FLOPs of ENet

    Question about the params and FLOPs of ENet

    Does anyone reproduce the ENet?Why the params and GFLOPs of my reproduced Network are about 10 and 4 times bigger than the values mentiond in the original paper (Table 3) respectively? My calculated value—— params: 3.5Million,GFLOPs: 16.9

    opened by mrzhouxixi 0
  • TypeError

    TypeError

    python3 create_enet_prototxt.py --source /ENet/dataset/train_fine_cityscapes.txt --mode train_encoder Traceback (most recent call last): File "create_enet_prototxt.py", line 284, in network, prev_layer = bottleneck(n, prev_layer, 1, 0, 64, 'downsampling') # stage, number_bottleneck, num_input, File "create_enet_prototxt.py", line 210, in bottleneck return n.to_proto(), last_layer File "/ENet/caffe-enet/python/caffe/net_spec.py", line 189, in to_proto top._to_proto(layers, names, autonames) File "/ENet/caffe-enet/python/caffe/net_spec.py", line 97, in _to_proto return self.fn._to_proto(layers, names, autonames) File "/ENet/caffe-enet/python/caffe/net_spec.py", line 156, in _to_proto _param_names[self.type_name] + '_param'), k, v) File "/ENet/caffe-enet/python/caffe/net_spec.py", line 79, in assign_proto setattr(proto, name, val) TypeError: 16.0 has type float, but expected one of: int, long

    opened by ghost 1
  • Your example ENet prediction looks wrong

    Your example ENet prediction looks wrong

    Those unlabeled area is red in your example prediction. I checked your implementation, you have 19 output classes for Cityscape, but just do argmax to get the class. How do you deal with unlabeled class? Thanks.

    opened by ethanyanjiali 0
  • use ENet to train Pascal_voc, and the result wasn't convergency.

    use ENet to train Pascal_voc, and the result wasn't convergency.

    Thanks for your works,but when I use this net to train the datasets of Pascal_voc,the result wasn't convergency. What should I do to solve this error?

    opened by aiyanxiao 4
  • object segmentation

    object segmentation

    hi,I started recently dealing with object segmentation and I want to know what are the output points positions of the segmentation operation ### (the points positions of the detector edge) and how i got it.any help??

    opened by hakS07 0
Owner
Timo Sämann
 Timo Sämann
TCNN Temporal convolutional neural network for real-time speech enhancement in the time domain

TCNN Pandey A, Wang D L. TCNN: Temporal convolutional neural network for real-time speech enhancement in the time domain[C]//ICASSP 2019-2019 IEEE Int

凌逆战 16 Dec 30, 2022
Real-Time-Student-Attendence-System - Real Time Student Attendence System

Real-Time-Student-Attendence-System The Student Attendance Management System Pro

Rounak Das 1 Feb 15, 2022
Use tensorflow to implement a Deep Neural Network for real time lane detection

LaneNet-Lane-Detection Use tensorflow to implement a Deep Neural Network for real time lane detection mainly based on the IEEE IV conference paper "To

MaybeShewill-CV 1.9k Jan 8, 2023
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation Official PyTorch Implementation

: We present a novel, real-time, semantic segmentation network in which the encoder both encodes and generates the parameters (weights) of the decoder. Furthermore, to allow maximal adaptivity, the weights at each decoder block vary spatially. For this purpose, we design a new type of hypernetwork, composed of a nested U-Net for drawing higher level context features

Yuval Nirkin 182 Dec 14, 2022
FANet - Real-time Semantic Segmentation with Fast Attention

FANet Real-time Semantic Segmentation with Fast Attention Ping Hu, Federico Perazzi, Fabian Caba Heilbron, Oliver Wang, Zhe Lin, Kate Saenko , Stan Sc

Ping Hu 42 Nov 30, 2022
code for paper "Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?"

Does Unsupervised Architecture Representation Learning Help Neural Architecture Search? Code for paper: Does Unsupervised Architecture Representation

null 39 Dec 17, 2022
Train neural network for semantic segmentation (deep lab V3) with pytorch in less then 50 lines of code

Train neural network for semantic segmentation (deep lab V3) with pytorch in 50 lines of code Train net semantic segmentation net using Trans10K datas

null 17 Dec 19, 2022
Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation)

Recall Loss for Semantic Segmentation (This repo implements the paper: Recall Loss for Semantic Segmentation) Download Synthia dataset The model uses

null 32 Sep 21, 2022
DeepHyper: Scalable Asynchronous Neural Architecture and Hyperparameter Search for Deep Neural Networks

What is DeepHyper? DeepHyper is a software package that uses learning, optimization, and parallel computing to automate the design and development of

DeepHyper Team 214 Jan 8, 2023
A real world application of a Recurrent Neural Network on a binary classification of time series data

What is this This is a real world application of a Recurrent Neural Network on a binary classification of time series data. This project includes data

Josep Maria Salvia Hornos 2 Jan 30, 2022
Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs.

Lunar Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs. About Lunar can be modified to work

Zeyad Mansour 276 Jan 7, 2023
Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021)

Semantic Segmentation for Real Point Cloud Scenes via Bilateral Augmentation and Adaptive Fusion (CVPR 2021) This repository is for BAAF-Net introduce

null 90 Dec 29, 2022
Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network."

R2RNet Official code of "R2RNet: Low-light Image Enhancement via Real-low to Real-normal Network." Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu

null 77 Dec 24, 2022
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
[CVPR 2022] Official code for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved Neural Network Calibration"

MDCA Calibration This is the official PyTorch implementation for the paper: "A Stitch in Time Saves Nine: A Train-Time Regularizing Loss for Improved

MDCA Calibration 21 Dec 22, 2022
[CVPR21] LightTrack: Finding Lightweight Neural Network for Object Tracking via One-Shot Architecture Search

LightTrack: Finding Lightweight Neural Networks for Object Tracking via One-Shot Architecture Search The official implementation of the paper LightTra

Multimedia Research 290 Dec 24, 2022
PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and Multi-Step Knowledge Distillation

PocketNet This is the official repository of the paper: PocketNet: Extreme Lightweight Face Recognition Network using Neural Architecture Search and M

Fadi Boutros 40 Dec 22, 2022
A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries.

Yolo-Powered-Detector A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries

Luke Wilson 1 Dec 3, 2021