Trained on Simulated Data, Tested in the Real World

Related tags

Deep Learning livox_detection_simu
Overview

Livox Detection-simu V1.0: Trained on Simulated Data, Tested in the Real World [Livox Simu-dataset]

Introduction

Livox Detection-simu is a robust and real-time detection package trained on Livox Simu-dataset. It only uses 14k frames of simulated data for training, and performs effective detection in the real world. The inference time is about 50ms on 2080Ti for 200m*100m range detection.
We hope this project can help you make better use of Livox Simu-dataset. In order to improve the performance of the detector, data augmentation such as object insertion and background mix-up is necessary.

Demo

Dependencies

  • python3.6+
  • tensorflow1.13+ (tested on 1.13.0)
  • pybind11
  • ros

Installation

  1. Clone this repository.
  2. Clone pybind11 from pybind11.
$ cd utils/lib_cpp
$ git clone https://github.com/pybind/pybind11.git
  1. Compile C++ module in utils/lib_cpp by running the following command.
$ mkdir build && cd build
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make
  1. copy the lib_cpp.so to root directory:
$ cp lib_cpp.so ../../../
  1. Download the pre_trained model and unzip it to the root directory.

Run

For sequence frame detection

Download the provided rosbags : rosbag and then

$ roscore

$ rviz -d ./config/show.rviz

$ python livox_detection_simu.py

$ rosbag play *.bag -r 0.1

The network inference time is around 25ms, but the point cloud data preprocessing module takes a lot of time based on python. If you want to get a faster real time detection demo, you can modify the point cloud data preprocessing module with c++.

To play with your own rosbag, please change your rosbag topic to /livox/lidar.

Support

You can get support from Livox with the following methods :

  • Send email to [email protected] with a clear description of your problem and your setup
  • Report issue on github
Comments
  • Gazebo plugin or the mathmatical formula of the pattern?

    Gazebo plugin or the mathmatical formula of the pattern?

    Hi, I am trying to apply Livox Horizon in autonomous exploraion task. However there is neither the gazebo plug-in for Livox horizon nor the principle of the scanning pattern of the horizon. Any ideas? So I may try to implement a gazebo plug-in myself.

    opened by EpsAvlc 4
  • Tensorflow Error

    Tensorflow Error

    My environment

    • ubuntu 18.04 LTS
    • python 3.6.9
    • tensorflow 1.15

    I got the following error when I ran "python livox_detection_simu.py"

    suzuki@ubuntu:~/Downloads/livox_detection_simu$ python3 livox_detection_simu.py WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

    • https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
    • https://github.com/tensorflow/addons
    • https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

    1008 224 30 WARNING:tensorflow:From /home/suzuki/Downloads/livox_detection_simu/networks/model.py:27: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

    (1, 1008, 224, 30) WARNING:tensorflow:From /home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. WARNING:tensorflow:From /home/suzuki/Downloads/livox_detection_simu/networks/model.py:70: The name tf.image.resize_bilinear is deprecated. Please use tf.compat.v1.image.resize_bilinear instead.

    WARNING:tensorflow:From livox_detection_simu.py:66: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

    WARNING:tensorflow:From livox_detection_simu.py:67: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    WARNING:tensorflow:From livox_detection_simu.py:71: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

    2021-04-28 01:42:50.203964: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-04-28 01:42:50.225177: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592005000 Hz 2021-04-28 01:42:50.226874: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4db39c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-04-28 01:42:50.226934: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-04-28 01:42:50.231211: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/suzuki/catkin_ws/devel/lib:/home/suzuki/Downloads/ws_livox/devel/lib:/opt/ros/melodic/lib 2021-04-28 01:42:50.231847: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303) 2021-04-28 01:42:50.231894: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ubuntu): /proc/driver/nvidia/version does not exist Traceback (most recent call last): File "livox_detection_simu.py", line 320, in livox = Detector() File "livox_detection_simu.py", line 72, in init saver.restore(self.sess, cfg.MODEL_PATH) File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1280, in restore if not checkpoint_management.checkpoint_exists_internal(checkpoint_prefix): File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_management.py", line 366, in checkpoint_exists_internal if file_io.get_matching_files(pathname): File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py", line 363, in get_matching_files return get_matching_files_v2(filename) File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py", line 384, in get_matching_files_v2 compat.as_bytes(pattern)) tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory

    How do I get it to work?

    Thank you for your answer

    opened by suzuki3366 3
  • “python livox_detection_simu.py” error

    “python livox_detection_simu.py” error

    Traceback (most recent call last): File "livox_detection_simu.py", line 7, in import lib_cpp ImportError: /home/xjllhy/Documents/livox_detection_simu/lib_cpp.so: undefined symbol: PyIndex_Check

    opened by xjllhy 0
  • Is it a bug or typo in libcpp cal_result()

    Is it a bug or typo in libcpp cal_result()

    Hi ,

    lib_cpp.cpp: https://github.com/Livox-SDK/livox_detection_simu/blob/master/utils/lib_cpp/lib_cpp.cpp#L398

     if(is_obj>obj_th &&0)

    It is weird "&& 0 ".

    But in livox_detection repo, it is normal: if (is_obj > obj_th ).

    opened by LHHCAL 0
  • tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory

    tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory

    Hi guys, i get a trouble such as "tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory" when I try to run "python3 livox_detection_simu.py" , Is there anybody met this question? could you tell me how to solve it? THANKS!

    opened by applededipan 1
  • Can't detect pedestrian

    Can't detect pedestrian

    Hi Livox guys, I am using livox detection sim to detect vehicle and pedestrian. But I found that using the pre-trained module you provided, it can not detect pedestrian. I also tried the other project "livox_detection" and its pre-trained model with the same bag file I used. It can detect pedestrian.But for vehicle detection, it is worse than livox_detection_sim. So I want to ask why?

    opened by blueskych 2
  • How do I pre-train myself?

    How do I pre-train myself?

    Livox - SDK, hello! How do you reuse livox_simu for pre-training?I want to test the AP and APH of the detection , so I want to repretrain, get a pretrained model. Looking forward to your reply!

    opened by clytze0216 2
Owner
livox
https://www.livoxtech.com/
livox
GitHub
Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data

Real-ESRGAN Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data Ported from https://github.com/xinntao/Real-ESRGAN Depend

Holy Wu 44 Dec 27, 2022
A complete end-to-end demonstration in which we collect training data in Unity and use that data to train a deep neural network to predict the pose of a cube. This model is then deployed in a simulated robotic pick-and-place task.

Object Pose Estimation Demo This tutorial will go through the steps necessary to perform pose estimation with a UR3 robotic arm in Unity. You’ll gain

Unity Technologies 187 Dec 24, 2022
A framework for analyzing computer vision models with simulated data

3DB: A framework for analyzing computer vision models with simulated data Paper Quickstart guide Blog post Installation Follow instructions on: https:

3DB 112 Jan 1, 2023
In this project we investigate the performance of the SetCon model on realistic video footage. Therefore, we implemented the model in PyTorch and tested the model on two example videos.

Contrastive Learning of Object Representations Supervisor: Prof. Dr. Gemma Roig Institutions: Goethe University CVAI - Computational Vision & Artifici

Dirk Neuhäuser 6 Dec 8, 2022
This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models are Pix2Pix, Pix2PixHD, CycleGAN and PointWise.

RGB2NIR_Experimental This repository contains several image-to-image translation models, whcih were tested for RGB to NIR image generation. The models

null 5 Jan 4, 2023
A toolkit for making real world machine learning and data analysis applications in C++

dlib C++ library Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real worl

Davis E. King 11.6k Jan 1, 2023
A real world application of a Recurrent Neural Network on a binary classification of time series data

What is this This is a real world application of a Recurrent Neural Network on a binary classification of time series data. This project includes data

Josep Maria Salvia Hornos 2 Jan 30, 2022
Annotate datasets with a semi-trained or fully trained YOLOv5 model

YOLOv5 Auto Annotator Annotate datasets with a semi-trained or fully trained YOLOv5 model Prerequisites Ubuntu >=20.04 Python >=3.7 System dependencie

Akash James 3 May 14, 2022
A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution.

Awesome Pretrained StyleGAN2 A collection of pre-trained StyleGAN2 models trained on different datasets at different resolution. Note the readme is a

Justin 1.1k Dec 24, 2022
Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

郭飞 3.7k Jan 3, 2023
A 35mm camera, based on the Canonet G-III QL17 rangefinder, simulated in Python.

c is for Camera A 35mm camera, based on the Canonet G-III QL17 rangefinder, simulated in Python. The purpose of this project is to explore and underst

Daniele Procida 146 Sep 26, 2022
"Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback"

This is code repo for our EMNLP 2017 paper "Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback", which implements the A2C algorithm on top of a neural encoder-decoder model and benchmarks the combination under simulated noisy rewards.

Khanh Nguyen 131 Oct 21, 2022
Simulated garment dataset for virtual try-on

Simulated garment dataset for virtual try-on This repository contains the dataset used in the following papers: Self-Supervised Collision Handling via

null 33 Dec 20, 2022
PINN Burgers - 1D Burgers equation simulated by PINN

PINN(s): Physics-Informed Neural Network(s) for Burgers equation This is an impl

ShotaDEGUCHI 1 Feb 12, 2022
Real-world Anomaly Detection in Surveillance Videos- pytorch Re-implementation

Real world Anomaly Detection in Surveillance Videos : Pytorch RE-Implementation This repository is a re-implementation of "Real-world Anomaly Detectio

seominseok 62 Dec 8, 2022
The first dataset on shadow generation for the foreground object in real-world scenes.

Object-Shadow-Generation-Dataset-DESOBA Object Shadow Generation is to deal with the shadow inconsistency between the foreground object and the backgr

BCMI 105 Dec 30, 2022
HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021)

Code for HDR Video Reconstruction HDR Video Reconstruction: A Coarse-to-fine Network and A Real-world Benchmark Dataset (ICCV 2021) Guanying Chen, Cha

Guanying Chen 64 Nov 19, 2022
Learning Generative Models of Textured 3D Meshes from Real-World Images, ICCV 2021

Learning Generative Models of Textured 3D Meshes from Real-World Images This is the reference implementation of "Learning Generative Models of Texture

Dario Pavllo 115 Jan 7, 2023
Repo for FUZE project. I will also publish some Linux kernel LPE exploits for various real world kernel vulnerabilities here. the samples are uploaded for education purposes for red and blue teams.

Linux_kernel_exploits Some Linux kernel exploits for various real world kernel vulnerabilities here. More exploits are yet to come. This repo contains

Wei Wu 472 Dec 21, 2022