Trained on Simulated Data, Tested in the Real World

Related tags

livox_detection_simu
Overview

Livox Detection-simu V1.0: Trained on Simulated Data, Tested in the Real World [Livox Simu-dataset]

Introduction

Livox Detection-simu is a robust and real-time detection package trained on Livox Simu-dataset. It only uses 14k frames of simulated data for training, and performs effective detection in the real world. The inference time is about 50ms on 2080Ti for 200m*100m range detection.
We hope this project can help you make better use of Livox Simu-dataset. In order to improve the performance of the detector, data augmentation such as object insertion and background mix-up is necessary.

Demo

Dependencies

  • python3.6+
  • tensorflow1.13+ (tested on 1.13.0)
  • pybind11
  • ros

Installation

  1. Clone this repository.
  2. Clone pybind11 from pybind11.
$ cd utils/lib_cpp
$ git clone https://github.com/pybind/pybind11.git
  1. Compile C++ module in utils/lib_cpp by running the following command.
$ mkdir build && cd build
$ cmake -DCMAKE_BUILD_TYPE=Release ..
$ make
  1. copy the lib_cpp.so to root directory:
$ cp lib_cpp.so ../../../
  1. Download the pre_trained model and unzip it to the root directory.

Run

For sequence frame detection

Download the provided rosbags : rosbag and then

$ roscore

$ rviz -d ./config/show.rviz

$ python livox_detection_simu.py

$ rosbag play *.bag -r 0.1

The network inference time is around 25ms, but the point cloud data preprocessing module takes a lot of time based on python. If you want to get a faster real time detection demo, you can modify the point cloud data preprocessing module with c++.

To play with your own rosbag, please change your rosbag topic to /livox/lidar.

Support

You can get support from Livox with the following methods :

  • Send email to [email protected] with a clear description of your problem and your setup
  • Report issue on github
Issues
  • Gazebo plugin or the mathmatical formula of the pattern?

    Gazebo plugin or the mathmatical formula of the pattern?

    Hi, I am trying to apply Livox Horizon in autonomous exploraion task. However there is neither the gazebo plug-in for Livox horizon nor the principle of the scanning pattern of the horizon. Any ideas? So I may try to implement a gazebo plug-in myself.

    opened by EpsAvlc 4
  • Tensorflow Error

    Tensorflow Error

    My environment

    • ubuntu 18.04 LTS
    • python 3.6.9
    • tensorflow 1.15

    I got the following error when I ran "python livox_detection_simu.py"

    [email protected]:~/Downloads/livox_detection_simu$ python3 livox_detection_simu.py WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see:

    • https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
    • https://github.com/tensorflow/addons
    • https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue.

    1008 224 30 WARNING:tensorflow:From /home/suzuki/Downloads/livox_detection_simu/networks/model.py:27: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

    (1, 1008, 224, 30) WARNING:tensorflow:From /home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1057: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: Please use layer.__call__ method instead. WARNING:tensorflow:From /home/suzuki/Downloads/livox_detection_simu/networks/model.py:70: The name tf.image.resize_bilinear is deprecated. Please use tf.compat.v1.image.resize_bilinear instead.

    WARNING:tensorflow:From livox_detection_simu.py:66: The name tf.train.Saver is deprecated. Please use tf.compat.v1.train.Saver instead.

    WARNING:tensorflow:From livox_detection_simu.py:67: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

    WARNING:tensorflow:From livox_detection_simu.py:71: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

    2021-04-28 01:42:50.203964: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA 2021-04-28 01:42:50.225177: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2592005000 Hz 2021-04-28 01:42:50.226874: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x4db39c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2021-04-28 01:42:50.226934: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2021-04-28 01:42:50.231211: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/suzuki/catkin_ws/devel/lib:/home/suzuki/Downloads/ws_livox/devel/lib:/opt/ros/melodic/lib 2021-04-28 01:42:50.231847: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: UNKNOWN ERROR (303) 2021-04-28 01:42:50.231894: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (ubuntu): /proc/driver/nvidia/version does not exist Traceback (most recent call last): File "livox_detection_simu.py", line 320, in livox = Detector() File "livox_detection_simu.py", line 72, in init saver.restore(self.sess, cfg.MODEL_PATH) File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/training/saver.py", line 1280, in restore if not checkpoint_management.checkpoint_exists_internal(checkpoint_prefix): File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/training/checkpoint_management.py", line 366, in checkpoint_exists_internal if file_io.get_matching_files(pathname): File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py", line 363, in get_matching_files return get_matching_files_v2(filename) File "/home/suzuki/venv/lib/python3.6/site-packages/tensorflow_core/python/lib/io/file_io.py", line 384, in get_matching_files_v2 compat.as_bytes(pattern)) tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory

    How do I get it to work?

    Thank you for your answer

    opened by suzuki3366 3
  • tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory

    tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory

    Hi guys, i get a trouble such as "tensorflow.python.framework.errors_impl.NotFoundError: model; No such file or directory" when I try to run "python3 livox_detection_simu.py" , Is there anybody met this question? could you tell me how to solve it? THANKS!

    opened by applededipan 1
  • Can't detect pedestrian

    Can't detect pedestrian

    Hi Livox guys, I am using livox detection sim to detect vehicle and pedestrian. But I found that using the pre-trained module you provided, it can not detect pedestrian. I also tried the other project "livox_detection" and its pre-trained model with the same bag file I used. It can detect pedestrian.But for vehicle detection, it is worse than livox_detection_sim. So I want to ask why?

    opened by blueskych 2
  • How do I pre-train myself?

    How do I pre-train myself?

    Livox - SDK, hello! How do you reuse livox_simu for pre-training?I want to test the AP and APH of the detection , so I want to repretrain, get a pretrained model. Looking forward to your reply!

    opened by clytze0216 2
  • error:module'lib_cpp' has no arrtibute 'cal_result’

    error:module'lib_cpp' has no arrtibute 'cal_result’

    hello!I met an error :module'lib_cpp' has no arrtibute 'cal_result’ how can I sovle it? thanks a lot! looking forward to your reply!

    opened by clytze0216 1
  • tracking id

    tracking id

    Hello, how to use this to get each car's track ID, like the video in this URL?https://www.bilibili.com/video/av374079324/

    opened by Lowbeee 2
  • inference time

    inference time

    Hi. I follow this repo

    $ roscore
    
    $ rviz -d ./config/show.rviz
    
    $ python livox_detection_simu.py
    
    $ rosbag play *.bag -r 0.1
    

    and my command window shows like this:

    det_time(ms) 47.05095291137695
    det_numbers 10
    det_time(ms) 43.282270431518555
    det_numbers 11
    det_time(ms) 45.348167419433594
    det_numbers 10
    det_time(ms) 42.76132583618164
    det_numbers 12
    det_time(ms) 45.06182670593262
    det_numbers 11
    det_time(ms) 44.640302658081055
    det_numbers 10
    det_time(ms) 43.58649253845215
    det_numbers 10
    det_time(ms) 43.75743865966797
    det_numbers 11
    
    

    Dose it mean that my PC(Nvidia 1060Ti) can run this about 40 ms / per frame when I buy a livox? I also watch my GPU and nothing to be used when I run this demo. Dose it mean this demo do not use GPU? I have installed TF1.14_gpu. Many thanks!

    opened by zhucheng725 8
Owner
livox
https://www.livoxtech.com/
livox
🎓Automatically Update CV Papers Daily using Github Actions (Update at 12:00 UTC Every Day)

??Automatically Update CV Papers Daily using Github Actions (Update at 12:00 UTC Every Day)

Realcat 4 Oct 21, 2021
MaRS - a recursive filtering framework that allows for truly modular multi-sensor integration

The Modular and Robust State-Estimation Framework, or short, MaRS, is a recursive filtering framework that allows for truly modular multi-sensor integration

Control of Networked Systems - University of Klagenfurt 97 Oct 20, 2021
DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing

DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing Figure: Joint multi-attribute edits using DyStyle model. Great diversity

null 63 Oct 12, 2021
S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration (CVPR 2021)

S2-BNN (Self-supervised Binary Neural Networks Using Distillation Loss) This is the official pytorch implementation of our paper: "S2-BNN: Bridging th

Zhiqiang Shen 28 Oct 15, 2021
Dynamical Wasserstein Barycenters for Time Series Modeling

Dynamical Wasserstein Barycenters for Time Series Modeling This is the code related for the Dynamical Wasserstein Barycenter model published in Neurip

null 2 Oct 15, 2021
SwinIR: Image Restoration Using Swin Transformer

SwinIR: Image Restoration Using Swin Transformer This repository is the official PyTorch implementation of SwinIR: Image Restoration Using Shifted Win

Jingyun Liang 571 Oct 22, 2021
Spatial Intention Maps for Multi-Agent Mobile Manipulation (ICRA 2021)

spatial-intention-maps This code release accompanies the following paper: Spatial Intention Maps for Multi-Agent Mobile Manipulation Jimmy Wu, Xingyua

Jimmy Wu 47 Sep 18, 2021
PyTorch implementations of Generative Adversarial Networks.

This repository has gone stale as I unfortunately do not have the time to maintain it anymore. If you would like to continue the development of it as

Erik Linder-Norén 10.4k Oct 22, 2021
Official codebase for Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World

Legged Robots that Keep on Learning Official codebase for Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World, whic

Laura Smith 28 Oct 20, 2021
Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Optimization Algorithm,Immune Algorithm, Artificial Fish Swarm Algorithm, Differential Evolution and TSP(Traveling salesman)

scikit-opt Swarm Intelligence in Python (Genetic Algorithm, Particle Swarm Optimization, Simulated Annealing, Ant Colony Algorithm, Immune Algorithm,A

郭飞 2.6k Oct 22, 2021
Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021) Jiaxi Jiang, Kai Zhang, Radu Timofte Computer Vision Lab, ETH Zurich, Switzerland ??

Jiaxi Jiang 80 Sep 30, 2021
Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Towards Flexible Blind JPEG Artifacts Removal (FBCNN, ICCV 2021)

Jiaxi Jiang 114 Oct 20, 2021
CLIPort: What and Where Pathways for Robotic Manipulation

CLIPort CLIPort: What and Where Pathways for Robotic Manipulation Mohit Shridhar, Lucas Manuelli, Dieter Fox CoRL 2021 CLIPort is an end-to-end imitat

null 84 Oct 15, 2021
Pseudo-Visual Speech Denoising

Pseudo-Visual Speech Denoising This code is for our paper titled: Visual Speech Enhancement Without A Real Visual Stream published at WACV 2021. Autho

Sindhu 69 Oct 12, 2021
An easier way to build neural search on the cloud

An easier way to build neural search on the cloud Jina is a deep learning-powered search framework for building cross-/multi-modal search systems (e.g

Jina AI 11.8k Oct 26, 2021
This is the research repository for Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition.

Vid2Doppler: Synthesizing Doppler Radar Data from Videos for Training Privacy-Preserving Activity Recognition This is the research repository for Vid2

Future Interfaces Group (CMU) 19 Oct 4, 2021
robomimic: A Modular Framework for Robot Learning from Demonstration

robomimic [Homepage]   [Documentation]   [Study Paper]   [Study Website]   [ARISE Initiative] Latest Updates [08/09/2021] v0.1.0: Initial code and pap

ARISE Initiative 59 Oct 18, 2021
A curated (most recent) list of resources for Learning with Noisy Labels

A curated (most recent) list of resources for Learning with Noisy Labels

Jiaheng Wei 99 Oct 17, 2021
《LXMERT: Learning Cross-Modality Encoder Representations from Transformers》(EMNLP 2020)

The Most Important Thing. Our code is developed based on: LXMERT: Learning Cross-Modality Encoder Representations from Transformers

null 37 Oct 13, 2021