Real-time object detection on Android using the YOLO network with TensorFlow

Last update: Jan 3, 2023

Overview

TensorFlow YOLO object detection on Android

android-yolo is the first implementation of YOLO for TensorFlow on an Android device. It is compatible with Android Studio and usable out of the box. It can detect the 20 classes of objects in the Pascal VOC dataset: aeroplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, sofa, train and tv/monitor. The network only outputs one predicted bounding box at a time for now. The code can and will be extended in the future to output several predictions.

To use this demo first clone the repository. Download the TensorFlow YOLO model and put it in android-yolo/app/src/main/assets. Then open the project on Android Studio. Once the project is open you can run the project on your Android device using the Run 'app' command and selecting your device.

NEW: The standalone APK has been released and you can find it here. Just open your browser on your Android device and download the APK file. When the file has been downloaded it should begin installing on your device after you grant the required permissions.

GPUs are not currently supported by TensorFlow on Android. If you have a decent Android device you will have around two frames per second of processed images.

Here is a video showing a small demo of the app.

Nataniel Ruiz
School of Interactive Computing
Georgia Institute of Technology

Credits: App launch icon made by Freepik from Flaticon is licensed by Creative Commons BY 3.0.

Disclaimer: The app is hardcoded for 20 classes and for the tiny-yolo network final output layer. You can check the following code if you want to change this:

https://github.com/natanielruiz/android-yolo/blob/master/app/src/main/java/org/tensorflow/demo/TensorflowClassifier.java

The code describes the interpretation of the output.

The code for the network inference pass is written in C++ and the output is passed to Java. The output of the network is in the form of a String which is converted to a StringTokenizer and is then converted into an array of Floats in line 87 of TensorflowClassifier.java

You can work from there and read the papers to transform the new yolo model output into something that makes sense. (I did it only for one bounding box and also obtained the confidence of this bounding box). This part of the code is commented by me so you can understand what I did. Also read the paper here: https://arxiv.org/abs/1506.02640

Comments

jni error

Build fingerprint: 'Xiaomi/libra/libra:5.1.1/LMY47V/5.11.1:user/release-keys' Revision: '0' ABI: 'arm' pid: 10250, tid: 10273, name: ImageListener >>> org.tensorflow.tensorflowdemo <<< signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr -------- Abort message: 'jni_utils.cc:92 'asset' Must be non NULL' r0 00000000 r1 00002821 r2 00000006 r3 00000000 r4 e29c4dd8 r5 00000006 r6 00000000 r7 0000010c r8 e29c3ad4 r9 e29c39d8 sl 00000000 fp 00000000 ip 00002821 sp e29c3980 lr f7072e89 pc f709972c cpsr 600f0010

backtrace: #00 pc 0003a72c /system/lib/libc.so (tgkill+12) #01 pc 00013e85 /system/lib/libc.so (pthread_kill+52) #02 pc 00014aa3 /system/lib/libc.so (raise+10) #03 pc 0001131d /system/lib/libc.so (__libc_android_abort+36) #04 pc 0000f54c /system/lib/libc.so (abort+4) #05 pc 0053d03c /data/app/org.tensorflow.tensorflowdemo-1/lib/arm/libtensorflow_demo.so #06 pc 0053d1bc /data/app/org.tensorflow.tensorflowdemo-1/lib/arm/libtensorflow_demo.so #07 pc 0053d1d8 /data/app/org.tensorflow.tensorflowdemo-1/lib/arm/libtensorflow_demo.so #08 pc 000746b4 /data/app/org.tensorflow.tensorflowdemo-1/lib/arm/libtensorflow_demo.so #09 pc 00075594 /data/app/org.tensorflow.tensorflowdemo-1/lib/arm/libtensorflow_demo.so (ReadFileToProto(AAssetManager*, char const*, google::protobuf::MessageLite*)+1144) #10 pc 00077bdc /data/app/org.tensorflow.tensorflowdemo-1/lib/arm/libtensorflow_demo.so (Java_org_tensorflow_demo_TensorFlowClassifier_initializeTensorFlow+1084) #11 pc 000a2511 /system/lib/libart.so (art_quick_generic_jni_trampoline+32) #12 pc 00136c1c /dev/ashmem/dalvik-main space (deleted)

opened by haophancs 11
Training new model

Very usefully app, I want to include my own training model to the app to be able to detect different objects or features. Any suggestion where to start?, thank you!

opened by ThomasLengeling 8
How to make ".pb" file

Hello! Thank you for your nice implementation.

I would like to know how to make ".pb" file form output of darknet. Did you use this ( https://github.com/thtrieu/darkflow ) ?

I am really glad if you help me!

opened by LittleWat 5
using a different pb file

Hi,

I believe the pb file you derived is for yolo-voc, however, I would like to have a pb file for YOLOv2 608x608 ( https://pjreddie.com/darknet/yolo/).

Use darkflow (https://github.com/thtrieu/darkflow) I was able to create a pb file using yolo.cfg and yolo.weights.

I copied the yolo.pb file into the assets folder and when I try to install the apk, I get the following error:

03-31 16:08:26.398 19432-19652/org.tensorflow.tensorflowdemo I/tensorflow: CameraConnectionFragment: Opening camera preview: 480x640 03-31 16:08:26.402 19432-19432/org.tensorflow.tensorflowdemo V/BoundingBox: width: 1080.0 height: 1794.0 03-31 16:08:26.415 19432-19652/org.tensorflow.tensorflowdemo I/tensorflow: CameraConnectionFragment: Getting assets. 03-31 16:08:26.415 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:115 Loading TensorFlow. 03-31 16:08:26.415 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:117 Making new SessionOptions. 03-31 16:08:26.415 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:120 Got config, 0 devices 03-31 16:08:26.417 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:123 Session created. 03-31 16:08:26.417 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:126 Graph created. 03-31 16:08:26.417 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:130 Acquired AssetManager. 03-31 16:08:26.417 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:132 Reading file to proto: file:///android_asset/yolo.pb 03-31 16:08:26.797 19432-19652/org.tensorflow.tensorflowdemo I/native: stat_summarizer.cc:33 StatSummarizer found 349 nodes 03-31 16:08:26.798 19432-19652/org.tensorflow.tensorflowdemo I/native: tensorflow_jni.cc:137 Creating session. 03-31 16:08:26.827 19432-19652/org.tensorflow.tensorflowdemo A/native: tensorflow_jni.cc:140 Could not create TensorFlow Graph: Not found: Op type not registered 'RealDiv' 03-31 16:08:26.834 19432-19652/org.tensorflow.tensorflowdemo A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 19652 (ImageListener) 03-31 16:08:26.834 368: 368 W/ ] debuggerd: handling request: pid=19432 uid=10094 gid=10094 tid=19652

opened by ramarajan09 4
To customize it for other trained model files

@hashimshafiq @natanielruiz Hi can u pls guide what all parameters to be changed if we change the model file trained for custom object detection. I have converted my model file using darkflow to .pb format i have to modify the current android yolo to detect other objects

opened by abhigoku10 3
ndk-build on OS X

Hi @natanielruiz - thanks for the library. Building on OS X 10.11.6 (El Capitan) using latest NDK (14.0.3770861) gives the following during ndk-build -

[armeabi-v7a] Compile++ arm : tensorflow_demo <= tensorflow_jni.cc jni/./tensorflow_jni.cc: In function 'std::string ClassifyImage(const {anonymous}::RGBA*)': jni/./tensorflow_jni.cc:318:33: error: 'output' cannot be used as a function const float value = output(i); ^ jni/./tensorflow_jni.cc:319:16: error: base operand of '->' has non-pointer type 'std::vector' top_results->push_back(value); ^ jni/./tensorflow_jni.cc:321:27: error: base operand of '->' has non-pointer type 'std::vector' std::reverse(top_results->begin(), top_results->end()); ^ jni/./tensorflow_jni.cc:321:49: error: base operand of '->' has non-pointer type 'std::vector' std::reverse(top_results->begin(), top_results->end()); ^ jni/./tensorflow_jni.cc: At global scope: jni/./tensorflow_jni.cc:171:13: warning: 'void GetTopN(const Eigen::TensorMap<Eigen::Tensor<float, 1, 1, int>, 16>&, int, float, std::vector<std::pair<float, int> >*)' defined but not used [-Wunused-function] static void GetTopN( ^ make[1]: *** [obj/local/armeabi-v7a/objs/tensorflow_demo/./tensorflow_jni.o] Error 1

output is a pointer to a tensor not a function - not sure how this compiles?

opened by victorv 2
Replacing the model and label strings for google inception resnet v2

Hi,

Thanks for your good work. Can you guide me, I want to replace the default MODEL_FILE with inception resnet v2. It would be really helpful for me if you provide detailed answer. Thanks

opened by hannan-sultan 2
crash when change android_graph.pb

Hello, this project is very good when i test, but when I use darkflow tool to convert my weight and cfg file to new android_graph.pb, then replace old one, and change the class_labels[] in java file, then got new apk, but test got crash message as below, Is it anything wrong in my step? anybody can helo me?

... I/native: stat_summarizer.cc:33 StatSummarizer found 349 nodes tensorflow_jni.cc:137 Creating session. A/native: tensorflow_jni.cc:140 Could not create TensorFlow Graph: Not found: Op type not registered 'RealDiv' A/libc: Fatal signal 6 (SIGABRT), code -6 in tid 3109 (ImageListener) Application terminated.

opened by xhzzhang 1
Fatal signal 6 (SIGABRT)

I am getting: "Android YOLO keeps stopping"

jni_utils.cc:92 'asset' Must be non NULL

Fatal signal 6 (SIGABRT), code -6 in tid 9860 (ImageListener) [ 05-03 22:49:05.003 604: 604 W/ ] debuggerd: handling request: pid=9839 uid=10357 gid=10357 tid=9860

opened by appmaker123 1
Object detection with custom pb file

Hello sir, I had custom pb file which I trained using tiny-yolo-voc with Darknet and then convert it using darkflow command ./flow --model cfg/tiny-yolo-voc.cfg --load bin/tiny-yolo-voc.weights --savepb --verbalise My dataset is very simple which has only 1 class (cat) so the cfg I am using is changed at the final numer of filter `[convolutional] size=1 stride=1 pad=1 filters=30 activation=linear

[region] anchors = 1.08,1.19, 3.42,4.41, 6.63,11.38, 9.42,5.11, 16.62,10.52 bias_match=1 classes=1 coords=4 num=5 softmax=1 jitter=.2 rescore=1` As you said from the readme file, you hardcoded your code with voc dataset with 20 classes. So i would like to ask just in my case when I only have 1 class then what changes should I made or would you give brief explanation to help me ? Thank you so much !

opened by phongnhhn92 1
to add additional labels
@natanielruiz i thanks for sharing the implementation ,wonderful work . I have couple of questions

Where can add/delete the labels

Can i use weights trained on darknet having .weight file format
opened by abhigoku10 1
CVE-2007-4559 Patch

Patching CVE-2007-4559

Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

opened by TrellixVulnTeam 0

showing the error unsatisfied link error

java.lang.UnsatisfiedLinkError: dalvik.system.PathClassLoader[DexPathList[[zip file "/system/framework/org.apache.http.legacy.boot.jar", zip file "/data/app/org.tensorflow.tensorflowdemo-FebBpWvV3iIxT9KrXSaTRw==/base.apk"],nativeLibraryDirectories=[/data/app/org.tensorflow.tensorflowdemo-FebBpWvV3iIxT9KrXSaTRw==/lib/arm64, /system/lib64]]] couldn't find "libtensorflow_demo.so"

opened by kittusster 0

Can I adapt this project to yolo v3?

Since I noticed that your project used .pb file for graph, how you get that? We tried several projects, and got errors like Op type not registered 'LeakyRelu' in binary running on xxx. Make sure the Op and Kernel are registered in the binary running in this process. How you get a proper .pb file?

opened by locoda 1
detect custom object and notify

What is the output of the model? How can I capture the output for further processing? Can it be made to detect custom objects and send notification emails?

opened by nahcyrret 0
Can't rebuild NDK because of errors in tensorflow_jni.cc

[armeabi-v7a] Compile++ arm : tensorflow_demo <= tensorflow_jni.cc jni/./tensorflow_jni.cc: In function 'std::string ClassifyImage(const {anonymous}::RGBA*)': jni/./tensorflow_jni.cc:319:33: error: 'output' cannot be used as a function const float value = output(i); ^ jni/./tensorflow_jni.cc:320:16: error: base operand of '->' has non-pointer type 'std::vector' top_results->push_back(value); ^ jni/./tensorflow_jni.cc:322:27: error: base operand of '->' has non-pointer type 'std::vector' std::reverse(top_results->begin(), top_results->end()); ^ jni/./tensorflow_jni.cc:322:49: error: base operand of '->' has non-pointer type 'std::vector' std::reverse(top_results->begin(), top_results->end()); ^ jni/./tensorflow_jni.cc: At global scope: jni/./tensorflow_jni.cc:172:13: warning: 'void GetTopN(const Eigen::TensorMap<Eigen::Tensor<float, 1, 1, int>, 16>&, int, float, std::vector<std::pair<float, int>

*)' defined but not used [-Wunused-function] static void GetTopN( ^ make: *** [obj/local/armeabi-v7a/objs/tensorflow_demo/./tensorflow_jni.o] Error 1

opened by haophancs 0

Owner

Nataniel Ruiz

PhD candidate at Boston University doing Computer Vision and ML. M.S. from Georgia Tech, BA/M.S. from Ecole Polytechnique

GitHub

Implementation for the paper 'YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs'

YOLO-ReT This is the original implementation of the paper: YOLO-ReT: Towards High Accuracy Real-time Object Detection on Edge GPUs. Prakhar Ganesh, Ya

69 Oct 19, 2022

Real-time multi-object tracker using YOLO v5 and deep sort

This repository contains a two-stage-tracker. The detections generated by YOLOv5, a family of object detection architectures and models pretrained on the COCO dataset, are passed to a Deep Sort algorithm which tracks the objects. It can track any object that your Yolov5 model was trained to detect.

3.6k Jan 5, 2023

LF-YOLO (Lighter and Faster YOLO) is used to detect defect of X-ray weld image.

This project is based on ultralytics/yolov3. LF-YOLO (Lighter and Faster YOLO) is used to detect defect of X-ray weld image. Download $ git clone http

26 Dec 13, 2022

Yolo ros - YOLO-ROS for HUAWEI ATLAS200

YOLO-ROS YOLO-ROS for NVIDIA YOLO-ROS for HUAWEI ATLAS200, please checkout for b

5 Oct 18, 2022

A tutorial showing how to train, convert, and run TensorFlow Lite object detection models on Android devices, the Raspberry Pi, and more!

1.3k Jan 2, 2023

Object tracking and object detection is applied to track golf puts in real time and display stats/games.

Putting_Game Object tracking and object detection is applied to track golf puts in real time and display stats/games. Works best with the Perfect Prac

1 Dec 29, 2021

Object detection using yolo-tiny model and opencv used as backend

Object detection Algorithm used : Yolo algorithm Backend : opencv Library required: opencv = 4.5.4-dev' Quick Overview about structure 1) main.py Load

2 Jul 6, 2022

Face and other object detection using OpenCV and ML Yolo

Object-and-Face-Detection-Using-Yolo- Opencv and YOLO object and face detection is implemented. You only look once (YOLO) is a state-of-the-art, real-

3 Feb 15, 2022

Use tensorflow to implement a Deep Neural Network for real time lane detection

LaneNet-Lane-Detection Use tensorflow to implement a Deep Neural Network for real time lane detection mainly based on the IEEE IV conference paper "To

1.9k Jan 8, 2023

Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs.

Lunar Lunar is a neural network aimbot that uses real-time object detection accelerated with CUDA on Nvidia GPUs. About Lunar can be modified to work

276 Jan 7, 2023

A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries.

Yolo-Powered-Detector A object detecting neural network powered by the yolo architecture and leveraging the PyTorch framework and associated libraries