GBIM(Gesture-Based Interaction map)

Overview

GBIM

Python 3.6 PaddleX License

手势交互地图 GBIM(Gesture-Based Interaction map),基于视觉深度神经网络的交互地图,通过电脑摄像头观察使用者的手势变化,进而控制地图进行简单的交互。网络使用PaddleX提供的轻量级模型PPYOLO Tiny以及MobileNet V3 small,使得整个模型大小约10MB左右,即使在CPU下也能快速定位和识别手势。

手势

手势 交互 手势 交互 手势 交互
向上滑动 向左滑动 地图放大
手势 交互 手势 交互 手势 交互
向下滑动 向右滑动 地图缩小

进度安排

基础

  • 确认用于交互的手势。
  • 使用det_acq.py采集一些电脑摄像头拍摄的人手姿势数据。
  • 数据标注,训练手的目标检测模型
  • 捕获目标手,使用clas_acq.py获取手部图像进行标注,并用于训练手势分类模型。
  • 交互手势的检测与识别组合验证。
  • 打开百度地图网页版,进行模拟按键交互。
  • 组合功能,验证基本功能。

进阶

  • 将图像分类改为序列图像分类,提高手势识别的流畅度和准确度。
  • 重新采集和标注数据,调参训练模型。
  • 搭建可用于参数调节的地图。
  • 界面整合,整理及美化。

数据集 & 模型

手势检测

  • 数据集使用来自联想小新笔记本摄像头采集的数据,使用labelImg标注为VOC格式,共1011张。该数据集场景、环境和人物单一,仅作为测试使用,不提供数据集下载。数据组织参考PaddelX下的PascalVOC数据组织方式。
  • 模型使用超轻量级PPYOLO Tiny,模型大小小于4MB,随便训练了100轮后保留best_model作为测试模型,由于数据集和未调参训练的原因,当前默认识别效果较差

手势分类

  • 数据集使用来自联想小新笔记本摄像头采集的数据,通过手势检测模型提出出手图像,人工分为7类,分别为6种交互手势以及“其他”,共1102张。该数据集数量较少,手型及手势单一,仅作为测试使用,不提供数据集下载。数据组织形式如下:
dataset
	├-- Images
	|     ├-- up
	┆     ┆    └-- xxx.jpg
	|     └-- other
	┆          └-- xxx.jpg
	├-- labels.txt
	├-- train_list.txt
	└-- val_list.txt
  • 模型使用超轻量级MobileNet V3 small,模型大小小于7MB,由于数据量很小,随便训练了20轮后保留best_model作为测试模型,当前识别分类效果较差

模型文件上传使用LFS,下拉时注意需要安装LFS,参考LFS文档。后续将重新采集和标注更加多样的大量数据集,并采用更好的调参方法获得更加准确的识别模型

演示

手势识别

地图交互

*未显示Capture界面

使用

  1. 克隆当前项目到本地,按照requirements.txt安装所依赖的包opencv、paddlex以及pynput。PaddleX对应请安装最新版的PaddlePaddle,由于模型轻量,CPU版本足矣,参考下面代码,细节参考官方网站
python -m pip install paddlepaddle -i https://mirror.baidu.com/pypi/simple
  1. 进入demo.py,将浏览器路径修改为自己使用的浏览器路径:
web_path = '"D:/Twinkstar/Twinkstar Browser/twinkstar.exe"'  # 自己的浏览器路径
  1. 运行demo.py启动程序:
cd GBIM
python demo.py

常见问题及解决

  1. Q: 拉项目时卡住不动

    A:首先确认按照文档安装LFS。如果已经安装那极大可能是网络问题,可以等待一段时间,或先跳过LFS文件,再单独拉取,参考下面git代码:

    // 开启跳过无法clone的LFS文件
    git lfs install --skip-smudge 
    // clone当前项目
    git clone "current project" 
    // 进入当前项目,单独拉取LFS文件
    cd "current project" 
    git lfs pull 
    // 恢复LFS设置
    git lfs install --force
  2. Q:按q或者手势交互无效

    A:请注意当前鼠标点击的焦点,焦点在Capture,则接受q退出;焦点在浏览器,则交互结果将驱动浏览器中的地图进行变换。

  3. Q:安装PaddleX时报错,关于MV C++

    A:若在Windows下安装coco tool时报错,则可能缺少Microsoft Visual C++,可在微软官方下载网页进行下载安装后重启,即可解决。

  4. Q:运行未报错,但没有保存数据到本地

    A:请检查路径是否有中文,cv2.imwrite保存图像时不能有中文路径。

参考

  1. 玩腻了小游戏?Paddle手势识别玩转游戏玩出新花样!
  2. https://github.com/PaddlePaddle/PaddleX

交流与反馈

Email:[email protected]

You might also like...
Unified learning approach for egocentric hand gesture recognition and fingertip detection
Unified learning approach for egocentric hand gesture recognition and fingertip detection

Unified Gesture Recognition and Fingertip Detection A unified convolutional neural network (CNN) algorithm for both hand gesture recognition and finge

Hand Gesture Volume Control | Open CV | Computer Vision
Hand Gesture Volume Control | Open CV | Computer Vision

Gesture Volume Control Hand Gesture Volume Control | Open CV | Computer Vision Use gesture control to change the volume of a computer. First we look i

use machine learning to recognize gesture on raspberrypi

Raspberrypi_Gesture-Recognition use machine learning to recognize gesture on raspberrypi 說明 利用 tensorflow lite 訓練手部辨識模型 分辨 "剪刀"、"石頭"、"布" 之手勢 再將訓練模型匯入

YouRefIt: Embodied Reference Understanding with Language and Gesture
YouRefIt: Embodied Reference Understanding with Language and Gesture

YouRefIt: Embodied Reference Understanding with Language and Gesture YouRefIt: Embodied Reference Understanding with Language and Gesture by Yixin Che

 Gesture Volume Control Using OpenCV and MediaPipe
Gesture Volume Control Using OpenCV and MediaPipe

This Project Uses OpenCV and MediaPipe Hand solutions to identify hands and Change system volume by taking thumb and index finger positions

Virtual hand gesture mouse using a webcam
Virtual hand gesture mouse using a webcam

NonMouse 日本語のREADMEはこちら This is an application that allows you to use your hand itself as a mouse. The program uses a web camera to recognize your han

Gesture-Volume-Control - This Python program can adjust the system's volume by using hand gestures
Gesture-Volume-Control - This Python program can adjust the system's volume by using hand gestures

Gesture-Volume-Control This Python program can adjust the system's volume by usi

GazeScroller - Using Facial Movements to perform Hands-free Gesture on the system

GazeScroller Using Facial Movements to perform Hands-free Gesture on the system

Gesture recognition on Event Data

Event based Gesture Recognition Gesture recognition on Event Data usually involv

Owner
null
Deep learning based hand gesture recognition using LSTM and MediaPipie.

Hand Gesture Recognition Deep learning based hand gesture recognition using LSTM and MediaPipie. Demo video using PingPong Robot Files Pretrained mode

Brad 24 Nov 11, 2022
A hobby project which includes a hand-gesture based virtual piano using a mobile phone camera and OpenCV library functions

Overview This is a hobby project which includes a hand-gesture controlled virtual piano using an android phone camera and some OpenCV library. My moti

Abhinav Gupta 1 Nov 19, 2021
Hand Gesture Volume Control is AIML based project which uses image processing to control the volume of your Computer.

Hand Gesture Volume Control Modules There are basically three modules Handtracking Program Handtracking Module Volume Control Program Handtracking Pro

VITTAL 1 Jan 12, 2022
PaddleRobotics is an open-source algorithm library for robots based on Paddle, including open-source parts such as human-robot interaction, complex motion control, environment perception, SLAM positioning, and navigation.

简体中文 | English PaddleRobotics paddleRobotics是基于paddle的机器人开源算法库集,包括人机交互、复杂运动控制、环境感知、slam定位导航等开源算法部分。 人机交互 主动多模交互技术TFVT-HRI 主动多模交互技术是通过视觉、语音、触摸传感器等输入机器人

null 185 Dec 26, 2022
Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

null 105 Dec 23, 2022
Code for KDD'20 "An Efficient Neighborhood-based Interaction Model for Recommendation on Heterogeneous Graph"

Heterogeneous INteract and aggreGatE (GraphHINGE) This is a pytorch implementation of GraphHINGE model. This is the experiment code in the following w

Jinjiarui 69 Nov 24, 2022
Shuwa Gesture Toolkit is a framework that detects and classifies arbitrary gestures in short videos

Shuwa Gesture Toolkit is a framework that detects and classifies arbitrary gestures in short videos

Google 89 Dec 22, 2022
A gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor.

OpenHands OpenHands is a gesture recognition system powered by OpenPose, k-nearest neighbours, and local outlier factor. Currently the system can iden

Paul Treanor 12 Jan 10, 2022
Gesture-controlled Video Game. Just swing your finger and play the game without touching your PC

Gesture Controlled Video Game Detailed Blog : https://www.analyticsvidhya.com/blog/2021/06/gesture-controlled-video-game/ Introduction This project is

Devbrat Anuragi 35 Jan 6, 2023
Implementation of QuickDraw - an online game developed by Google, combined with AirGesture - a simple gesture recognition application

QuickDraw - AirGesture Introduction Here is my python source code for QuickDraw - an online game developed by google, combined with AirGesture - a sim

Viet Nguyen 89 Dec 18, 2022