RealTime Sign Language Detection using Action Recognition
Approach
Real-Time Sign Language is commonly predicted using models whose architecture consists of multiple CNN layers followed by multiple LSTM layers. However , the accuracy of these state of the art models is pretty low. On the other hand, this approach , Mediapipe Holistic with LSTM Model gives a much better accuracy. This approach produced better results with very less amount of data . Since this model trained on fewer parameters, it trained much faster thus resulting in lesser computation time.
Project
This project is divided into two parts:
- Keypoints extraction using MediaPipe Holistic
- LSTM Model trained on these keypoints to predict realtime sign language using video sequences.
Dataset
Data is collected using MediaPipe Holistic for 3 actions :
- Hello
- Thanks
- I Love You
30 frames have been collected for each action and 30 sequences for each frame have been collected from real time actions using Computer Vision and MediaPipe Holistic. For each sequence , 1662 keypoints have been extracted.
- Face Landmarks - 468*3
- Pose Landmarks - 33*4
- Left Hand Landmarks - 21*3
- Right Hand Landmarks - 21*3
The dataset can be accessed from the Feature_Extraction Folder.
Model
LSTM Model is trained using the extracted keypoints from the Feature_Extraction folder and later used for real time predictions.
The Weights of the model are saved in the lstm_model.h5 file.
How to Use
-
Clone the repository using :
$ git clone https://github.com/rishusiva/Pose-Network -
Install the requirements using:
$ cd Pose-Network/ $ pip install -r requirements.txt -
To Predict Sign Languages in Real Time , run :
$ cd Pose-Network/Code $ python3 realtime_testing.py
Results
- Our LSTM Model, after training for only 100 epochs, has an accuracy of 70%
- It produced an accuracy score of 1.0 on a test set of 5 images.
- Our Trained LSTM Model is then used for real time testing.
Prediction Results:
.gif)
Author
- Rishikesh Sivakumar
28 Dec 19, 2022
140 Nov 23, 2022
76 Jan 3, 2023
8 May 22, 2022
14 Dec 30, 2022
128 Dec 8, 2022
3 Feb 24, 2022
49 Dec 22, 2022
84 Jan 1, 2023
1 Feb 11, 2022
49 Dec 5, 2022
271 Nov 29, 2022
4 Mar 30, 2022
66 Nov 16, 2022
217 Dec 12, 2022
30 Dec 18, 2022
1.3k Jan 4, 2023
71 Dec 25, 2022