Framework for the Complete Gaze Tracking Pipeline

Related tags

Computer Vision gaze-tracking-pipeline
Overview

Framework for the Complete Gaze Tracking Pipeline

The figure below shows a general representation of the camera-to-screen gaze tracking pipeline [1]. The webcam image is preprocessed to create a normalized image of the eyes and face, from left to right. These images are fed into a model, which predicts the 3D gaze vector. The predicted gaze vector can be projected onto the screen once the user’s head pose is known.
This framework allows for the implementation of a real-time approach to predict the viewing position on the screen based only on the input image.

camera-to-screen gaze tracking pipeline

  1. pip install -r requirements.txt
  2. If necessary, calibrate the camera using the provided interactive script python calibrate_camera.py, see Camera Calibration by OpenCV.
  3. For higher accuracy, it is also advisable to calibrate the position of the screen as described by Takahashiet al., which provide an OpenCV and matlab implementation.
  4. To make reliable predictions, the proposed model needs to be specially calibration for each user. A software is provided to collect this calibration data.
  5. Train a model or download a pretrained model.
  6. If all previous steps are fulfilled, python main.py --calibration_matrix_path=./calibration_matrix.yaml --model_path=./p00.ckpt can be executed and a "red laser pointer" should be visible on the screen. main.py also provides multiple visualization options like:
    1. --visualize_preprocessing to visualize the preprocessed images
    2. --visualize_laser_pointer to show the gaze point the person is looking at on the screen like a red laserpointer dot, see the right monitor on the image below
    3. --visualize_3d to visualize the head, the screen, and the gaze vector in a 3D scene, see left monitor on the image below

live-example

[1] Amogh Gudi, Xin Li, and Jan van Gemert, “Efficiency in real-time webcam gaze tracking”, in Computer Vision - ECCV 2020 Workshops - Glasgow, UK, August 23-28, 2020, Proceedings, Part I, Adrien Bartoli and Andrea Fusiello, Eds., ser. Lecture Notes in Computer Science, vol. 12535, Springer, 2020, pp. 529–543. DOI : 10.1007/978-3-030-66415-2_34. [Online]. Available: https://doi.org/10.1007/978-3-030-66415-2_34.

You might also like...
DouZero is a reinforcement learning framework for DouDizhu - 斗地主AI
DouZero is a reinforcement learning framework for DouDizhu - 斗地主AI

[ICML 2021] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning | 斗地主AI

Kwai 3.1k Jan 5, 2023
pulse2percept: A Python-based simulation framework for bionic vision
pulse2percept: A Python-based simulation framework for bionic vision

pulse2percept: A Python-based simulation framework for bionic vision Retinal degenerative diseases such as retinitis pigmentosa and macular degenerati

null 67 Dec 29, 2022
MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?
MOT-Tracking-by-Detection-Pipeline - For Tracking-by-Detection format MOT (Multi Object Tracking), is it a framework that separates Detection and Tracking processes?

MOT-Tracking-by-Detection-Pipeline Tracking-by-Detection形式のMOT(Multi Object Trac

KazuhitoTakahashi 41 Nov 23, 2022
collect training and calibration data for gaze tracking
collect training and calibration data for gaze tracking

Collect Training and Calibration Data for Gaze Tracking This tool allows collecting gaze data necessary for personal calibration or training of eye-tr

Pascal 5 Dec 17, 2022
Implementation of gaze tracking and demo
Implementation of gaze tracking and demo

Predicting Customer Demand by Using Gaze Detecting and Object Tracking This project is the integration of gaze detecting and object tracking. Predict

null 2 Oct 20, 2022
Implementation of gaze tracking and demo
Implementation of gaze tracking and demo

Predicting Customer Demand by Using Gaze Detecting and Object Tracking This project is the integration of gaze detecting and object tracking. Predict

null 2 Oct 20, 2022
Tracking Pipeline helps you to solve the tracking problem more easily
Tracking Pipeline helps you to solve the tracking problem more easily

Tracking_Pipeline Tracking_Pipeline helps you to solve the tracking problem more easily I integrate detection algorithms like: Yolov5, Yolov4, YoloX,

VNOpenAI 32 Dec 21, 2022
Complete portable pipeline for masking of Aadhaar Number adhering to Govt. Privacy Guidelines.
Complete portable pipeline for masking of Aadhaar Number adhering to Govt. Privacy Guidelines.

Aadhaar Number Masking Pipeline Implementation of a complete pipeline that masks the Aadhaar Number in given images to adhere to Govt. of India's Priv

null 1 Nov 6, 2021
PyTorch framework A simple and complete framework for PyTorch, providing a variety of data loading and simple task solutions that are easy to extend and migrate

PyTorch framework A simple and complete framework for PyTorch, providing a variety of data loading and simple task solutions that are easy to extend and migrate

Cong Cai 12 Dec 19, 2021
The codes and models in 'Gaze Estimation using Transformer'.
The codes and models in 'Gaze Estimation using Transformer'.

GazeTR We provide the code of GazeTR-Hybrid in "Gaze Estimation using Transformer". We recommend you to use data processing codes provided in GazeHub.

null 65 Dec 27, 2022
Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0
Web service for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation based on OpenFace 2.0

OpenGaze: Web Service for OpenFace Facial Behaviour Analysis Toolkit Overview OpenFace is a fantastic tool intended for computer vision and machine le

Sayom Shakib 4 Nov 3, 2022
OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.
OpenFace – a state-of-the art tool intended for facial landmark detection, head pose estimation, facial action unit recognition, and eye-gaze estimation.

OpenFace 2.2.0: a facial behavior analysis toolkit Over the past few years, there has been an increased interest in automatic facial behavior analysis

Tadas Baltrusaitis 5.8k Dec 31, 2022
Shitty gaze mouse controller

demo.mp4 shitty_gaze_mouse_cotroller install tensofflow, cv2 run the main.py and as it starts it will collect data so first raise your left eyebrow(bo

null 16 Aug 30, 2022
This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.
This is a repository for the Duke University Cloud Computing course project on Serveless Data Engineering Pipeline. For this project, I recreated the below pipeline.

AWS Data Engineering Pipeline This is a repository for the Duke University Cloud Computing course project on Serverless Data Engineering Pipeline. For

null 15 Jul 28, 2021
SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).

SNV Pipeline SNV calling pipeline developed explicitly to process individual or trio vcf files obtained from Illumina based pipeline (grch37/grch38).

East Genomics 1 Nov 2, 2021
Two phase pipeline + StreamlitTwo phase pipeline + Streamlit
Two phase pipeline + StreamlitTwo phase pipeline + Streamlit

Two phase pipeline + Streamlit This is an example project that demonstrates how to create a pipeline that consists of two phases of execution. In betw

Rick Lamers 1 Nov 17, 2021
Using Data Science with Machine Learning techniques (ETL pipeline and ML pipeline) to classify received messages after disasters.
Using Data Science with Machine Learning techniques (ETL pipeline and ML pipeline) to classify received messages after disasters.

Using Data Science with Machine Learning techniques (ETL pipeline and ML pipeline) to classify received messages after disasters.

null 1 Feb 11, 2022
Udacity-api-reporting-pipeline - Udacity api reporting pipeline

udacity-api-reporting-pipeline In this exercise, you'll use portions of each of

Fabio Barbazza 1 Feb 15, 2022
A practical ML pipeline for data labeling with experiment tracking using DVC.

Auto Label Pipeline A practical ML pipeline for data labeling with experiment tracking using DVC Goals: Demonstrate reproducible ML Use DVC to build a

Todd Cook 4 Mar 8, 2022
Comments
  • error for getting calibration output

    error for getting calibration output

    Hello, thank you for this work, I want to have an idea about how will be the output. I want to run the code in the colab: https://colab.research.google.com/drive/1LvhTnsOw3MnVR5YDOP8euNbcpMvzdUC8?usp=sharing to get the calibration_matrix.yaml file, while I got this below error: [ WARN:0] global /tmp/pip-req-build-7m_g9lbm/opencv/modules/videoio/src/cap_v4l.cpp (893) open VIDEOIO(V4L2:/dev/video0): can't open camera by index OpenCV: FFMPEG: tag 0x5634504d/'MP4V' is not supported with codec id 12 and format 'mp4 / MP4 (MPEG-4 Part 14)' OpenCV: FFMPEG: fallback to use tag 0x7634706d/'mp4v' Traceback (most recent call last): File "camera_calibration.py", line 103, in <module> record_video(width=1280, height=720, fps=30) File "camera_calibration.py", line 24, in record_video for idx, frame in enumerate(source): File "/content/gaze-tracking-pipeline/webcam.py", line 30, in __iter__ raise StopIteration StopIteration I appreciate let me know how I can fix this issue. Thanks

    opened by FereshtehFeiz 0
  • using calibration data

    using calibration data

    Hi, is there a way to convert the received data (https://github.com/pperle/gaze-data-collection) to train a model from your other repository https://github.com/pperle/gaze-tracking . train.py expects the MPIIFaceGaze dataset format preprocessed, but I would like to train the model on my own collected data, is there a way to do this?

    opened by Lareton 0
  • Gaze tracking is not accurate

    Gaze tracking is not accurate

    Hi! First of all thanks for publishing your work, it is really helpful! I have been working on my graduation project which is very similar to yours and I wanted to run your program to get some idea on how it works.

    I was able to calibrate and get the yaml file, then I ran the main,py and manually set entered the screen sizes. However, when I run it the screen appears to be way smaller then the actual screen (playing with the numbers didn't help), the laser doesn't appear , and it shows the red line mostly in the center unless I move my head than the line starts moving towards the side my head moved. Can it be because of the calibration? Or what might cause it? How did it work for you, can you share with more details please?

    Thank you!

    opened by begaiym-k 2
  • use of calibration data

    use of calibration data

    Hi, I cannot understand how the calibration data obtained with the main.py script in the gaze-data-collection project is used in this project. In that case a .csv file is produced and the calibration images are saved. How and where are this data used to optimize the gaze vector projection from the 3D space to the point on the monitor on the 2D space? The paper "Efficiency in Real-time Webcam Gaze Tracking" talks about three ways to perform this type of monitor calibration. Geometric, machine learning and hybrid? What kind of optimization do you apply?

    Thanks

    opened by enrico310786 2
Owner
Pascal
Pascal
GitHub
End-to-end pipeline for real-time scene text detection and recognition.

Real-time-Scene-Text-Detection-and-Recognition-System End-to-end pipeline for real-time scene text detection and recognition. The detection model use

Fangneng Zhan 89 Aug 4, 2022
This is a c++ project deploying a deep scene text reading pipeline with tensorflow. It reads text from natural scene images. It uses frozen tensorflow graphs. The detector detect scene text locations. The recognizer reads word from each detected bounding box.

DeepSceneTextReader This is a c++ project deploying a deep scene text reading pipeline. It reads text from natural scene images. Prerequsites The proj

Dafang He 49 Sep 10, 2022
An expandable and scalable OCR pipeline

Overview Nidaba is the central controller for the entire OGL OCR pipeline. It oversees and automates the process of converting raw images into citable

null 81 Jan 4, 2023
Tracking the latest progress in Scene Text Detection and Recognition: Must-read papers well organized

SceneTextPapers Tracking the latest progress in Scene Text Detection and Recognition: must-read papers well organized Information about this repositor

Shangbang Long 763 Jan 1, 2023
Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd.

Head Detector Code for the head detector (HeadHunter) proposed in our CVPR 2021 paper Tracking Pedestrian Heads in Dense Crowd. The head_detection mod

Ramana Subramanyam 76 Dec 6, 2022
Train custom VR face tracking parameters

Pal Buddy Guy: The anipal's best friend This is a small script to improve upon the tracking capabilities of the Vive Pro Eye and facial tracker. You c

null 7 Dec 12, 2021
The Open Source Framework for Machine Vision

SimpleCV Quick Links: About Installation [Docker] (#docker) Ubuntu Virtual Environment Arch Linux Fedora MacOS Windows Raspberry Pi SimpleCV Shell Vid

Sight Machine 2.6k Dec 31, 2022
Generic framework for historical document processing

dhSegment dhSegment is a tool for Historical Document Processing. Its generic approach allows to segment regions and extract content from different ty

Digital Humanities Laboratory 343 Dec 24, 2022
a Deep Learning Framework for Text

DeLFT DeLFT (Deep Learning Framework for Text) is a Keras and TensorFlow framework for text processing, focusing on sequence labelling (e.g. named ent

Patrice Lopez 350 Dec 19, 2022
When Age-Invariant Face Recognition Meets Face Age Synthesis: A Multi-Task Learning Framework (CVPR 2021 oral)

MTLFace This repository contains the PyTorch implementation and the dataset of the paper: When Age-Invariant Face Recognition Meets Face Age Synthesis

Hzzone 120 Jan 5, 2023