Deep learning (neural network) based remote photoplethysmography: how to extract pulse signal from video using deep learning tools

Overview

Deep-rPPG: Camera-based pulse estimation using deep learning tools

Deep learning (neural network) based remote photoplethysmography: how to extract pulse signal from video using deep learning tools Source code of the master thesis titled "Camera-based pulse estimation using deep learning tools"

Implemented networks

DeepPhys

Chen, Weixuan, and Daniel McDuff. "Deepphys: Video-based physiological measurement using convolutional attention networks." Proceedings of the European Conference on Computer Vision (ECCV). 2018.

PhysNet

Yu, Zitong, Xiaobai Li, and Guoying Zhao. "Remote photoplethysmograph signal measurement from facial videos using spatio-temporal networks." Proc. BMVC. 2019.

NVIDIA Jetson Nano inference

The running speed of the networks are tested on NVIDIA Jetson Nano. Results and the installation steps of PyTorch and OpenCV are in the nano folder.

Abstract of the corresponding master thesis

titled "Camera-based pulse estimation using deep learning tools" (also uploaded in this repository)

Lately, it has been shown that an average color camera can detect the subtle color variations of the skin (caused by the cardiac cycle) – enabling us to monitor the pulse remotely in a non-contact manner with a camera. Since then, the field of remote photoplethysmography (rPPG) has been formed and advanced quickly in order the overcome its main limitations, namely: motion robustness and low signal quality. Most recently, deep learning (DL) methods have also appeared in the field – but applied only to adults so far. In this work, we utilize DL approaches for long-term, continuous premature infant monitoring in the Neonatal Intensive Care Unit (NICU).

The technology used in NICU for monitoring vital signs of infants has hardly changed in the past 30 years (i.e., ECG and pulse-oximetry). Even though these technologies have been of great importance for the reliable measurement of essential vital signs (like heart-rate, respiration-rate, and blood oxygenation), they also have considerable disadvantages – originating from their contact nature. The skin of premature infants is fragile, and contact sensors may cause discomfort, stress, pain, and even injuries – thus can harm the early development of the neonate. For the well-being of not exclusively newborns, but also every patient or subject who requires long-term monitoring (e.g., elders) or for whom contact sensors are not applicable (e.g., burn patients), it would be beneficial to replace contact-based technologies with non-contact alternatives without significantly sacrificing accuracy. Therefore, the topic of this study is camera-based (remote) pulse monitoring -- utilizing DL methods -- in the specific use-case of infant monitoring in the NICU.

First of all, as there is no publicly available infant database for rPPG purposes currently to our knowledge, it had to be collected for Deep Neural Network (DNN) training and evaluation. Video data from infants were collected in the $I$st Dept. of Neonatology of Pediatrics, Dept. of Obstetrics and Gynecology, Semmelweis University, Budapest, Hungary and a database was created for DNN training and evaluation with a total length of around 1 day.

Two state-of-the-art DNNs were implemented (and trained on our data) which were developed specifically for the task of pulse extraction from video, namely DeepPhys and PhysNet. Besides, two classical algorithms were implemented, namely POS and FVP, to be able to compare the two approaches: in our dataset DL methods outperform classical ones. A novel data augmentation technique is introduced for rPPG DNN training, namely frequency augmentation, which is essentially a temporal resampling of a video and corresponding label segment (while keeping the original camera sampling rate parameter unchanged) resulting in a modified pulse-rate. This method significantly improved the generalization capability of the DNNs.

In case of some external condition, the efficacy of remote sensing the vital signs are degraded (e.g., inadequate illumination, heavy subject motion, limited visible skin surface, etc.). In these situations, the prediction of the methods might be inaccurate or might give a completely wrong estimate blindly without warning -- which is undesirable, especially in medical applications. To solve this problem, the technique of Stochastic Neural Networks (SNNs) is proposed which yields a probability distribution over the whole output space instead of a single point estimate. In other words, SNNs associate a certainty/confidence/quality measure to their prediction, therefore we know how reliable an estimate is. In the spirit of this, a probabilistic neural network was designed for pulse-rate estimation, called RateProbEst, fused and trained together with PhysNet. This method has not been applied in this field before to our knowledge. Each method was evaluated and compared with each other on a large benchmark dataset.

Finally, the feasibility of rPPG DNN applications in a resource-limited environment is inspected on an NVIDIA Jetson Nano embedded system. The results demonstrate that the implemented DNNs are capable of (quasi) real-time inference even on limited hardware.

Cite as

Dániel Terbe. (2021, January 25). Camera-Based Pulse Monitoring Using Deep Learning Tools.

Special application on neonates

A custom YOLO network is used to crop the baby as a preprocessing step. This network was created based on this repo: https://github.com/eriklindernoren/PyTorch-YOLOv3

Our modified version: https://github.com/terbed/PyTorch-YOLOv3

You might also like...
[CVPR2021] Invertible Image Signal Processing
[CVPR2021] Invertible Image Signal Processing

Invertible Image Signal Processing This repository includes official codes for "Invertible Image Signal Processing (CVPR2021)". Figure: Our framework

Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation".

PixelTransformer Code release for the ICML 2021 paper "PixelTransformer: Sample Conditioned Signal Generation". Project Page Installation Please insta

DI-smartcross - Decision Intelligence Platform for Traffic Crossing Signal Control
DI-smartcross - Decision Intelligence Platform for Traffic Crossing Signal Control

DI-smartcross DI-smartcross - Decision Intelligence Platform for Traffic Crossin

[ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators
[ICLR 2022] Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators

AMOS This repository contains the scripts for fine-tuning AMOS pretrained models on GLUE and SQuAD 2.0 benchmarks. Paper: Pretraining Text Encoders wi

[CVPR 2022] Official PyTorch Implementation for
[CVPR 2022] Official PyTorch Implementation for "Reference-based Video Super-Resolution Using Multi-Camera Video Triplets"

Reference-based Video Super-Resolution (RefVSR) Official PyTorch Implementation of the CVPR 2022 Paper Project | arXiv | RealMCVSR Dataset This repo c

CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Detection in Remote Sensing Images
CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Detection in Remote Sensing Images

CFC-Net This project hosts the official implementation for the paper: CFC-Net: A Critical Feature Capturing Network for Arbitrary-Oriented Object Dete

Official implement of Paper:A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening  images
Official implement of Paper:A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sening images

A deeply supervised image fusion network for change detection in high resolution bi-temporal remote sensing images 深度监督影像融合网络DSIFN用于高分辨率双时相遥感影像变化检测 Of

From this paper "SESNet: A Semantically Enhanced Siamese Network for Remote Sensing Change Detection"

SESNet for remote sensing image change detection It is the implementation of the paper: "SESNet: A Semantically Enhanced Siamese Network for Remote Se

This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CNPs), Neural Processes (NPs), Attentive Neural Processes (ANPs).

The Neural Process Family This repository contains notebook implementations of the following Neural Process variants: Conditional Neural Processes (CN

Owner
Terbe Dániel
Terbe Dániel
PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models

PULSE: Self-Supervised Photo Upsampling via Latent Space Exploration of Generative Models Code accompanying CVPR'20 paper of the same title. Paper lin

Alex Damian 7k Dec 30, 2022
paper: Hyperspectral Remote Sensing Image Classification Using Deep Convolutional Capsule Network

DC-CapsNet This is a tensorflow and keras based implementation of DC-CapsNet for HSI in the Remote Sensing Letters R. Lei et al., "Hyperspectral Remot

LEI 7 Nov 29, 2022
The source code of the paper "Understanding Graph Neural Networks from Graph Signal Denoising Perspectives"

GSDN-F and GSDN-EF This repository provides a reference implementation of GSDN-F and GSDN-EF as described in the paper "Understanding Graph Neural Net

Guoji Fu 18 Nov 14, 2022
We present a framework for training multi-modal deep learning models on unlabelled video data by forcing the network to learn invariances to transformations applied to both the audio and video streams.

Multi-Modal Self-Supervision using GDT and StiCa This is an official pytorch implementation of papers: Multi-modal Self-Supervision from Generalized D

Facebook Research 42 Dec 9, 2022
A Simple LSTM-Based Solution for "Heartbeat Signal Classification and Prediction" in Tianchi

LSTM-Time-Series-Prediction A Simple LSTM-Based Solution for "Heartbeat Signal Classification and Prediction" in Tianchi Contest. The Link of the Cont

KevinCHEN 1 Jun 13, 2022
Video-Captioning - A machine Learning project to generate captions for video frames indicating the relationship between the objects in the video

Video-Captioning - A machine Learning project to generate captions for video frames indicating the relationship between the objects in the video

null 1 Jan 23, 2022
This is a model made out of Neural Network specifically a Convolutional Neural Network model

This is a model made out of Neural Network specifically a Convolutional Neural Network model. This was done with a pre-built dataset from the tensorflow and keras packages. There are other alternative libraries that can be used for this purpose, one of which is the PyTorch library.

null 9 Oct 18, 2022
git《Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser》(2021) GitHub: [fig5]

Pseudo-ISP: Learning Pseudo In-camera Signal Processing Pipeline from A Color Image Denoiser Abstract The success of deep denoisers on real-world colo

Yue Cao 51 Nov 22, 2022
Data manipulation and transformation for audio signal processing, powered by PyTorch

torchaudio: an audio library for PyTorch The aim of torchaudio is to apply PyTorch to the audio domain. By supporting PyTorch, torchaudio follows the

null 1.9k Dec 28, 2022