EP4698045A1

EP4698045A1 - System and method for estimating at least one physiological sign of a subject in dark environments

Info

Publication number: EP4698045A1
Application number: EP24721748.2A
Authority: EP
Inventors: Ankit Gupta; Fernando Manuel ROSMANINHO MORGADO FERRÃO; Antonio Gabriel RAVELO GARCÍA
Original assignee: Universidad de las Palmas de Gran Canaria; Universidade Da Madeira
Current assignee: Universidad de las Palmas de Gran Canaria; Universidade Da Madeira
Priority date: 2023-05-30
Filing date: 2024-04-10
Publication date: 2026-02-25
Also published as: WO2024246626A1

Abstract

The present invention describes a system and method for estimating physiological signs in dark environment with luminance threshold of 1 lux or less. This invention, which is technically related to the biomedical engineering area, enables contactless sleep monitoring, in home-based monitoring for example, where physiological signs could be used for health monitoring, or to identify the diseased state of the subject. The proposed invention can also be applied in various healthcare facilities, general wards, ICUs (intensive care unit), NICUs (Neonatal Intensive Care Unit), etc. for physiological signs estimations or contactless sleep monitoring. Furthermore, this invention also encourages and enables home-based health monitoring through a InSite implementation, particularly in a subject's sleeping bedroom. Furthermore, it can be used in scenarios that avoid physical contact with the subject's body organs, such as burnt or sensitive skin, neonatal physiological signs estimations, contactless sleep monitoring, and driving scenarios during extremely dark conditions.

Description

SYSTEM AND METHOD FOR ESTIMATING AT LEAST ONE PHYSIOLOGICAL SIGN OF A SUBJECT IN DARK ENVIRONMENTS

Technical Field

The present invention describes a system and method for estimating physiological signs in dark environment with luminance threshold of 1 lux or less.

Background art

Physiological signs such as Heart rate (HR), HR variability (HRV), Breathing rate (BR), BR variability (BRV), oxygen saturation (SpO2), Blood Pressure (BP), and Body Temperature (BT) are known to serve as an objective indicators of the individual's physiological health. Therefore, their estimation and monitoring play a pivotal role in identifying the diseased state of the subject due to their abnormal ranges. One of the approaches for extracting the physiological signs is to determine the temporal volume variations due to the blood flow synchronized with the cardio-vascular wave flowing through the subject's body. This technique is called plethysmography.

Physiological signs are extracted using plethysmography by exploiting the optical properties of blood and skin tissues, therefore called photoplethysmography (PPG). When the light from an illumination source is passed through the body organ, the transmitted or reflected light captured by the photodetector is used to extract the signal containing PPG information for estimating physiological signs.

Conventional Pulse oximeters estimate physiological signs using the transmitted light through the skin, blood tissues, and bones. These are generally clips that fit the subject's body organs, such as fingers, earlobe, etc., and have illumination sources and photodetector opposite to each other. Their contact with the skin limits their applicability in the scenarios such as prolonged monitoring, sleep analysis, burnt or sensitive skin and neonatal physiological signs monitoring. On the other hand, photoplethysmography using cameras (also referred to as remote photoplethysmography (rPPG) or imaging photoplethysmography (iPPG)) works on the principle of reflected light from the body organ such as the face, palm, finger, etc. facing it. Being a non-contact approach, physical contact with the skin is not required which makes it applicable for above mentioned scenarios as well. This technique allows for the measurement of physiological signs using video cameras and image processing algorithms instead of traditional sensors like ECG electrodes or pulse oximeters. This technology is commonly used in settings where traditional sensors are impractical or uncomfortable, such as during sleep studies or in virtual health consultations.

This approach is usually challenging because of the lower pulsatile strength of a rPPG signal, and because of the presence of InSite motion and illumination variation artifacts. Since the physiological sign estimation needs the extraction of the frequency of interest (FOI), it may lie in the motion spectrum for higher degrees of the motion, while illumination variation artifacts cause difficulty in selecting the appropriate region of interest. Consequently, these artifacts can be avoided by imposing constrained laboratory conditions by allowing constrained motion of the subject or constant illumination using single or multiple illumination sources focussed to the region of interest

(ROI).

Several studies published in this technical field have claimed to test their methods in clinical conditions such as neonatal intensive care units or other medical conditions, which requires low light conditions.

Furthermore, limited studies have been focused on physiological signs estimations in dark environments as sleeping, but all these studies have utilized infrared spectra to develop respective methods in such scenarios.

Following studies disclosed a proof-of-concept based on a sleep study (dark environment) where in NIR camera synchronized with the PSG box was used to extract vital signs followed by detecting sleep disorders.

The demerit of using Infrared (IR) spectra is its low pulsatile strength. Hence, an alternative is highly recommended to provide relatively better pulsatile strength in the dark scenario. It is known that visible spectra provide the strongest pulsatile strength among other available spectra. However, image color models working on the principle of visible spectra are not suitable for limited or no illumination conditions due to challenging conditions for appropriate ROI selection which is a preliminary condition to estimate at least one physiological sign.

Document W02021/074036A1 disclosed an invention presenting the device, system and method for physiological sign extraction during inhomogeneous illumination, wherein the camera exposure time is altered and selecting the high- quality skin pixels for vital signs extraction. However, the disclosure cannot be used for aforementioned dark scenarios, since it works on the principle of acquiring high quality skin pixels, which is not always possible in such scenarios. Therefore, existing methods would not work for the dark environments due to poor pulsatile strength of IR spectra, and inability of visible spectra methods to extract appropriate ROI in dark or extremely dark environments.

Summary

The present invention describes a system for estimating at least one physiological sign of a subject in a dark environment, comprising: a capturing device, located within a predetermined distance from the subject, comprising a detector configured to acquire at least one of a plurality of image frames; an electronic device, connected to the capturing device, comprising at least an internal memory and a central processing unit; and an information display medium connected to the electronic device; characterized by collecting a remote photoplethysmography signal from a region of interest of the subject, followed by the estimation of at least one physiological sign of said subject.

In a proposed embodiment of present invention, the capturing device further comprises an illumination source configured to provide a light source with a luminance threshold of at most 1 lux over an exposed region of the subject.

Yet in another proposed embodiment of present invention, the detector is configured to detect a visible electromagnetic spectrum from a region of interest of the subject, said region of interest comprising a skin region.

Yet in another proposed embodiment of present invention, the visible electromagnetic spectrum comprises a light reflection enabled by the illumination source which is configured to emit different wavelengths of a visible electromagnetic spectrum radiation.

Yet in another proposed embodiment of present invention, the electronic device is configured to store in the internal memory at least one of the plurality of image frames, said plurality of image frames comprising at least one normal environment light conditions image frame, and consecutive low exposure image frames with a fixed sampling rate.

Yet in another proposed embodiment of present invention, the display medium is configured to display a set of information comprising at least a remote photoplethysmography signal and physiological sign of the subject, said set of information being determined and provided by the electronic device after performing a set of image processing steps in the central processing unit based on the stored plurality of image frames in the internal memory.

The present invention describes also describes a method for estimating at least one physiological sign of a subject in a dark environment, according to the previously described system, said method comprising the steps of: acquiring at least one of a plurality of image frames by means of the capturing device; storing the acquired at least one of a plurality of image frames into the internal memory of the electronic device; applying an image enhancement step to the stored at least one of a plurality of image frames; storing the enhanced plurality of enhanced image frames in the internal memory of the electronic device; performing a remote photoplethysmography signal extraction from the enhanced plurality of enhanced image frames; performing a signal refinement to the remote photoplethysmography signal extraction; and estimating physiological sign from the resulting signal refinement; said image enhancement step, remote photoplethysmography signal extraction, signal refinement and physiological sign estimations being performed the central processing unit of the electronic device.

In a proposed embodiment of the invention, the image enhancement step enables the enhancing of the consecutive low exposure image frames for extracting information related to remote photoplethysmography signal extraction; the remote photoplethysmography signal extraction step enables the extraction of the remote photoplethysmography signal from the consecutive low exposure image frames; the signal refinement step enables the extraction of the signal of interest from the remote photoplethysmography signal depending on the physiological sign to estimate; and the physiological sign estimation step, enables the estimation of the desired physiological sign of the subject.

Yet in another proposed embodiment of the invention, the image enhancement step comprises a deep learning model enabling an output based on the least one normal environment light conditions image frame, and the consecutive low exposure image frames to be enhanced.

Yet in another proposed embodiment of the invention, the deep learning model comprise two encoders for extracting features from the normal environment light conditions image frame and the consecutive low exposure image frames in a hierarchy, and a decoder for enhancing the consecutive low exposure image frames based on the extracted features from both the encoders using hierarchical and residual connections, respectively. Yet in another proposed embodiment of the invention, the encoders comprise a sequence of convolutions with Rectified Linear Unit activations, and pooling layers with kernel size k × k, and number of kernels defined by set K_e = {k₁,k₂,...,k_n :k₁ < k₂ < •••< k_n}; and the decoder comprises of sequence of transposed convolutions with Rectified Linear Unit activation and convolution layers with kernel size k × k and number of kernels K_d = {k_n,k_n-1,...,k₁ :k_n > k_n--1 > •••> k₁}.

Yet in another proposed embodiment of the invention, the image enhancement step further comprises a loss function defined as: Loss=α₁*L1+α₂*MS-SSIM+α₃ *L2 which consists of linear combinations of pixel intensity difference (L1), multi-scale structured similarity index (MS-SSIM), and mean square (L2) losses, optimized using an optimization algorithm with a few parameters, and single or multiple stopping criteria.

Yet in another proposed embodiment of the invention, the deep learning model further enables a trained or optimized output model stored in the internal memory of the electronic device, which is used to enhance the image frames, enhancing low-exposure image frames based on normal light image.

Yet in another proposed embodiment of the invention, the image enhancement step outputs consecutive enhanced image frames by enhancing the pixel intensities of the plurality of image frames.

Yet in another proposed embodiment of the invention, the enhanced image frames are stored in the internal memory of the electronic device for posterior processing by the central processing unit to extract the remote photoplethysmography signal corresponding to remote photoplethysmography signal extraction .

Yet in another proposed embodiment of the invention, the central processing unit is configured to execute machine instructions to extract an appropriate region of interest of the subject followed by skin segmentation of the subject from the consecutive enhanced image frames.

Yet in another proposed embodiment of the invention, the remote photoplethysmography signal extraction comprises at least a detection and tracking of a region of interest from the consecutive enhanced image frames, a skin segmentation from the region of interest using skin pixel extraction models, a spatial averaging of consecutive image frames resulting in n-raw signals generations, and a remote photoplethysmography signal Extraction using undercomplete independent component analysis based method.

Yet in another proposed embodiment of the invention, the remote photoplethysmography signal extraction step performs a channel-wise spatial averaging operation and preprocessing to create multiple raw signals, defined by the number of channels of the image, also referred to as mixture signals.

Yet in another proposed embodiment of the invention, the remote photoplethysmography signal extraction step comprises an undercomplete independent component analysis-based method equipped with an entropy based objective function and optimization procedure, to extract the remote photoplethysmography signal using an assumption of under completeness, leading to one independent component extracted from more than one mixture signals. Yet in another proposed embodiment of the invention, the undercomplete independent component analysis-based method estimates a matrix, which multiply to the mixture signals produces the remote photoplethysmography signal.

Yet in another proposed embodiment of the invention, the entropy based objective function is based on a cumulative density function of the independent component with maximum entropy, optimized using a modified Levenberg-Marquardt algorithm for entropy maximization to estimate a matrix.

Yet in another proposed embodiment of the invention, the optimization procedure is performed iteratively on the matrix based on entropy maximization of the cumulative density function of the independent component, until the convergence of the objective function.

Yet in another proposed embodiment of the invention, a linear combination of the optimized matrix and a mixture of signals produces the remote photoplethysmography signal.

Yet in another proposed embodiment of the invention, the remote photoplethysmography signal will be further refined to extract the signal of interest based on physiological sign to be estimated.

Yet in another proposed embodiment of the invention, the physiological sign can be extracted from the signal of interest by extracting the desired frequency from the signal.

The invention further describes a data processing system, characterized by comprising the physical means necessary for the execution of the system and method previously described.

The invention further describes a computer program, characterized by comprising programming code or instructions suitable for operating the system and executing the method previously described, in which said computer program is stored, and is executed in a data processing system, remote or in-site, for example a server, performing the respective steps described in the claims.

The invention further describes a computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.

General Description

The present application describes a system and method that enables, by means of a contactless monitoring approach, collecting a remote photoplethysmography (rPPG) signal from an appropriate region of interest (ROI) a of a patient, which will be followed by the estimation of at least one physiological sign by extracting said signal of interest from the rPPG signal.

This invention discloses a system and a method for estimating at least one physiological sign from a patient in dark environments with luminance of 1 lux or less. The overall developed system is comprised of a capturing device and an additional electronic device. The capturing device, which will be faced to a subject or patient under analysis, comprises a detector and an illumination source. The electronic device comprises a central processing unit, an internal memory, and a display screen.

The capturing device is configured to, in an initial phase of the procedure, capture a single image in ambient light, and, afterwards, capture a consecutive sequence of image frames in dark environment with above mentioned luminance, from hereafter referred to as image dataset. The device captures a sequence of image frames from the exposed region of the subject for a definite time interval, not being supported by additional external light sources than the one provided by the capturing device.

The central processing unit is configured to execute machine instructions related to the method herein disclosed, where the internal memory is used for storing data, executable machine instructions, trained data models, data inputs and enhanced image datasets used along said method calculations. The display screen is used to display the remote photoplethysmography signal and physiological signs estimated from the captured signals from the subject under analysis.

Operationally speaking, the system is configured to capture an image dataset, which will be transferred to the electronic device within a predefined bandwidth. The image dataset is then stored in its internal memory, and consequently processed by the aforementioned system's central processing unit using the above-mentioned machine instructions for image enhancement, remote photoplethysmography signal extraction, and physiological sign estimations. The remote photoplethysmography signal and physiological sign information is then transferred to the display screen. The information on the display screen can be used by medical expert for the health monitoring of the subject under surveillance .

The image enhancement step of the proposed method, from hereafter referred to as IE, is supported by a deep learning model consisting of two encoders, and one decoder, respectively. The encoders of the IE (image enhancement) consist of a sequence of convolution layers, rectified linear unit (ReLU) activation and pooling layers with increasing number of kernels than its preceding layers. On the other hand, the decoder of the IE consists of a sequence of transposed convolutions, ReLU activation and convolution layers with decreasing number of kernels than the preceding layers. While the two encoders of the IE are used for hierarchical extraction of features from the reference image with natural light and from the dark image, respectively, the decoder of the IE combines the features from both of the IE encoders to produce an enhanced image.

The IE component step comprises hierarchical connections, like cascaded connections, for transferring the relevant features from the encoder of the dark image to the decoder. Essentially, these hierarchical connections are responsible for passing the reflectance feature properties from the dark image. The IE step also comprises residual connections, like skip connections, from each layer of the reference image encoder to respective levels of the decoder after performing the transposed convolutions for its succedent layers.

The IE component step also comprises a neural network architecture for further refinement of the enhanced image for preserving related info to accurately estimate remote photoplethysmography signal, enabling the extraction of subtle colour variations and details. This neural network architecture essentially, consists on a sequence of convolution layers with ReLU activations with k number of kernels and kernel size. The last convolution layer is without the ReLU activation.

The IE component also comprises a loss function which is a linear combination of pixel intensity difference (L1), multi-scale structured similarity index (MS-SSIM) and mean square (L2) losses. The objective function is represented as follows by: Loss=α₁*L1+α₂*MS-SSIM+α₃*L2 (1)

The IE component step also comprises an optimization algorithm which optimizes the above-mentioned loss function during the IE component's training process. This optimization algorithm requires a learning rate, momentum terms, and a few network parameters for optimization. The network optimization requires one or more stopping criteria, to terminate training process. This step uses early stopping with a patience value, and number of iterations as stopping criteria, while optimizing the loss function. Optimization conducted with appropriate loss function, optimization algorithm, and parameters yield better performance, generalization capacity, and robustness of the IE component. Once trained, the IE component enhances the consecutive sequence of image frames of a subject, using a reference image, of the same subject. The enhanced images from hereafter referred to as enhanced image dataset, are stored in the internal memory of the electronic device for further CPU processing in order to extract rPPG signal of the subject for estimating at least one physiological sign.

The rPPG signal extraction component step from hereafter referred to as rPPGSig component comprises a ROI selection which selects a suitable skin region from the subject under analysis for extracting the remote photoplethysmography signal. The ROI selection initially detects the facial region from the first image frame of the enhanced image dataset, tracks the detected region in all consecutive image frames, and performs skin segmentation steps. If the region is not found in the frame, then the corresponding frame will be removed from the consecutive image sequence. Those skilled in the art will understand that the skin segmentation needs few additional metrics but not limited to signal-to-noise ratio, to ensure accurate rPPG signal. Finally, the raw signals are constructed from each channel, by spatially averaging the skin region from the individual frames of the enhanced image sequence. The raw signals can be further refined using, but are not limited to, pre-processing steps like removing the slowly varying trends.

The rPPGSig component assumes the remote photoplethysmography signal extraction problem as an undercomplete problem, which states that the number of independent components to be extracted is less than number of mixture components. Therefore, a single independent component representative of remote photoplethysmography signal is to be extracted from more than one mixture signals defined by number of image channels, each representing different wavelengths in visible electromagnetic spectrum.

The rPPGSig component step also comprises an independent component analysis-based method for extracting m independent components from n mixture signals, each corresponding to a channel of the image frame. The method estimates a matrix which, when linearly combined with n mixtures signals, produces a single independent component which represents the rPPG signal.

The undercomplete independent component analysis of rPPGSig component step consists of an objective function which is to be optimized to accurately estimate a matrix W. Since independent components can be extracted based on the degree of randomness between them, the objective function for rPPGSig component step is an entropy-based function. The function is based on the mathematical theorem which states that the cumulative density function of the independent components has maximum entropy. Therefore, this objective function calculates the entropy of the cumulative density function of the independent component, which is mathematically defined as: where W is a matrix, which is used to extract the rPPG signal from a 2-D matrix of mixture signals with n rows equal to the number of channels, and T columns with T equals to length of the enhanced image dataset; S is a diagonal matrix consisting of covariance values of mixture signals, y is the independent component; sec indicates the secant trigonometric ratio, and Tr corresponds to matrix transpose. The undercomplete independent component analysis of rPPGSig also comprises a optimization function for optimizing the above-mentioned objective function. This optimization function is based on Levenberg-Marquardt algorithm, which is modified accordingly to ensure maximization of the cost function, representing the entropy of the cumulative density function of the independent component. The optimization function also includes a stopping criterion to terminate the optimization process. Those skilled in the art would understand that single or multiple stopping criteria could be used for optimization. Optimizing the objective function contributes to the update the values of matrix W as follows: where is the pseudo inverse of matrix W with respect to S, positive definite matrix.

The undercomplete independent component analysis of the rPPGSig component step will output an independent component by accurately estimating matrix W after optimizing the above- mentioned objective function using modified Levenberg- Marquardt algorithm for entropy maximization. The resultant independent component is the remote photplethysmography signal, which can be processed using suitable signal processing techniques for extracting the signal followed by the frequency of interest for physiological sign estimation.

In general terms, the estimation method comprises using image enhancement of a sequence of low exposure image frames based on a single image in ambient light of the subject or patient, rPPG signal extraction based on undercomplete independent component analysis-based method with entropy based objective function and modified Levenberg-Marquardt algorithm for optimization, signal refinement for extracting the signal of interest, followed by identifying the frequency related to the physiological sign.

The rPPG signal can be used to estimate at least one physiological sign of a patient by extracting the respective signal of interest from this rPPG signal. The system enables the extraction of videos in the dark environment along with the normal light image from the subject/patient, followed by a computer implemented method that enables the estimation of at least one physiological sign. The disclosed method consists in two steps: image enhancement and rPPG signal extraction. The extracted rPPG signal is then used to estimate at least one physiological sign.

In one of the preferential embodiments of the present invention, the visible spectra will serve as a competent and a reliable alternative for better signal strength of captured signal, better than the commonly used infrared spectra in dark environments with at most permissible luminance of 1 lux. Precisely, a system and two step end-to-end process determining procedures is presented for estimating physiological signs for health monitoring in dark environments. The video acquired, in the dark environment, is first enhanced using an image enhancement component, subsequently leading the remote photoplethysmography signal to be extracted by means of an undercomplete independent component analysis-based method equipped with an entropy based objective function and Levenberg-Marquardt based optimization algorithm, for physiological sign estimations.

This technological development encourages the use of contactless approach for physiological sign estimations, which are useful for several medical applications like telemedicine, home-based monitoring, medical solution for medical facilities such as hospitals, clinics, sleep monitoring centres etc. Additionally, this invention will help medical experts to deal with the elderly people which cannot come on a regular basis to a medical facility. Besides that, it promotes an extensive real time health monitoring use in non-medical facilities such as vehicles etc.

This invention expands the applicability of contactless approach to work in the luminance constrained situations such as physiological sign estimation during sleeping, driving during night-time, overcoming the performance of existing methods for physiological sign estimation based on the use of sensors, electrodes, and clips, which are not suitable for prolonged monitoring, sensitive skin, subjects performing daily or professional activities, constrained luminance conditions etc. Brief description of the drawings

For better understanding of the present application, figures representing preferred embodiments are herein attached which, however, are not intended to limit the technique disclosed herein.

Fig. 1 - illustrates one of the preferred embodiments of the developed system for estimating at least one physiological sign in dark environment (luminance<llux), where the reference numbers are related with: 1 - subject; 2 capturing device; 21 - detector; 22 - illumination source; 3 - electronic device; 31 - internal memory; 32 - central processing unit (CPU); 4 - display screen.

Fig. 2 - depicts the detailed steps of the method, which comprise a IE component step, an rPPGSig step for enhancing plurality of consecutive image frames, and a remote photoplethysmography signal extraction step for estimating at least one physiological parameter sign. The reference numbers are related with: 101a - normal light image; 101b - low exposure image sequence, or video, captured in a luminance environment inferior to a 1 lux, i.e., consecutive low exposure image frames; 102 - image enhancement (IE) based on deep learning architectures; 103 - rPPG signal extraction based on rPPGSig component; 104 - rPPG signal refinement based on signal processing techniques; 105 - physiological sign estimation.

Fig. 3 - illustrates the deep learning architecture for image enhancement (102) of a plurality of image frames. The reference numbers are related with: KI - Layer 1; K2 - Layer 2; K3 - Layer 3; K4 - Layer 4; 1021 - ReLU (Conv) + Pool; 1022 - ReLU (Conv); 1023 - ReLU (Trans-Conv); 1024 - consecutive sequence of enhanced image frames. Fig. 4 - illustrates the encoder architecture of the IE (102), where the reference numbers relate to: KI - Layer 1; K2 - Layer 2; K3 - Layer 3; K4 - Layer 4; 1021 - ReLU (Conv) + Pool.

Fig. 5 - illustrates the decoder architecture of the IE (102), where the reference numbers relate to: KI - Layer 1; K2 - Layer 2; K3 - Layer 3; K4 - Layer 4; 1022 - ReLU (Conv); 1023 - ReLU (Trans-Conv).

Fig. 6 - illustrates the detailed steps of the rPPGSig component for extracting the rPPG signal, i.e., rPPG signal extraction based on rPPGSig component (103), followed by the rPPG signal refinement using signal processing techniques (104) with will lead to the physiological estimation of at least one sign (105). The reference numbers relate to: 1024 -consecutive sequence of enhanced image frames; 202 - Region of interest detection and tracking from all image frames;

203 - Skin segmentation using a skin pixel extraction model;

204 - n-raw signals generations by spatial averaging of consecutive image frames; 205 - remote photoplethysmography signal Extraction using undercomplete independent component analysis based method ; 103 - rPPG signal extraction based on rPPGSig component; 104 - rPPG signal refinement based on signal processing techniques (physiological signs dependent); 105 - physiological sign estimation.

Description of Embodiments

With reference to the figures, some embodiments are now described in more detail, which are however not intended to limit the scope of the present application.

The proposed system, illustrated in figure 1, comprises a capturing device (2) and an electronic device (3). The capturing device (2) comprises a detector (21) and an illumination source (22), and the electronic device (3), which is connected to the capturing device (2), comprises an internal memory (31), a central processing unit (32) and a display screen (4). The capturing device (2) in one of the embodiments is pointed out to the subject (1) body in order to ensure a correct visual framework of the same, being therefore able to capture a ROI, in a visible spectrum with negligible illumination (luminance<llux) emitted by it.

The detector (21) comprised of the capturing device (2) operates within the visible light spectrum and is configured to capture a sequence of consecutive image frames in the dark (< 1 lux) and is also configured to capture images of the subject in normal environment light conditions. The illumination source (22) is configured to provide a minimum required luminance which is set to be inferior to 1 lux.

The internal memory (31) of the electronic device (3) comprises enough storage capacity to enable storing machine instructions corresponding to the steps of the operating method, a sequence of consecutive input and enhanced image frames, and trained deep learning models. The CPU (32) of the electronic device (3) is configured to execute machine instructions related to the steps of the method herein disclosure, which comprise at least image enhancement, rPPG signal extraction and physiological sign estimations.

The determined remote photoplethysmography signal and physiological sign estimation (105) is displayed on the display screen (4), which can be analysed and monitored for health monitoring purposes or for identifying the diseased state of the subject (1).

Figure 2 illustrates a step-by-step method of the disclosed method for estimating at least one physiological sign (105) of a subject (1). The method comprises a set of steps which include capturing an image of a subject (1) in the normal environment light conditions (101a), and a corresponding consecutive low exposure image frames (101b) captured in a dark environment condition, the luminance of said environment being set to be inferior to 1 lux. The image capturing is performed by the capturing device (2). The colour intensity of the consecutive low exposure image frames (101b) is enhanced through image enhancing step (102) based on a trained deep learning model, which consists of two encoders and one decoder, which are trained using iterative optimization of a loss function. The resulting enhanced images (1024) will then be used in the rPPGSig component (103) step, from which remote photoplethysmography signal extraction is produced using undercomplete independent component analysis-based method equipped with an objective function and optimization process.

Subsequently, as illustrated in Fig. 2, the remote photoplethysmography signal will be refined based on signal processing techniques (104). Both the rPPG signal, and the estimated physiological sign (105) will be displayed on the display screen (4) for health monitoring or identification of diseased state of the subject (1).

In Fig. 3 it is illustrated the schematic architecture of the trained deep learning model for image enhancement (102), which consists of two encoders configured to extract the invariant features, one decoder combining the relevant features from the reference image (101a) with the reflectance features of the consecutive low exposure image frames (101b), and refinement component for image enhancement for preserving minute details required for accurate remote photoplethysmography signal extraction (103). Fig.4 illustrates the schematic diagram depicting the architecture of an encoder of the image enhancement (102) step, which consist of a set of hierarchically arranged layers. Each layer comprises of convolutions with Rectified Linear Unit (ReLU) activation and pooling for extracting relevant features, using a set of kernels defined by K=k_i :i=1,2,...,4 and k_i<k_i+1}, and kernel size as k_size. The encoders ensure hierarchical extraction of invariant global and local features from the image, for image enhancement.

Fig. 5 discloses the architecture of the decoder of the image enhancement (102) which comprises a set of layers containing transposed convolutions with ReLU activation and convolutions with set of Kernels defined by K={K=k_i :i=1,2,...,4 and k_i<k_i+1 }, and kernel size as k_size. The decoder possesses hierarchical connections which ensures decoding of the features from the encoder of consecutive low exposure image frames (101b). The decoder also possesses residual connections which ensure the layer-wise fusion of features from the encoder of the reference image at each consecutive layer of the decoder.

The IE step (102) also consists of a deep learning architecture for further refinement of enhanced images. The architecture comprises of a sequence of convolution layers with k kernels of kernel size kernel_size, which further refines the enhanced images from encoder-decoder for preserving useful information for accurate remote photoplethysmography signal generation, and better perceptual visibility.

All components of the IE step (102) shown in Fig.3, are iteratively optimized using a loss function which consists of a linear combination of pixel intensity difference (L1), multi-scale structured similarity index (MS-SSIM), and mean square (L2) losses. The loss function is mathematically defined as:

Loss=α₁*L1+α₂*MS-SSIM+α₃*L2 (4)

The optimization procedure requires learning of parameters in the IE step (102), using metrics such as learning rate, and momentum terms. The termination of optimization procedure is governed by the stopping criteria. The trained deep learning architecture used in the representation of the IE step (102) enhances a sequence of image frames representative of a subject (1).

As illustrated in Fig.6, the consecutive sequence of enhanced image frames (1024) is stored in the internal memory (31) of the electronic device (3), before being sent to the remote photoplethysmography signal extraction based on rPPGSig component (103). A step-by-step procedure of the rPPGSig component is illustrated in Fig 6., which includes at least four steps, in particular steps 202 to 205. The first of said steps comprises the extraction of the region of interest from first frame of the image and tracking it to all consecutive image frames of the subject (202). If the region of interest cannot be tracked in an image frame, it must be discarded. Subsequently, the second step comprises the skin segmentation resorting to skin pixel extraction models (203) of the region of interest which is performed based on metrics but are not limited to signal-to-noise ratio. On the third step, in each image frame, a representation of the facial skin region of the subject (1) is spatially averaged over the channels (204), each one representing the different wavelength intervals of the visible electromagnetic spectrum. The resultant signals are further pre-processed using signal processing but are not limited to techniques such as detrending or normalization. The fourth step, which comprises the remote photoplethysmography signal extraction (205) is assumed to be an undercomplete problem which states that m independent signals can be derived from n mixture signals, satisfying the condition m<n. In short, this method for rPPG signal Extraction (205) is based on an undercomplete independent component analysis, aiming to estimate the unmixing matrix using a non-linear Cumulative Density Function (CDF) that has been optimized using the customized Levenberg-Marquardt algorithm. An undercomplete independent component analysis-based method equipped with an objective function and optimization procedure, performed rPPG signal extraction. The objective function is based on a mathematical theorem which states that the cumulative density function of the independent component has the maximum entropy, and is mathematically defined as: h(W)= 0.5*log\WSW^Tr\+ E[log(sec²(y))] (5) where W is a matrix, which is used to extract the rPPG signal from a 2-D matrix of mixture signals with n rows equal to the number of image channels and T columns representing length of the image sequence; S is the diagonal matrix containing covariance values of mixture signals; y is the independent component sec represents the secant trigonometric ratio, and Tr represents matrix transpose. The optimization procedure uses a modified Levenberg Marquardt algorithm which ensures the increase in the entropy of cumulative density function of the independent component, iteratively. Each iteration during optimization updates the values of matrix W, defined by: where all the variables in the equation have their annotations as mentioned above.

The estimated matrix W, after successful optimization is linearly combined with mixture signals, to produce an independent component, which in turn represents the remote photoplethysmography signal.

The rPPG signal can be further processed to extract the signal of interest, which is based on the type of physiological sign to be estimated. Subsequently, the frequency to interest is identified from the signal of interest for physiological sign estimation (105).

The electronic device (3) can be any transitory or nonelectronic device with an illumination source but are not limited to laptops and smartphones. The proposed invention can be applied to various healthcare facilities, general wards, ICUs, NICUs etc. for physiological signs estimations or contactless sleep monitoring. Furthermore, this invention also encourages home-based health monitoring by implementing it in subject's bedroom. Furthermore, it can be used in scenarios that avoid physical contact with the subject's body organs, such as burnt or sensitive skin, neonatal physiological signs estimations, contactless sleep monitoring, and driving scenarios during extremely dark conditions .

It is also disclosed that the computer implemented method for estimating at least one physiological sign from a patient in dark environments with luminance of 1 lux or less, comprising steps of image enhancement (102) within a sequence of image frames based on a single image in ambient light of the subject (1), rPPG signal extraction (103) based on undercomplete independent component analysis-based method with entropy based objective function and modified Levenberg-Marquardt algorithm for optimization, signal refinement (104) for extracting the signal of interest, followed by identifying the frequency related to the physiological sign (105).

The disclosed invention extends the application of camera based physiological signs estimations in extremely dark environments with luminance less than 1 lux, which allows physiological signs monitoring during scenarios but not limited to such as sleeping, driving during night time, or clinical scenarios where limited or no luminance lights are permissible.

The herein disclosed method, in one of the proposed embodiments of the invention, may be implemented using one or more processing units, one or more processing devices, any means for processing, such as a processor, a computer or a programmable hardware component being operable with accordingly adapted software. In other words, the herein described method is then executed on one or more programmable hardware components. Such hardware components may comprise but not limited to a general-purpose processor, a Digital Signal Processor (DSP), a micro-controller, etc.

The disclosed invention, illustrated and explained in detail by means of drawings, and the above description, is to be considered exemplary and not restrictive; the invention is not limited to disclosed components. Upon reading the disclosure by means of accompanying drawings and foregoing description of the invention, those skilled in the art will also understand its applications and modifications or variations to the disclosed embodiments in practicing the claimed inventions. Those skilled in the art will understand that these applications and modifications or variations lies within the scope of this invention.

In the appended claims, words like comprising, consisting, or includes does not exclude other possible steps or components, and the indefinite articles "a", and "an" does not exclude plurality. Furthermore, the word "subject" refers to the human being, irrespective of the age, and gender. The components presented by means of drawings are not necessarily, essential, and can be appropriately selected in a manner such that it should not construe the scope of present invention. Any physiological signs in the claims should not be construed as limiting the scope of this invention.

Claims

1. System for estimating at least one physiological sign of a subject (1) in a dark environment, comprising: a capturing device (2), located within a predetermined distance from the subject (1), comprising a detector (21) configured to acquire at least one of a plurality of image frames; an electronic device (3), connected to the capturing device (2), comprising at least an internal memory (31) and a central processing unit (32); and an information display medium (4) connected to the electronic device (3); characterized by collecting a remote photoplethysmography signal from a region of interest of the subject (1), followed by the estimation of at least one physiological sign of said subject (1).

2. System according to the previous claim 1, wherein the capturing device (2) further comprises an illumination source (22) configured to provide a light source with a luminance threshold of at most 1 lux over an exposed region of the subject (1).

3. System according to the previous claims 1 and 2, wherein the detector (21) is configured to detect a visible electromagnetic spectrum from a region of interest of the subject (1), said region of interest comprising a skin region.

4. System according to the previous claims 1 to 3, wherein the visible electromagnetic spectrum comprises a light reflection enabled by the illumination source (22) which is configured to emit different wavelengths of a visible electromagnetic spectrum radiation.

5. System according to previous claim 1, wherein the electronic device (3) is configured to store in the internal memory (31) at least one of the plurality of image frames, said plurality of image frames comprising at least one normal environment light conditions image frame (101a), and consecutive low exposure image frames (101b) with a fixed sampling rate.

6. System according to previous claim 1 and 5, wherein the display medium (4) is configured to display a set of information comprising at least a remote photoplethysmography signal and physiological sign of the subject (1), said set of information being determined and provided by the electronic device (3) after performing a set of image processing steps in the central processing unit (32) based on the stored plurality of image frames in the internal memory (31).

7. Method for estimating at least one physiological sign of a subject (1) in a dark environment, according to any of the previous claims comprising the steps of: acquiring at least one of a plurality of image frames by means of the capturing device (2); storing the acquired at least one of a plurality of image frames into the internal memory (31) of the electronic device (3); applying an image enhancement (102) step to the stored at least one of a plurality of image frames; storing the enhanced plurality of enhanced image frames in the internal memory (31) of the electronic device (3); performing a remote photoplethysmography signal extraction (103) from the enhanced plurality of enhanced image frames; performing a signal refinement (104) to the remote photoplethysmography signal extraction; determining physiological sign estimations (105) based on the resulting signal refinement (104); said image enhancement (102) step, remote photoplethysmography signal extraction (103), signal refinement (104) and physiological sign estimations (105) being performed the central processing unit (32) of the electronic device (3).

8. Method according to the previous claim 7, wherein: the image enhancement (102) step enables the enhancing of the consecutive low exposure image frames (101b) for extracting information related to remote photoplethysmography signal extraction (103); the remote photoplethysmography signal extraction (103) step enables the extraction of the remote photoplethysmography signal from the consecutive low exposure image frames (101b); the signal refinement (104) step enables the extraction of the signal of interest from the remote photoplethysmography signal depending on the physiological sign to estimate; and the physiological sign estimation (105) step, enables the estimation of the desired physiological sign of the subject (1)•

9. Method according to the previous claims 7 and 8, wherein the image enhancement (102) step comprises a deep learning model enabling an output based on the least one normal environment light conditions image frame (101a), and the consecutive low exposure image frames (101b) to be enhanced.

10. Method according to the previous claims 9, wherein the deep learning model comprise two encoders for extracting features from the normal environment light conditions image frame (101a) and the consecutive low exposure image frames (101b) in a hierarchy, and a decoder for enhancing the consecutive low exposure image frames (101b) based on the extracted features from both the encoders using hierarchical and residual connections, respectively .

11. Method according to the previous claims 10, wherein the encoders comprise a sequence of convolutions with Rectified Linear Unit activations, and pooling layers with kernel sizef k ×,k and number of kernels defined by set K_e = {k₁,k₂,...,k_n :k₁ < k₂ < •••< k_n}; and the decoder comprises of sequence of transposed convolutions with Rectified Linear Unit activation and convolution layers with kernel sizefc X k and number of kernels K_d = {k_n,k_n-1,...,k₁ :k_n > k_n-1 > •••> k₁}•

12. Method according to the previous claims 7 to 9, wherein the image enhancement (102) step further comprises a loss function defined as:

Loss=α₁*L1+α₂*MS-SSIM+α₃*L2 which consists of linear combinations of pixel intensity difference (L1), multi-scale structured similarity index (MS-SSIM), and mean square (L2) losses, optimized using an optimization algorithm with a few parameters, and single or multiple stopping criteria.

13. Method according to the previous claims 9 and 10, wherein the deep learning model further enables a trained or optimized output model stored in the internal memory (31) of the electronic device (3), which is used to enhance the image frames (101a, 101b), enhancing low-exposure image frames (101b) based on normal light image (101a).

14. Method according to the previous claims 7 to 9 and 13, wherein the image enhancement (102) step outputs consecutive enhanced image frames (1024) by enhancing the pixel intensities of the plurality of image frames.

15. Method according to the previous claim 14, wherein the enhanced image frames (1024) are stored in the internal memory (31) of the electronic device (3) for posterior processing by the central processing unit (32) to extract the remote photoplethysmography signal corresponding to remote photoplethysmography signal extraction (103).

16. Method according to the previous claims 7 and 15, wherein the central processing unit (32) is configured to execute machine instructions to extract an appropriate region of interest of the subject (1) followed by skin segmentation of the subject (1) from the consecutive enhanced image frames (1024).

17. Method according to the previous claims 7, 8 and 15, wherein the remote photoplethysmography signal extraction (103) comprises at least a detection and tracking of a region of interest from the consecutive enhanced image frames (1024), a skin segmentation (203) from the region of interest (202) using skin pixel extraction models, a spatial averaging of consecutive image frames resulting in n-raw signals generations (204), and a remote photoplethysmography signal extraction (103) using undercomplete independent component analysis based method.

18. Method according to the previous claims 7, 8, 15 and 17, wherein the remote photoplethysmography signal extraction (103) step performs a channel-wise spatial averaging operation and pre-processing to create multiple raw signals, defined by the number of channels of the image, also referred to as mixture signals.

19. Method according to the previous claims 7, 8, 15, 17 and 18, wherein the remote photoplethysmography signal extraction (103) step comprises an undercomplete independent component analysis-based method equipped with an entropy based objective function and optimization procedure, to extract the remote photoplethysmography signal using an assumption of under completeness, leading to one independent component extracted from more than one mixture signals.

20. Method according to the previous claim 19, wherein the undercomplete independent component analysis-based method estimates a matrix, which multiply to the mixture signals produces the remote photoplethysmography signal.

21. Method according to the previous claim 19, wherein the entropy based objective function is based on a cumulative density function of the independent component with maximum entropy, optimized using a modified Levenberg-Marquardt algorithm for entropy maximization to estimate a matrix.

22. Method according to the previous claim 19, wherein the optimization procedure is performed iteratively on the matrix based on entropy maximization of the cumulative density function of the independent component, until the convergence of the objective function.

23. Method according to the previous claims 20 to 22, wherein a linear combination of the optimized matrix and a mixture of signals produces the remote photoplethysmography signal.

24. Method according to the previous claim 23, wherein the remote photoplethysmography signal will be further refined to extract the signal of interest based on physiological sign to be estimated.

25. Method according to the previous claim 24, wherein the physiological sign can be extracted from the signal of interest by extracting the desired frequency from the signal.

26. Data processing system, characterized by comprising the physical means necessary for the execution of the system and method described in any one of the preceding claims.

27. Computer program, characterized by comprising programming code or instructions suitable for operating the system and executing the method described in any of the previous claims, in which said computer program is stored, and is executed in a data processing system, remote or insite, for example a server, performing the respective steps described in the claims.

28. Computer readable physical data storage device, in which the programming code or instructions of the computer program are stored.