KR20240021000A - Method and Apparatus for measuring HR information based on Vision System - Google Patents

Abstract

A vision system-based heart rate information measurement method and device are described. Method for measuring heart rate information: taking a facial image of a subject using a camera; Detecting facial fine movement data by tracking facial feature points from the facial image at a random sampling frequency by an image processing unit; extracting heart rate (HR) data by detecting RRI (R-peak to R-peak interval) from the facial fine movement data by a signal processor; Extract HRV variables from the HR data, at least one of SDNN (standard deviation normal to normal) and rMSSD (root mean square of successful differences) in the time domain, or TP (Total Power), lnTP, and VLF (in the frequency domain) Very Low Frequency), extracting _the value of _{at least} one HRV variable _{( X S} , Values of independent variables such as SDNN, rMSSD, TP, lnTP _{, VLF} , _lnVLF , etc. Target values of dependent variables for each of the above independent variables by regression equation from _{X S ,} It includes calculating Y _S , Y _M , Y _T , Y _LT , Y _V and Y _LV .

Description

Vision-based heart rate information measuring method and device {Method and Apparatus for measuring HR information based on Vision System}

This disclosure relates to a method and device for measuring heart rate information based on a vision system, and more specifically, to a method and device for measuring heart rate information that can be applied to devices with limited resources such as mobile devices.

Heart rate is an important and representative indicator for monitoring cardiac activity. Heart rate is traditionally measured by skin-contact sensors such as Electrocardiography (ECG), Photoplethysmography (PPG), and Ballistocardiography (BCG). Although these methods are standardized and accurate measurement methods, they are difficult to wear for long periods of time due to the discomfort caused by attaching the sensor, skin damage, and measurement burden.

With the development of vision technology, that is, image processing technology, heart rate can now be measured from facial images without skin contact, making it possible to measure heart rate in everyday life.

Despite the convenience of image-based measurement methods, their application in everyday life is still limited for several reasons. In image-based measurement methods, the main research issues are improving the signal to noise ratio (SNR) and overcoming noise from illumination variance and motion artifacts.

Existing image-based measurement methods commonly include three steps: 1) Micro-Movement Measurement, 2) Feature Extraction, and 3) Heart Rate Estimation.

To solve the problem of noise caused by lighting changes and motion noise, a method is used to minimize noise generation in the fine motion measurement stage and feature extraction stage, and in the heart rate inference stage, heart rate is inferred from the frequency value that accounts for the largest proportion of the entire frequency band. A method was presented. However, these existing methods still show limitations in overcoming noise.

Additionally, existing image processing technology requires processing speed and memory for image processing. The frequency analysis interval for extracting HRV (heart rate variability) variables through image processing and data processing is 0.04Hz to 0.4Hz, and the sampling frequency is set to at least 200Hz.

Increasing the sampling frequency like this is for accurate signal restoration, but it is necessary to adjust the sampling frequency appropriately to maintain the performance of the analysis system and manage resources.

In a high-performance computer environment, very high frequency sampling from heart rate information is possible, but since mobile devices such as mobile phones have limited hardware resources such as signal processing speed and data storage capacity limits, the sampling frequency is higher than that in a high-performance computer environment. There is a limit to increasing .

The present disclosure presents an image-based heart rate information extraction method and device that can effectively extract heart rate information such as HR and HRV from low frame rate image information in a hardware environment with limited resources.

Method for measuring heart rate information according to exemplary embodiment:

Taking a facial image of a subject using a camera;

Detecting facial fine movement data by tracking facial feature points from the facial image at a random sampling frequency by an image processing unit;

extracting heart rate (HR) data by detecting RRI (R-peak to R-peak interval) from the facial fine movement data by a signal processor;

HRV variables are extracted from the HR data, including SDNN (standard deviation normal to normal) and rMSSD (root Mean Square of Successive Differences) in the time domain, and TP (Total Power), lnTP, and VLF (Very Low Frequency) in the frequency domain. extracting the value of at least one HRV variable (X _S , X _M , X _T , X _LT , X _V );

From the values _of independent variables such as _SDNN , rMSSD, _TP , _lnTP , and _VLF , It includes calculating _S , Y _M , Y _T , Y _LT and Y _V.

<Regression formula>

Target value of SDDN (Y _S ) = 1.802 * X _S ±0.141

Target value of rMSSD (Y _M ) = 6.173 * X _M ±0.741

Target value of TP (Y _T ) = 2.981 * X _T ± 0.346

Target value of lnTP (Y _{LT )} =1.131

Target value of VLF (Yv) =1.019 _*

According to another embodiment of the present disclosure,

The independent variable includes lnVLF, and the target value (Y _LV ) of lnVLF can be calculated from the value of lnVLF (X _LV ) by the regression equation below.

Y _LV =0.888 * X _LV ±0.127

According to a specific embodiment of the present disclosure, the method:

Detecting PPG data from a subject and extracting HR reference data from it;

Extracting frequency-specific relative power density (RPD) data from the HR reference data as a reference RPD;

Clustering the HR reference data into multiple clusters; and

The method may further include using the plurality of clusters to form a heart rate determination rule base for inferring the heart rate based on a reference RPD corresponding to the HR reference data.

The heart rate measurement method according to the exemplary embodiment may further include extracting a component showing high periodicity from the fine motion data as fine motion data for final use.

In the heart rate measurement method according to the exemplary embodiment, the RPD can be calculated using the equation below.

From above,

PS _s : Power spectrum in frequency band for time series signal s

P _total : sum of the entire power spectrum

Heart rate measurement method according to an exemplary embodiment: configure a data set by assigning the HR reference data to the RPD as a label, perform clustering on the data set, and distribute the labels of each cluster If the kurtosis value of is greater than or equal to the normal distribution, it can be registered as a rule-base.

Heart rate measurement method according to an exemplary embodiment: converts the fine movement data into frequency series data through frequency analysis, extracts the power spectrum in a predetermined range of frequency bands, and extracts the power of each frequency for the power spectrum. The RPD can be calculated by normalizing the value by dividing it by the sum of the entire power spectrum.

A method for measuring heart rate according to an exemplary embodiment: extracting fine movement data from a subject to be measured; extracting relative power density (RPD) data in a predetermined frequency band from the fine motion data as a comparison RPD; Obtaining similarity by calculating the distance of the comparison RPD to the reference RPD; The method may further include inferring the HR of the subject to be measured using the similarity.

A vision system-based heart rate information measuring device according to an exemplary embodiment: a camera that acquires a face image from a subject to be measured, an image processor that processes the face image to extract fine movement data; It may include a process device that extracts RPD from the fine movement data and determines the HR of the subject based on the rule base.

In the vision system-based heart rate information measuring device according to the exemplary embodiment, the process device: extracts relative power density (RPD) data in a predetermined frequency band from facial fine movement data as a comparison RPD, and determines the comparison RPD for the reference RPD. Similarity can be obtained by calculating the distance, and the HR of the subject to be measured can be inferred using the similarity.

The vision system-based heart rate information measuring device according to the exemplary embodiment may further include an ECG sensor for measuring an electrocardiogram from the subject or a PPG sensor for measuring PPG.

Figure 1 shows a heart rate measurement process according to an exemplary embodiment.
Figure 2 is a flowchart of the first stage, Micro-Movement Measurement, which is a process for measuring micro-movements from a face image.
Figure 3 shows an application example of the step-by-step process of Figure 2.
Figure 4 is a flowchart of the second feature extraction step, which is a process for extracting features from fine movements.
Figure 5 shows an application example of the step-by-step process of Figure 4.
Figure 6 is a flowchart of the third heart rate estimation (Heart Rate Estimation) step, which is a process for inferring heart rate from features.
Figure 7 is a flowchart of the process of extracting labels from PPG to construct a data set used for rule-base learning.
Figure 8 shows an application example of the step-by-step process of Figure 7.
Figure 9 illustrates the structure of a data set constructed according to an exemplary embodiment.
Figure 10 is a flowchart of the process of learning a rule-base from a data set according to an exemplary embodiment.
Figure 11 shows an application example of the step-by-step process of Figure 10.
Figure 12 illustrates the structure of a rule-base learned according to an exemplary embodiment.
Figure 13 shows an application example of a process for inferring heart rate using a rule-base learned according to an exemplary embodiment.
Figure 14 shows regression analysis results for HRV variables of raw ECG data (ECG 500/w) and down-sampling it to 50 and 10 frames/sec by sliding window.
Figures 15 to 27 are raw data charts of HRV variables used in the regression analysis.

Hereinafter, an image-based HRV measurement method and system according to exemplary embodiments will be described in detail with reference to the attached drawings.

The image-based heart rate measurement method according to the exemplary embodiment can be executed on a heart rate measurement device based on a mobile device such as a PC or a mobile phone, and the application structure and method will be described below with a focus on the PC.

The heart rate information measuring device includes a camera that photographs the subject's face, an ECG sensor that measures the subject's electrocardiogram or a PPG sensor that measures PPG, and an image processor and a signal processor that process signals from the camera and the PPG sensor. .

An exemplary embodiment infers heart rate based on images acquired using a camera and extracts HRV variables in the time domain and frequency domain.

As shown in Figure 1, the signal processing process for measuring heart rate includes Micro-Movement Measurement as the first step, Feature Extraction as the second step, and Heart Rate Inference as the third step ( Heart Rate Estimation) step, the fourth step is extraction of HRV variables in the time domain (time series) and frequency domain, and the fifth step is a correction step for the detected time series variable values and frequency series variable values. Below, the data processing process at each stage is explained.

Figure 2 is a flowchart of the first stage Micro-Movement Measurement, which is a process for extracting micro-movements containing components caused by cardiac responses from facial images, and Figure 3 is a flowchart of Figure 2. An example of application of the step-by-step process is shown.

1) Facial video acquisition: The upper body or head, including the subject's face, is filmed at approximately 30fps using a video camera.

2) Face Detection: Detects the face area from a head image or face image. For example, the face area can be detected using the Viola-Jones algorithm, which uses the light and dark characteristics that appear in each part of the face.

3) Area Selection: Select the forehead and nose areas that minimize noise caused by facial expressions from the detected face area.

4) Facial Feature Point Detection: Detects points that are good for tracking compared to other points from the selected area. In the experiment of this embodiment, feature points were detected using the GFTT (Good Feature to Track) algorithm (1994).

5) Facial Feature Point Tracking: Tracks the movement of each detected feature point. The KLT (Kanade-Lucas-Tomasi) feature tracking algorithm can be applied to such feature tracking.

6) Micro-Movement Measurement: Track the movement in the image X-Y plane of each feature point tracked in the current frame compared to the previous frame, that is, the moving distance (value) of the Extract Micro-Movement Data (MMD). The sliding window technique can be applied here, in which case the Window Size can be set to 30 seconds and the Interval Size can be set to 1 second. The KLT algorithm that can be used for feature tracking can measure fine motion data (MMD) equal to the number of feature points detected in facial feature point detection.

The fine motion data obtained as above goes through a feature extraction process as shown in FIG. 4.

In the feature extraction process, components based on cardiac response are extracted. In this process, only the features based on cardiac response are extracted from the extracted micro-motion data. Figure 4 is a flowchart of the feature extraction step, and Figure 5 shows an application example of the step-by-step process of Figure 4.

1) Bandpass Filtering: For each fine movement data, only the frequency band corresponding to the heart rate band is extracted using a Butterworth bandpass filter (5 Order, 0.75-5Hz).

2) Principal Component Analysis: A process for extracting one fine motion data with the same component from the fine motion data extracted from each feature point. Five components are extracted through principal component analysis. Using the characteristic that biological signals have periodicity for each component, the component showing the highest periodicity is extracted as fine movement data (MMD) related to the final heart rate. Periodicity ( PDs ) is calculated through the following process.

In the above equation, s is a time series signal, FFT is a time series signal ( s ), a Fourier analysis method for converting micromotion data (MMD) into a frequency band, and PS _s is the power spectrum of the frequency band for the time series signal s .

In the above equation, P _max means the largest power value in the power spectrum of the corresponding frequency band for the time series signal s .

In the above equation, P _total means the sum of the entire power spectrum of the corresponding frequency band for the time series signal s.

In the above equation, PD _s represents the periodicity of the time series signal s.

It is possible to extract RRI data in the 0.75 to 2.5 Hz band through FFT analysis of the above fine motion data (MMD).

Using the RRI data, time domain variables such as SDNN (standard deviation normal to normal) and rMSSD (Root Mean Square of Successive Differences) are calculated, and the RRI is sampled at 2Hz and analyzed through FFT (fast fourier transform). To extract variables in the frequency domain, the HRV (heart rate variability) spectrum, that is, the power spectrum corresponding to the heart rate band, is extracted.

The HRV spectrum in the frequency domain extracts the power of VLF (very low frequency, 0.0033-0.04 Hz), LF (low frequency, 0.04-0.15 Hz), and HF (high frequency, 0.15-0.4 Hz) bands, respectively, and the frequencies are calculated from these. Domain variables VLF/HF, LF/HF, lnVLF, lnLF, lnHF, TP (total power), lnTP, etc. can be calculated.

In this embodiment, HRV variables are extracted from the HR data, at least one of SDNN (standard deviation normal to normal) and rMSSD (root mean square of successful differences) in the time domain, or TP (Total Power) in the frequency domain. _, lnTP, and _VLF , extract the value of _at least one HRV variable ( _{X S} , _{From S , X M , _ _ _ _}

Target value of SDDN (Y _S ) = 1.802 * X _S ±0.141

Target value of rMSSD (Y _M ) = 6.173 * X _M ±0.741

Target value of TP (Y _T ) = 2.981 * X _T ± 0.346

Target value of lnTP ( _{Y LT)} = 1.131*

Target value of VLF (Yv) = 1.019 _*

The above regression equation was obtained through multi-stage down sampling of the reference signal obtained from the PPG signal and regression analysis thereof.

Figure 14 shows the results of regression analysis of HRV variables applying analysis through a sliding window of raw ECG data (ECG 500/w) and data downsampled at 50 and 10 frames/sec, and Figures 15 to 27 show the regression results This is a raw data chart of the HRV variables used in the analysis.

As shown in Figure 14, as a result of the regression analysis, among the many HRV variables analyzed, SDNN, rMSSD, VLF, lnVLF, TP, and lnTP showed significant and valid results despite down-sampling. The tolerance added to the regression equation follows the standard error.

By extracting HRV variables from facial fine movement data using the method described above and correcting each HRV variable value using the above regression equation, the final values of the HRV variables can be obtained.

According to this embodiment, it is possible to extract valid HRV variable values even from facial movement images of low frame rates.

The following describes the process of extracting RRI from a machine learning technique and extracting HRV from the RRI obtained here, according to the present disclosure.

3) Relative Power Density Extraction: The fine movement data, which is a time series signal, is converted into frequency series data by FFT analysis as described above, and the 0.75 to 2.5 Hz frequency corresponding to the heart rate band is extracted from the frequency series data. Extract the power spectrum of the band.

For the extracted power spectrum, each power value is normalized by dividing it by the sum of the entire power spectrum, and RPD (Relative Power Density) is extracted as a feature for heart rate inference.

The equation below calculates the RPD for the time series signal _S.

In the above equation, PS _s refers to the power spectrum of the frequency band for the time series signal s , and P _total refers to the sum of the power spectra.

The Heart Rate Estimation step describes the process for inferring the heart rate from the final extracted features. Figure 6 is a flow chart of the Heart Rate Estimation step.

1) Rule-Base Training: A process for learning the Rule-Base for heart rate inference. RPD extracted from fine movement signals and the corresponding PPG signal (actually measured with PPG sensor) ) is used. RPD and PPG signals to build the rule-base are measured with various subjects and under various environments.

Referring to Figures 7 and 8, the PPG signal is filtered using a Butterworth bandpass filter (2 Order, 0.75-2.5Hz), and PPI is obtained using peak detection from this. Calculate (Peak-to-Peak Interval) and infer HR (heart rate). The HR extracted from this is assigned a label, and the corresponding RPD is used as a feature to form a data set. Figure 9 illustrates the structure of the data set constructed in the above process.

Referring to Figures 10 and 11, clustering is performed on the constructed data set, and if the kurtosis value of the label distribution of each cluster is greater than the normal distribution (Gaussian), it is registered as a rule base. .

In this clustering, cluster membership is determined by extracting the center point (Centroid value) of each cluster and calculating the distance of the RPD to the centroid value. The clustering can use the Fuzzy C-Means algorithm.

In clustering, it is important to set the K value, which is a parameter that determines how many clusters the data will be grouped into. Here, since the appropriate K value is unknown, K is performed by increasing it one by one starting from 3. Since the rule base must be able to infer at least 50 to 130 bpm, clustering is performed until K exceeds 80. Finally, after clustering is completed, the cluster with the highest kurtosis value for each label is defined as the final rule base.

The rule-base obtained in this way has a relative power density (RPD) of different sizes in the third direction on a two-dimensional plane consisting of the heart rate frequency (Frequency Range) and the BPM range (BPM Range), as shown in Figure 12. It has a distributed structure.

2) Heart Rate Estimation: Figure 13 shows the training process and Figure 12 shows the process for determining or inferring the heart rate using the learned rule base. In order to determine or infer heart rate using the learned rule base, the extracted RPD is compared with the rule base, and the label value of the rule base showing the most similar pattern is determined as the final heart rate. The Euclidian Distance, which can be used as similarity for determining the heart rate, is calculated as follows.

In the above equation, RPD _real is RPD input in real time, RPD _rule means rule-based RPD. According to the above equation, similarity (DS) is obtained from the distance between the RPD input in real time and the rule-based RPD.

If there is one RRI value per input obtained by the rule base as above, time series RRI data is obtained by accumulating this RRI through all input data.

From the RRI data obtained in this process, time series SDDN and frequency are analyzed, and various HRV variables as described above are output through this. For the output HRV, the output target value of each variable is determined according to the above-described regression equation, and the final HRV variable is obtained.

In extracting heart rate estimation and HRV using a vision system, the present invention does not simply infer the heart rate from the dominant frequency in the frequency band of fine movement data, but uses a machine learning method to estimate the heart rate in the frequency band. It infers heart rate based on patterns. As a machine learning method, it is possible to build a rule-base for heart rate inference that excludes the influence of noise by using a method of extracting patterns by clustering data. This invention is more practical because it requires less computational cost and makes it easier to find common patterns compared to the neural network model being studied recently.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely exemplary, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible. Therefore, the true scope of protection of the present invention should be determined only by the appended claims.

Claims (12)

Taking a facial image of a subject using a camera;
Detecting facial fine movement data by tracking facial feature points from the facial image at a random sampling frequency by an image processing unit;
extracting heart rate (HR) data by detecting RRI (R-peak to R-peak interval) from the facial fine movement data by a signal processor;
HRV variables are extracted from the HR data, at least one of SDNN (standard deviation normal to normal) and rMSSD (root mean square of successful differences) in the time domain, or TP (Total Power), lnTP, and VLF ( Extracting the value of at least one HRV variable ( _{X S ,} X _M , X _T ,
From the values _of independent variables such as _SDNN , rMSSD, _TP , _lnTP , and _VLF , A vision system-based heart rate information measurement method comprising calculating _S , Y _M , Y _T , Y _LT , and Y _V.
Target value of SDDN (Y _S ) = 1.802 * X _S ±0.141
Target value of rMSSD (Y _M ) = 6.173 * X _M ±0.741
Target value of TP (Y _T ) = 2.981 * X _T ± 0.346
Target value of lnTP ( _{Y LT)} =1.131*
Target value of VLF (Yv) =1.019 _* According to paragraph 1,
The independent variables include lnVLF,
A vision system-based heart rate information extraction method that calculates the target value of lnVLF (Y _LV ) from the value of lnVLF (X _LV ) by the regression equation below.
Y _LV =0.888 * X _LV ±0.127 According to paragraph 1,
Detecting PPG data from a subject and extracting HR reference data from it;
Extracting frequency-specific relative power density (RPD) data from the HR reference data as a reference RPD;
Clustering the HR reference data into multiple clusters; and
Using the plurality of clusters, forming a heart rate determination rule base for inferring the heart rate based on a reference RPD corresponding to the HR reference data. According to paragraph 3,
A vision system-based heart rate information measurement method in which the RPD is calculated by the equation below.

From above,
PS _s : Power spectrum in frequency band for time series signal s
P _total : sum of the entire power spectrum According to paragraph 3,
Construct a data set by assigning a label to the HR standard data,
A vision system-based heart rate information measurement method that performs clustering on the data set and registers it as a rule base if the kurtosis value of the label distribution of each cluster is greater than or equal to the normal distribution (Gaussian). According to paragraph 2,
Convert the facial fine movement data into frequency series data through frequency analysis, extract the power spectrum in a predetermined range of frequency bands, and
A vision system-based heart rate information measurement method that calculates the RPD by dividing the power value of each frequency with respect to the power spectrum by the sum of the entire power spectrum and normalizing it. According to paragraph 2,
extracting relative power density (RPD) data in a predetermined frequency band from the facial fine movement data as a comparison RPD;
Obtaining similarity by calculating the distance of the comparison RPD to the reference RPD; and
A vision system-based heart rate information measuring method further comprising: inferring the HR of the subject to be measured using the similarity. In clause 7,
A vision system-based heart rate information measurement method that calculates RRI data from the inferred HR of the subject to be measured and extracts HRV variables in the frequency domain through frequency analysis of the RRI. According to any one of claims 3 to 6,
A vision system-based heart rate information measurement method that calculates RRI data from the inferred HR of the subject to be measured and extracts HRV variables in the frequency domain through frequency analysis of the RRI. In the vision system-based heart rate information measuring device that performs the method according to any one of claims 1 to 8,
A camera that acquires facial images from the subject being measured,
an image processing unit that processes the face image to extract facial fine movement data; and
From the facial fine movement data, at least one of SDNN (standard deviation normal to normal) and rMSSD (root mean square of successful differences) in the time domain, or TP (Total Power), lnTP, and VLF (Very Low Frequency) in the frequency domain ) A process device for extracting the target values _{Y S} , Y _M , Y _T , Y _LT , and Y _{V of} at least one of the HRV _variable values ( _{X S} , A vision system-based heart rate information measuring device including; According to clause 10,
The above process device:
Detect PPG data from the subject, extract HR reference data from it, and extract relative power density (RPD) data by frequency from the HR reference data as reference RPD,
Clustering the HR reference data into multiple clusters,
Using the plurality of clusters, a heart rate determination rule-base is formed for inferring the heart rate by a reference RPD corresponding to the HR reference data,
Relative power density (RPD) data in a predetermined frequency band is extracted from the facial fine movement data as a comparison RPD, a similarity is obtained by calculating the distance of the comparison RPD with respect to the reference RPD, and the measurement object is measured using the similarity. inferring the subject's HR;; Vision system-based heart rate information measuring device. According to clause 11,
A vision system-based heart rate information measuring device further comprising a PPG sensor for measuring PPG from the subject or an ECG sensor for measuring electrocardiogram.