KR102598219B1 - Method and apparatus for restoring remote photoplethysmography - Google Patents

Abstract

The present invention relates to a pulse wave detection method, and more specifically, to a method and device for restoring a non-contact pulse wave signal to a contact pulse wave signal using deep learning technology. According to an embodiment of the present invention, a non-contact pulse wave signal restoration method and device can obtain a heart rate signal in real time. Additionally, the non-contact pulse wave signal restoration method and device can measure a more accurate heart rate signal compared to before non-contact signal restoration. In addition, the non-contact pulse wave signal restoration method and device can analyze the morphological meaning of the pulse wave signal by restoring the shape of the pulse wave signal more clearly than before non-contact signal restoration.

Description

Non-contact pulse wave signal restoration method and device {METHOD AND APPARATUS FOR RESTORING REMOTE PHOTOPLETHYSMOGRAPHY}

The present invention relates to a pulse wave detection method, and more specifically, to a method and device for restoring a non-contact pulse wave signal to a contact pulse wave signal using deep learning technology.

Heart rate is a major biosignal that can be used in healthcare and emotional services. Most devices that measure heart rate are based on skin-contact sensors such as electrocardiography (ECG), photoplethysmography (PPG), and ballistocardiography (BCG). However, contact sensors are not convenient due to measurement burdens such as psychological burden and skin damage on the subject.

Recently, several methods have been proposed to measure heart rate in a non-contact manner by photographing the face with a camera.

Camera-based heart rate measurement methods so far include methods using photoplethysmography (PPG) and ballistic heart rate (BCG). Photoplethysmography (PPG) measures the amount of blood flow that fluctuates due to heartbeat from changes in facial color. Ballistic cardiography (BCG) measures the trajectory of the heartbeat transmitted through the carotid artery from subtle facial tremors.

In the conventional heart rate measurement method, the arc pulsation component is much smaller than the contact-based method, and the shape is blurred, so only heart rate information can be obtained, so there is a problem in that information on the shape of the pulse wave (PPG) signal cannot be obtained.

1. Republic of Korea Patent Registration No. 10-2215557 “Camera-based heart rate measurement method and system using facial color and tremor” (Registration date: February 5, 2021)

The present invention provides a non-contact pulse wave signal restoration method and device for obtaining a non-contact pulse wave signal in real time through a face image as an image, then restoring the signal to the level of a contact pulse wave signal and performing morphological analysis as well as heart rate of the pulse wave signal. .

According to one aspect of the present invention, a non-contact pulse wave signal restoration method and device are provided.

A non-contact pulse wave signal restoration device according to an embodiment of the present invention includes an input unit that collects contact pulse wave signals and non-contact pulse wave signals from a subject, a pre-processing unit that pre-processes the collected contact pulse wave signals and non-contact pulse wave signals, and a pre-processed contact pulse wave signal and non-contact pulse wave signal. It may include a learning unit that generates a learning model based on SVR and deep learning for the pulse wave signal, and an output unit that restores the non-contact pulse wave signal in response to the contact pulse wave signal based on the generated learning model.

According to one aspect of the present invention, a non-contact pulse wave signal restoration method and a computer program for executing the same are provided.

A non-contact pulse wave signal restoration method according to an embodiment of the present invention includes collecting contact pulse wave signals and non-contact pulse wave signals from a subject, preprocessing the collected contact pulse wave signals and non-contact pulse wave signals, and preprocessing the contact pulse wave signals and non-contact pulse wave signals. It may include generating a learning model based on SVR and deep learning, and restoring the non-contact pulse wave signal in response to the contact pulse wave signal based on the generated learning model.

According to an embodiment of the present invention, a non-contact pulse wave signal restoration method and device can obtain a heart rate signal in real time.

Additionally, the non-contact pulse wave signal restoration method and device can measure a more accurate heart rate signal compared to before non-contact signal restoration.

In addition, the non-contact pulse wave signal restoration method and device can analyze the morphological meaning of the pulse wave signal by restoring the shape of the pulse wave signal more clearly than before non-contact signal restoration.

1 and 2 are diagrams to explain a non-contact pulse wave signal restoration method according to an embodiment of the present invention.
Figure 3 is a diagram for explaining morphological information of a contact pulse wave signal according to an embodiment of the present invention.
Figure 4 is a diagram for explaining the restoration result of a non-contact pulse wave signal according to an embodiment of the present invention.
Figure 5 is a diagram for explaining a non-contact pulse wave signal restoration device according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail through detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the present invention. In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Additionally, as used in this specification and claims, the singular expressions “a,” “a,” and “an” should generally be construed to mean “one or more,” unless otherwise specified.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, identical or corresponding components will be assigned the same drawing numbers and redundant description thereof will be omitted. Do this.

1 and 2 are diagrams to explain a non-contact pulse wave signal restoration method according to an embodiment of the present invention.

Referring to FIG. 1, the non-contact pulse wave signal restoration device is a non-contact pulse wave signal restoration method that utilizes deep learning on pulse waves measured non-contactly from the subject (rppg: remote photoplethysmography) and pulse waves measured contactly (cppg: contact photoplethysmography). Accordingly, the contact pulse wave signal (cppg) can be restored from the non-contact pulse wave signal (rppg).

At this time, the contact pulse wave signal is a signal that contact-measures the change in skin light absorption due to the change in blood flow, that is, the blood flow generated within the blood vessel by the pressure generated by the heartbeat.

Specifically, in step S10 of the non-contact pulse wave signal restoration method, the non-contact pulse wave signal restoration device may receive the contact pulse wave signal and the non-contact pulse wave signal simultaneously measured by the biometric recognition device.

In step S20, the non-contact pulse wave signal restoration device can remove noise from the collected contact pulse wave signal.

According to an embodiment, when receiving a contact pulse wave signal and a non-contact pulse wave signal measured by a plurality of people during a first period, the non-contact pulse wave signal restoration device does not use all the data corresponding to the first period and the portion without motion noise You can perform noise removal to use only At this time, a finite impulse response (FIR) filter may be used to process high-frequency noise for the contact pulse wave signal.

In step S30, the non-contact pulse wave signal restoration device can separate the noise-processed non-contact pulse wave signal and the contact pulse wave signal in a single cycle unit.

In step S40, the non-contact pulse wave signal restoration device can normalize the non-contact pulse wave signal and the contact pulse wave signal separated in single cycle units to between 0 and 1. The periods of the normalized non-contact pulse wave signal and the contact pulse wave signal correspond to a pair. At this time, Min-Max normalization may be used for normalization.

In step S50, the non-contact pulse wave signal restoration device may extract feature points at equal intervals in time from a single cycle pair of the normalized non-contact pulse wave signal and the contact pulse wave signal.

Depending on the embodiment, the number of feature points extracted by the non-contact pulse wave signal restoration device may be 30.

In step S60, the non-contact pulse wave signal restoration device performs SVR learning and deep learning learning. Specifically, the non-contact pulse wave signal restoration device can perform SVR learning and deep learning learning using 30 extracted feature point pairs as input. At this time, the non-contact pulse wave signal restoration device may generate learning data based on the non-contact pulse wave signal and the contact pulse wave signal through a preprocessing process. Therefore, the non-contact pulse wave signal restoration device extracts 30 corresponding feature points extracted from single cycle data and uses them as learning data for the model.

As shown in Figure 2, the extracted 30 feature point pairs include a non-contact pulse wave signal as input data and a contact pulse wave signal as target data. The non-contact pulse wave signal restoration device can perform learning on SVR and a three-layer deep learning model (S61, S62, S63, S64).

Specifically, the total data used for learning is 4551 pairs, 3640 pairs of training data and 911 pairs of result data (test data) are used to perform SVR learning, and 2912 pairs of training data are used for deep learning learning. (train data), 911 pairs of result data (test data), and 728 pairs of validation data were used.

The parameters of the SVR used in the non-contact pulse wave signal restoration device are the parameter cost for the error tolerance range is 50, the parameter degree for the order of classification is 4, and the kernel function used is poly. can be used.

Deep learning used in non-contact pulse wave signal restoration devices consists of three layers (Dense), and the number of units in each layer can be 16. He normal distribution can be used for weight initialization, and ELU (Exponential Linear Units), which solves the Dying ReLU problem while containing all the advantages of the existing ReLU (Rectified Linear Unit), can be used as the activation function, and L2 weight regulation is added to allow for excessive Conformity can be prevented.

Therefore, the non-contact pulse wave signal restoration device can generate 30 output data through SVR and a simple three-layer deep learning model.

Referring again to FIG. 1, in step S70, the non-contact pulse wave signal restoration device may restore the non-contact pulse wave signal to have a form close to the contact pulse wave signal based on the generated learning model. Through this, it will be possible to obtain medically meaningful morphological information as well as heart rate information through non-contact pulse wave signals.

Figure 3 is a diagram for explaining morphological information of a contact pulse wave signal according to an embodiment of the present invention.

Referring to FIG. 3, the form of the pulse wave signal (ppg) includes morphological information about aging, high blood pressure, and arteriosclerosis. That is, the non-contact pulse wave signal restoration device can obtain morphological information by analyzing the contact pulse wave signal (cppg) and the non-contact pulse wave signal (rppg).

As shown in FIG. 3, aging/hypertension/arteriosclerosis can be determined based on the data of the pulse wave signal (ppg), such as extension of SX time, decrease in time between P and D, increase in height of C and D, loss of D, etc. You can.

In addition, by observing a decrease in C height after alcohol administration in the pulse wave signal (ppg) data, it can be determined that C height corresponds to vasodilation. From the pulse wave signal (ppg) data, it can be determined that D generally corresponds to a person with healthy arterial elasticity. From the data of the pulse wave signal (ppg), it can be seen that the attenuation of the waveform is related to blood vessel stiffness. From the pulse wave signal (ppg) data, it can be determined that the SX time is extended in patients with vascular disease and high blood pressure.

Therefore, the non-contact pulse wave signal restoration device can obtain not only heart rate information but also morphological information by restoring the non-contact pulse wave signal (rppg) like the contact pulse wave signal (cppg). At this time, the morphological information may be information about aging, high blood pressure, and arteriosclerosis included in the form of the pulse wave signal.

Figure 4 is a diagram for explaining the restoration result of a non-contact pulse wave signal according to an embodiment of the present invention.

Referring to FIG. 4, the first signal 410 is an input non-contact pulse wave signal (rppg), the second signal 420 is an input contact pulse wave signal (cppg), and the third graph 430 shows the SVR and dip Shows the restoration results according to running.

The present invention can collect restoration results having a shape adjacent to the contact pulse wave signal (cppg) by processing the non-contact pulse wave signal (rppg), whose shape is blurred compared to the contact pulse wave signal (cppg), using SVR and deep learning learning as above.

The non-contact pulse wave signal restoration device used Cosine similarity and Pearson correlation coefficient to verify the similarity between the second signal 420 and the third signal 430.

Here, Cosine similarity is the similarity of two vectors that can be obtained using the cosine angle between the two vectors, and is a value that determines the similarity in direction rather than the size of the vectors. Additionally, the Pearson correlation coefficient is a number that quantifies the linear correlation between two variables, X and Y, and is a value standardized by dividing the covariance of the two variables by the product of their respective standard deviations.

The results of using Cosine similarity and Pearson correlation coefficient to evaluate the signal similarity of the non-contact pulse wave signal restoration device are shown in Table 1 below.

Similarity evaluation index Before restoration After restoration Cosine similarity About 89.4% About 94.5% Pearson correlation coefficient About 69.5% About 84.8%

As a result, cosine similarity is about 89.4% before restoration and about 94.5% after restoration. The Pearson correlation coefficient is about 69.5% before restoration and about 84.8% after restoration.

In addition, the results of evaluating the performance of the non-contact pulse wave signal restoration device by restoring a new non-contact pulse wave signal (rppg) that was not used for learning into the learned model are shown in Table 2 below.

Similarity evaluation index Before restoration After restoration Cosine similarity About 87.9% About 95.6% Pearson correlation coefficient About 63.7% About 88%

As a result, cosine similarity is about 87.9% before restoration and about 95.6% after restoration. The Pearson correlation coefficient is about 63.7% before restoration and about 88% after restoration.

Referring to Tables 1 and 2 above, it can be seen that the PPG signal 430 restored by the non-contact pulse wave signal restoration method according to the present invention has been restored to have a similar form to the contact pulse wave signal 420.

Figure 5 is a diagram for explaining a non-contact pulse wave signal restoration device according to an embodiment of the present invention.

Referring to FIG. 5 , the non-contact pulse wave signal restoration device may include an input unit 100, a pre-processing unit 200, a learning unit 300, and an output unit 400.

The input unit 100 may collect a contact pulse wave signal and a non-contact pulse wave signal from the subject.

The input unit 100 may receive a contact pulse wave signal and a non-contact pulse wave signal from a plurality of biometric devices.

The plurality of biometric devices may include a first authentication device and a second authentication device. Each biometric authentication device includes sensors that recognize different biometric information, enabling multiple biometric authentication.

The first authentication device is a device that includes a contact biometric sensor, and the second authentication device is a device that includes a camera and can take pictures of the user. Depending on the embodiment, the first authentication device may be a device that can recognize the heartbeat with a wearable biometric sensor attached to a wristband or watch, and the second authentication device may be a device that recognizes the heartbeat of the user through photography with a camera. It can be a device that measures fluctuating blood flow from changes in facial color.

The preprocessing unit 200 may preprocess the collected contact pulse wave signals and non-contact pulse wave signals.

The preprocessor 200 may remove noise from the collected contact pulse wave signal.

The preprocessor 200 may separate the noise-processed non-contact pulse wave signal and the contact pulse wave signal in a single cycle unit.

The pre-processing unit 200 can normalize the non-contact pulse wave signal and the contact pulse wave signal separated by a single cycle to a minimum-maximum value between 0 and 1.

The preprocessor 200 can extract 30 corresponding extraction points at equal intervals from the single cycle pair.

The learning unit 300 may generate a learning model based on SVR and deep learning using the preprocessed contact pulse wave signal and the non-contact pulse wave signal.

The learning unit 300 may perform SVR learning using the feature point pairs extracted by the preprocessor 200 as input.

Depending on the embodiment, in SVR learning, the parameter cost may be set to 50, the degree may be set to 4, and the kernel function may be set to poly.

The learning unit 300 can perform learning through a deep learning model composed of three layers using the output of SVR learning as input.

Depending on the embodiment, the deep learning model has three layers, the number of units in each layer is 16, He normal distribution is used for weight initialization, ELU is used for the activation function, and L2 weights are used to prevent overfitting. Regulations can be used.

The output unit 400 may restore the non-contact pulse wave signal corresponding to the contact pulse wave signal based on the learning model generated by the learning unit.

The above-described non-contact pulse wave signal restoration method may be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). You can. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device through a network such as the Internet, installed on the other computing device, and thus used on the other computing device.

In the above, even though all the components constituting the embodiment of the present invention have been described as being combined or operated in combination, the present invention is not necessarily limited to this embodiment. That is, within the scope of the purpose of the present invention, all of the components may be operated by selectively combining one or more of them.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. You must understand that it exists.

So far, the present invention has been examined focusing on its embodiments. A person skilled in the art to which the present invention pertains will understand that the present invention may be implemented in a modified form without departing from the essential characteristics of the present invention. Therefore, the disclosed embodiments should be considered from an illustrative rather than a restrictive perspective. The scope of the present invention is indicated in the claims rather than the foregoing description, and all differences within the equivalent scope should be construed as being included in the present invention.

100: input unit
200: Preprocessing unit
300: Learning Department
400: output unit

Claims (9)

an input unit that collects from the subject a contact pulse wave signal obtained from a wearable biometric sensor and a non-contact pulse wave signal obtained from a face image captured by a camera;
A preprocessing unit that preprocesses the collected contact pulse wave signals and non-contact pulse wave signals;
A learning unit that generates a learning model based on SVR (Support Vector Regression) and deep learning using the preprocessed contact pulse wave signal and non-contact pulse wave signal; and
It includes an output unit that restores the non-contact pulse wave signal in response to the contact pulse wave signal based on the generated learning model,
The preprocessor,
Applying a Finite Impulse Response (FIR) filter to the collected contact pulse wave signal and non-contact pulse wave signal to remove high frequency noise,
The contact pulse wave signal and the non-contact pulse wave signal from which the noise has been removed are separated into a single period and then normalized to between 0 and 1 based on Min-Max normalization,
Extract and preprocess a predetermined number of feature points at equal intervals according to time from a single period pair of the normalized contact pulse wave signal and the non-contact pulse wave signal,
The learning department,
Performing the SVR learning using the extracted feature points as input,
Perform learning through the deep learning model composed of three layers using the output of the SVR learning as input,
The SVR learning is,
The parameter cost is 50, the degree is 4, and the kernel function is poly.
The deep learning model is,
The number of units on each floor is 16,
He normal distribution is used for weight initialization,
Use ELU for the activation function,
It is characterized by using L2 weight regulation to prevent overfitting,
The output unit,
In the data of the restored contact pulse wave signal, the S Characterized in obtaining morphological information including at least one of aging, hypertension, and arteriosclerosis based on height gain and D loss,
Non-contact pulse wave signal restoration device.
delete delete delete In a method for restoring a non-contact pulse wave signal in a device,
Collecting a contact pulse wave signal obtained from a wearable biometric sensor and a non-contact pulse wave signal obtained from a face image captured by a camera from the subject;
Preprocessing the collected contact pulse wave signals and non-contact pulse wave signals;
Generating a learning model based on SVR (Support Vector Regression) and deep learning using the preprocessed contact pulse wave signal and non-contact pulse wave signal; and
A step of restoring a non-contact pulse wave signal in response to a contact pulse wave signal based on the generated learning model,
The preprocessing step is,
removing high-frequency noise by applying a Finite Impulse Response (FIR) filter to the collected contact pulse wave signal and the non-contact pulse wave signal;
Separating the noise-removed contact pulse wave signal and the non-contact pulse wave signal into a single period and then normalizing them between 0 and 1 based on Min-Max normalization; and
Extracting and preprocessing a predetermined number of feature points at equal intervals according to time from a single period pair of the normalized contact pulse wave signal and the non-contact pulse wave signal;
Including,
The step of creating the learning model is,
performing the SVR learning using the extracted feature points as input; and
Performing learning through the deep learning model composed of three layers using the output of the SVR learning as input;
Including,
The SVR learning is,
The parameter cost is 50, the degree is 4, and the kernel function is poly.
The deep learning model is,
The number of units on each floor is 16,
He normal distribution is used for weight initialization,
Use ELU for the activation function,
It is characterized by using L2 weight regulation to prevent overfitting,
The restoration step is,
In the data of the restored contact pulse wave signal, the S Obtaining morphological information including at least one of aging, hypertension, and arteriosclerosis based on height gain and D loss;
Containing more,
method.
delete delete delete A computer program that executes any one of the methods for restoring a non-contact pulse wave signal in the device of claim 5 and is recorded on a computer-readable recording medium.