KR101725357B1 - Method and Device for measuring PPG signal by using mobile device - Google Patents

Abstract

According to the present invention, depending on the systolic and diastolic phases of the heart, the oxygen saturation in the blood vessel will vary depending on the respective pressure, and the absorption of light will be different. Accordingly, it is possible to confirm that the value is affected by the oxygen saturation in the color information (RGB, contrast, etc.) seen on the skin (finger, face, etc.). Accordingly, a work for restoring a pulse wave is performed using the color information value that is changed.

Description

TECHNICAL FIELD [0001] The present invention relates to a method for measuring a pulse wave signal using a portable terminal,

The present invention relates to a pulse wave signal measuring method and an apparatus for applying the same, and more particularly, to a pulse wave signal measuring method using a portable terminal such as a smartphone and an apparatus for applying the method.

With the advent of the u-Health business, health care and health services that are available everywhere have become important. Therefore, they are interested in how to monitor human health. A representative method is a method of monitoring human health through bio-signals, such as brain, electrocardiogram, and blood pressure. However, in order to confirm the above-mentioned bio-signals, there are disadvantages that users have to visit specific places (research institutes, hospitals, etc.) and expensive cost problems for inspection. The crucial point of the u-health business is that the weight and volume of the sensor equipment are large in order to carry out where and when. Therefore, interest in the development of alternative sensing equipment naturally came to be interested in u-health business field. The purpose of the wearable devices is to increase the spatial and temporal advantages for users and to measure quality biological signals like existing sensing devices. However, this also has some drawbacks to purchase equipment separately.

GREGOSKI, M. J., MUELLER, M., VERTEGEL, A., SHAPOREV, A., JACKSON, B. B., FRENZEL, R. M., SPREHN, S. M. and TREIBER, F.A., 2012. Development and validation of a smartphone heart rate acquisition application for health promotion and wellness telehealth applications. International journal of telemedicine and applications, 2012, pp. One. GREGOSKI, M., VERTEGEL, A. and TREIBER, F., 2011. Photoplethysmograph (PPG) derived heart rate (hr) acquisition using an Android smart phone, Proceedings of Wireless Health 2011, ACM, pp. 23. KWON, S., KIM, H. and PARK, KS, 2012. Validation of heart rate extraction using video imaging on a built-in camera system of a smartphone, Engineering in Medicine and Biology Society (EMBC) the IEEE 2012, IEEE, pp. 2174-2177. POH, M., MCDUFF, D.J. and PICARD, R.W., 2010. Non-contact, automated cardiac pulse measurements using video imaging and blind source separation. POH, M., MCDUFF, D.J. and PICARD, R.W., 2011. Advances in noncontact, multiparameter physiological measurements using a webcam. Biomedical Engineering, IEEE Transactions on, 58 (1), pp. 7-11. ROALD, N.G., 2013. Estimation of Vital Signs from Ambient-Light Non-Contact Photoplethysmography,

The present invention provides a method and an apparatus for measuring a pulse wave using a portable terminal.

The present invention relates to a method for measuring a pulse wave by an optical method and an apparatus for applying the method.

Therefore, the present invention provides a method of easily measuring pulse waves without expensive equipment support or movement to a specific inspection site, and an apparatus for applying the method.

The pulse wave measuring method according to the present invention comprises:

Obtaining a first image and a second image of a subject's face and a finger, respectively, using the portable terminal;

Extracting a first signal and a second signal indicative of changes in blood flow of the finger and skin from the first image and the second image; And

And generating a pulse wave (PPG) signal of the subject using the first signal and the second signal.

According to a specific embodiment of the present invention, extracting the first signal includes extracting R (red), G (green), and B (blue) signal values from each of the first images.

According to a specific embodiment of the present invention, R, G and B signal values are extracted from the first image through ICA (Independent Component Analysis).

According to a specific embodiment of the present invention, in extracting the pulse wave signal of the subject, the weight vectors W1 and W2 between the respective signals are calculated using a regression analysis for the first signal and the second signal And combines the first signal and the second signal using the calculated two weight vectors to process the PPG signal.

According to an embodiment of the present invention, the second signal represents a change in HSI (intensity, brightness) in hue or HSV color space according to a blood flow change.

According to a specific embodiment of the present invention, the change in hue according to the blood flow change represents a change in the magnitude of red.

The basic background of the present invention is to extract a first signal and a second signal including pulse wave information using a change in oxygen saturation in a blood vessel according to respective pressures according to the systolic and diastolic phases of the heart.

The pulse wave measuring apparatus according to the present invention comprises:

By performing the pulse wave measuring method,

A first camera for acquiring the first image,

A second camera for acquiring the second image,

A body on which the first camera and the second camera are installed; and

A processing system provided in the body for processing / analyzing images from the first and second cameras; .

Wherein the body includes a first side and a second side opposite to the first side,

Wherein the first camera is a front camera provided on the first surface,

The second camera is a rear camera provided on the second surface.

do.

According to one embodiment of the pulse wave measuring apparatus according to the present invention,

A display device is provided on the first surface,

The first camera is provided on one side of the display device on the first side.

According to the present invention, a subject or a user can monitor a bio-signal required for checking his / her basic health condition through a simple application installed in a portable terminal such as a smart phone without purchasing a separate equipment. The present invention improves the spatial and temporal degrees of freedom of a subject in detecting a pulse wave and enables cost reduction.

1 (a) and 1 (b) schematically show the overall configuration of a portable terminal by showing a first side (front side) and a second side (rear side) of the portable terminal.
2 (a) and 2 (b) illustrate a pulse wave measuring method using a portable terminal such as a smart phone.
3 is a view for explaining the concept of a method of extracting a PPG using a portable terminal.
FIG. 4 is a block diagram illustrating a signal processing flow after photographing a face of a user using a front camera and taking a blood flow change photograph using a rear camera, illustrating a method of extracting PPG using a portable terminal.
FIG. 5 is a view for explaining signal processing using R, G, and B signals after a face image of a user.
6 is a view for explaining the concept of a generally known sliding window.
FIG. 7 is a flowchart of a signal processing process using the R (red) and Intensity (brightness) signals, which are color components of an image after the user's finger is photographed.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a pulse wave measuring method and an apparatus for applying the pulse wave measuring method according to the present invention will be described in detail with reference to the accompanying drawings.

The present invention uses a portable terminal equipped with independent cameras on the front and rear sides of smart phones, which are currently popular.

The body 10 of the portable terminal has a first surface 10a on which the display 12 is provided and a second surface 10b opposite the surface on which the display 12 is provided as shown in Figures 1 (a) and 1 (b) Rear).

A first camera (a front camera 11a) and a second camera (a rear camera 11b) are installed on the first surface 10a and the second surface 10b.

A processing system 13 for processing the first signal and the second signal acquired using the first camera 11a and the second camera 11b is provided in the body 10 of the portable terminal. The processing system 13 includes hardware based on a microprocessor, operating system (OS) and software running on it.

The first camera 11a and the second camera 11b simultaneously photograph the face of the subject and the inner surface of the finger. Such a photographing is intended to optically detect a change in blood flow in the subject's face and a finger.

Human blood consists of 45% red blood cells and 54.3% plasma. Conventional measurement of pulse wave signals used circulation of red blood cells through capillaries close to skin surface. There are many hemoglobin in the cytoplasm of erythrocytes, which is the iron that carries oxygen to the whole body. Oxygen saturation refers to the percentage of the concentration of oxidized hemoglobin (Hbo2) bound to oxygen for total hemoglobin concentration, which is a variable that varies the absorption of light

According to the present invention, it is possible to extract or generate a pulse wave signal of a subject by detecting a change in oxygen saturation through image processing using a front and rear camera mounted on a portable terminal such as a smart phone.

FIG. 2 illustrates a method of photographing by a portable terminal when detecting a pulse wave signal according to the present invention.

First, as shown in FIG. 2 (a), the user takes a picture with the upper body being displayed on the screen 12 by using the front camera 11a in a state of being held by the one hand 30. In this way, according to the present invention, the facial region 40 of the captured region is tracked as a region of interest (ROI) and is selected as the effective signal region. On the other hand, the inner surface of the finger is photographed in a state in which the cloth or the index finger 31 of one hand 30 is placed on the rear camera 11b. At this time, in order to photograph the blood flow of the finger, the finger portion must be bright, so that the finger portion can be illuminated when the outside is dark or when it is an indoor space.

Referring to FIG. 3, an overview of a pulse wave signal detection method according to the present invention will be described.

As a first step S31, photographing is performed by the method shown in Figs. 2A and 2B. That is, the subject holds the portable terminal 10 with his or her hand 30 to photograph the inside face and the face of the finger.

And obtains the photographed face image S32a and the finger image S32b in the second step S32.

In the third step S33, two signals corresponding to the subject's pulse wave, a blood flow change signal on the face, and a blood flow change signal of the finger are extracted from the face image S32a and the finger image S32b (S33). In step S33, the blood flow change signal (the first signal, S33a) and the blood flow change signal (the second signal, S33b) of the face are obtained from the signal extraction from the facial image and the finger image performed according to a method described later .

The first signal S33a and the second signal S33b are combined in the fourth step S34 and the PPG signal of the subject is formed using the combined signal in the following fifth step S35 )do.

Referring to FIG. 4, the pulse wave detecting method according to the present invention will be described in more detail.

First step (S41): Starts recognition of the region of interest (ROI). In the present invention, the region of interest is the photographing of the moving image for a certain period of time on the face of the user and the inside of the finger using the front and rear cameras of the portable device as the face (40 in Fig.

Second step S42: The moving image about the face and the finger inner surface is acquired and data is extracted therefrom.

Third step S43: R (red), G (green), and B (blue) data are extracted from the facial image acquired to obtain the PPG signal from the facial image acquired in real time using the smartphone forward camera . An ICA (Independent Component Analysis) algorithm is applied to R, G, and B values to extract a pure signal (first signal).

Fourth step (S44, S45): As in the third step, the R (red) value or the Intensity (HSI) value, which is the color component of the image, is obtained from the finger image obtained in real time using the rear camera of the smartphone And extracted.

Step 5 (S46): A second signal indicating the change in blood flow in the finger is extracted from the R (red) value or the Intensity (HSI) value.

Step S47: The weight factors W1 and W2 between the first signal (blood flow change signal on the face) and the second signal (blood flow change signal on the finger) are calculated using a regression analysis .

Step S48; And performs processing for combining the first signal and the second signal using the derived weight factors W1 and W2.

Step 8 (S49): The subject's PPG signal is measured in real time using the data obtained by the processing.

Hereinafter, a process of processing a signal obtained from the finger and the face will be described in detail. A method of extracting a signal from a facial image will be described in detail with reference to FIG.

Step S41: First, the upper part of the body is extracted using the front camera of the portable terminal 10. Here, the face (rectangular frame) acquired in real time is tracked as ROI (ROI) 40, and only the tracked image is acquired.

S42: R (red), G (green), and B (blue) values, which are the color components of the image, are extracted for each frame of the acquired facial image.

S43: An ICA (Independent Component Analysis) algorithm is applied to the extracted R, G and B values to extract a pure signal, which is a color component of the image, and a PPG signal showing a pattern similar to the PPG signal among the extracted pure signals is converted into a first signal .

S44: In order to acquire a first signal, i.e., a pulse wave signal, in real time, a sliding window algorithm as shown in FIG. 6 is applied, and data of a certain window size is moved by resolution, The pulse wave signal is extracted (see FIG. 7).

Hereinafter, a process of measuring a pulse with a finger will be described with reference to FIG.

S71: A finger image of a specific format, for example, YUV420SP format is acquired in real time using the rear camera 11b of the portable terminal.

S72: The finger image of the YUV420SP format is converted into an RGB format so as to enable RGB color separation for subsequent signal processing.

S73: The second (PPG) signal is extracted (formed) by extracting R value or Intensity (brightness) value as a color component for each frame of the finger image in the RGB format.

S74: A moving average algorithm is applied to the second signal to align the base line and remove the noise of the signal.

S75: In order to acquire the second signal in real time, the sliding window algorithm of FIG. 6 as described above is applied to move the data size of the window size by the resolution, The pulse wave signal is extracted.

A method of forming or restoring a final pulse wave signal using the first signal and the second signal obtained through the above process is as follows.

The first (pulse wave) signal measured from the face image and the second pulse wave measured from the finger are corrected and complemented by the above-described method to perform combined signal processing to obtain an accurate pulse wave signal.

For this purpose, a weight factor (W1, W2) is extracted by performing a regression analysis on the first signal and the second signal.

In order to restore the extracted weight value to one accurate PPG signal, the weight value of each signal type is applied to form or restore (restores) a single signal. The restored signal or the one integrated pulse wave signal PPGr is calculated by the following equation.

PPGr = W1 _* (Rf, Gf, Bf) + W2 (Vf)

In the above equation, Rf, Gf, and Bf are values of R, G, and B in the face image, Vf is a finger value, and W1 and W2 are weight factors, i.e., weights, obtained from the first signal and the second signal.

According to the method of the present invention as described above, a very accurate pulse wave (PPG) signal can be obtained using a portable terminal, and various measurements or tests using the same can be performed.

According to the present invention, a subject or a user can monitor a bio-signal required for checking his / her basic health condition through a simple application installed in a portable terminal such as a smart phone without purchasing a separate equipment. The present invention improves the spatial and temporal degrees of freedom of a subject in detecting a pulse wave and enables cost reduction.

In the foregoing, exemplary embodiments have been described and shown in the accompanying drawings to facilitate understanding of the present invention. It should be understood, however, that such embodiments are merely illustrative of the present invention and not limiting thereof. And it is to be understood that the invention is not limited to the details shown and described. Since various other modifications may occur to those of ordinary skill in the art.

10: Portable terminal
10a: First side (front side)
10b: Second side (rear side)
11a: first camera (front camera)
11b: second camera (rear camera)
30: (subject) hand
31: (subject) finger
40: area of interest

Claims (13)

Capturing a first face of a face and a second face of a finger by simultaneously photographing the subject's face and the face of the finger using the portable terminal having the first camera and the second camera;
Extracting a first signal indicative of a change in facial blood flow from the first image;
Extracting a second signal indicating a change in blood flow of a finger from a second image simultaneously captured with the first image;
Obtaining a weight factor for the first signal and the second signal; And
And restoring a pulse wave (PPGr) signal for the subject from the first signal and the second signal according to the following equation using the following equation.
<Expression>
PPGr = W1 _* (Rf, Gf, Bf) + W2 (Vf)
Rf, Gf and Bf are the values of R, G and B in the first image, Vf is the value of the finger, and W1 and W2 are the weights of the first and second signals. The method according to claim 1,
Extracting the first signal comprises extracting R (red), G (green) and B (blue) signal values from each of the first images. 3. The method of claim 2,
Wherein R, G, and B signal values are extracted from the first image through ICA (Independent Component Analysis). 4. The method according to any one of claims 1 to 3,
In the step of extracting the subject's pulse wave signal, the weight vectors W1 and W2 between the respective signals are calculated by using a regression analysis for the first signal and the second signal, and using the calculated two weight vectors And combining the first signal and the second signal to process the PPGr signal. The method according to claim 1,
Wherein the second signal represents a change in HSI (intensity, brightness) in hue or HSV color space according to a blood flow change. 6. The method of claim 5,
Wherein in the second signal, a change in hue according to the blood flow change indicates a change in a magnitude value of red. The method according to claim 5 or 6,
In the step of extracting the subject's pulse wave signal, the weight vectors W1 and W2 between the respective signals are calculated by using a regression analysis for the first signal and the second signal, and using the calculated two weight vectors And combining the first signal and the second signal to process the PPGr signal. A pulse wave measuring apparatus for performing the pulse wave measuring method according to any one of claims 1 to 3,
A first camera for acquiring the first image,
A second camera for acquiring the second image,
A body on which the first camera and the second camera are installed; and
A processing system provided in the body for processing / analyzing images from the first and second cameras; And a pulse wave measuring device. 9. The method of claim 8,
The processing system calculates the weight vectors W1 and W2 between the respective signals using a regression analysis for the first signal and the second signal in extracting the pulse wave signal of the subject, And combines the first signal and the second signal using two weight vectors to process the PPGr signal. 10. The method of claim 9,
Wherein the second signal indicates a change in HSI (intensity, brightness) in hue or HSV color space according to a blood flow change. 9. The method of claim 8,
Wherein the second signal indicates a change in HSI (intensity, brightness) in hue or HSV color space according to a blood flow change. 12. The method of claim 11,
Wherein the change in hue according to the change in blood flow represents a change in the magnitude of red. 11. The method of claim 10,
Wherein the change in hue according to the change in blood flow represents a change in the magnitude of red.

Images