KR101798228B1 - Pulse rate measuring method using image - Google Patents

Abstract

The present invention relates to a method for measuring a pulse by using an image which is robust to a change in brightness or the like. The method comprises the following steps. A computer device detects a skin area from an image having a plurality of frames. The computer device extracts a color data average value of the skin area according to a time flow. The computer device determines a first pulse by applying a method for adjusting the number of peaks to a pulse signal determined by applying a band pass filter to the color data average value. The computer device determines a second pulse by applying a regression analysis method to a pulse derived from the total average value of a frequency value having the strongest power in each of a plurality of sections to which short time Fourier transform (STFT) on the color data average value is applied. And the computer device determines a final pulse from a weighted average of the first and second pulses.

Description

PULSE RATE MEASURING METHOD USING IMAGE [0002]

The technique described below relates to a technique for measuring a pulse using an image.

The pulse can be measured using a PPG (Photo-plethysmography) meter. Furthermore, recently, a method of measuring pulse wave signal using a video image has been studied.

Korean Patent Publication No. 10-2015-0016903

There has been a limit in that a technique for measuring a pulse wave signal or a pulse using a conventional image has a problem that the measurement performance may be degraded when an image including a large amount of noise is used depending on the ambient illuminance and environmental changes. The technique described below is intended to provide a pulse measurement method that is robust to changes in illumination.

A method for measuring a pulse using an image includes the steps of: detecting a skin region in an image having a plurality of frames, the computer apparatus extracting color data of the skin region according to a passage of time; Extracting a pulse wave signal from the data; determining a heartbeat interval value according to the pulse wave signal by the computer device; and adding a peak to the pulse wave signal when the heartbeat interval value is within a critical range And measuring a pulse.

In another aspect, a method for measuring a pulse using an image includes the steps of: detecting a skin region in an image having a plurality of frames; extracting color data of the skin region according to the passage of time; (STFT) is applied to the color data, the computer device estimates a pulse using a frequency component of an interval to which the STFT is applied, and the computer device uses a relationship between a frequency component and a pulse And determining the final pulse by applying the estimated pulse to a regression analysis function prepared in advance.

According to another aspect of the present invention, a method for measuring a pulse using an image includes the steps of: detecting a skin region in an image having a plurality of frames; extracting color data of the skin region according to a time; Wherein the apparatus comprises a step of adjusting a number of peaks in an estimated pulse wave signal by applying a band pass filter in the color data to determine a first pulse, and the computer device further comprising: a plurality of short time fourier transform (STFT) Determining a second pulse by applying a regression analysis method to a pulse estimated from a total mean value of frequency values having respective greatest powers in each of the first and second pulse periods, and determining the second pulse from the weighted average of the first pulse and the second pulse, And determining a pulse.

The technique described below can easily measure a pulse using an image captured by a smart device. Further, the technique described below is more robust to noise due to the surrounding environment than the conventional non-contact pulse measurement technique.

Fig. 1 shows an example of a configuration of a device for measuring a pulse using an image.
2 is an example of a flowchart for a method of measuring a pulse using an image.
FIG. 3 shows an example of a process of measuring a pulse according to the method of FIG.
4 is another example of a flowchart for a method of measuring a pulse using an image.
5 is a diagram illustrating a process of measuring a pulse according to the method of FIG.
6 shows an example of a procedure for preparing a regression analysis function.
7 is another example of a flowchart for a method of measuring a pulse using an image.

The following description is intended to illustrate and describe specific embodiments in the drawings, since various changes may be made and the embodiments may have various embodiments. However, it should be understood that the following description does not limit the specific embodiments, but includes all changes, equivalents, and alternatives falling within the spirit and scope of the following description.

The terms first, second, A, B, etc., may be used to describe various components, but the components are not limited by the terms, but may be used to distinguish one component from another . For example, without departing from the scope of the following description, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

As used herein, the singular " include "should be understood to include a plurality of representations unless the context clearly dictates otherwise, and the terms" comprises & , Parts or combinations thereof, and does not preclude the presence or addition of one or more other features, integers, steps, components, components, or combinations thereof.

Before describing the drawings in detail, it is to be clarified that the division of constituent parts in this specification is merely a division by main functions of each constituent part. That is, two or more constituent parts to be described below may be combined into one constituent part, or one constituent part may be divided into two or more functions according to functions that are more subdivided. In addition, each of the constituent units described below may additionally perform some or all of the functions of other constituent units in addition to the main functions of the constituent units themselves, and that some of the main functions, And may be carried out in a dedicated manner.

Also, in performing a method or an operation method, each of the processes constituting the above method may occur in a different order than that described in the context without explicitly specifying a specific order in the context. That is, each process may occur in the same order as described, may be performed substantially concurrently, or may be performed in the opposite order.

The technique described below is a technique for measuring a pulse based on an image containing a user's skin. Fig. 1 shows an example of a configuration of a device for measuring a pulse using an image. FIG. 1 illustrates an example of detecting a face in an image including a user's face.

1 (a) is an example of measuring a pulse using a smart device 50 such as a smart phone. The user takes a face with the camera built in the smart device 50. The smart device 50 detects the face area in the source image and extracts the average value of the color data of the predetermined area. The smart device 50 measures a pulse by applying a frequency analysis method or a filtering application method to a color data average value or a weighted average method to a pulse calculated from the above two methods.

1 (a), the smart device 50 includes a camera 51, a storage device 52, a computing device 53, and an output device 54. The camera 51 acquires the source image. The arithmetic unit 53 extracts the color data average value for the skin region included in the source image, and performs an operation for measuring the pulse from the color data average value. The storage device 52 may temporarily store the source image. The storage device 52 may also store a regression analysis function used for pulse estimation. The output device 54 can output the measured pulse.

1 (b) is an example of measuring a pulse using a device such as a PC. The user shoots the face with the camera 81 connected to the computer 85. The computer 85 detects the face area from the source image and extracts the average value of the color data of the predetermined area. The computer 85 measures pulses by applying a frequency analysis method or a filtering method to the color data average value or a weighted average method to the pulses calculated from the above two methods.

1C is an example in which a server 95 at a remote location measures a pulse using an image acquired by the user terminal 91. [ The user takes a face with a camera built in the user terminal 91. The user terminal 91 transfers the photographed source image to the server 95 via the network. In this case, the user terminal 91 includes a communication module for data transmission. The server 95 detects the face area from the source image and extracts the average value of the color data of the predetermined area set in advance. The server 95 measures a pulse by applying a frequency analysis method or a filtering application method to the color data average value or a weighted average method to the pulse calculated from the above two methods. The server 95 may transmit the measured pulse to the user terminal 91. [

In some cases, the user terminal 91 may detect the skin region from the source image and transmit it to the server 95. In this case, the server 95 extracts a color data average value for the skin region, measures a pulse by applying a frequency analysis method, a filtering method, or a weighted average method to the color data average value, or the pulse calculated from the above two methods. Or the user terminal 91 may detect the skin region in the source region and extract the color data average value for the skin region. The user terminal 91 may transmit only the color data average value to the server 95 to the skin area. The server 95 may measure a pulse by applying a frequency analysis method or a filtering application method to the color data average value or a weighted average method to the pulse calculated from the above two methods.

As described above, various apparatuses extract an average value of color data of a skin region from an image, apply a frequency analysis method or a filtering method to a color data average value, or apply a weighted average method to the pulse calculated from the above two methods to measure a pulse . For convenience of explanation, it will be explained below that the computer device measures the pulse. Now we will explain the specific pulse measurement method. The technique described below can measure the pulse in three ways. I want to explain them in order.

2 is an example of a flowchart for a method 100 for measuring a pulse using an image. First, the camera acquires the source image (110). The source image includes a skin area of the user. The computer device detects the skin region of interest in the source image captured by the camera (120). The algorithm by which the computer device detects skin area of interest can utilize a variety of techniques known in the art.

The computer device extracts a color data average value for the skin region of interest (130). The computer device may extract an average of color data for the entire extracted skin area. In addition, the computer device may extract an average of color data for a specific skin region in the acquired image. The computer device continuously extracts the average of the color data from the continuous image (continuous frame).

Various values can be used for the color data. For example, (1) the color data may use at least one of an R value, a G value, and a B value based on the RGB color scheme. The color data may use the color data average value for at least one of the R value, the G value, and the B value. (2) The computer device can convert the RGB color scheme to a different color scheme. For example, the computer device may convert the RGB color scheme to various color schemes such as YUV, HSV, YCbCr, YCgCo, and the like. In this case, the color data may use one of color difference components that are less influenced by the surrounding environment (illumination, etc.). For example, in the case of YCbCr, at least one of Cb value and Cr value can be used. In the case of YCgCo, at least one of Cg value and Co value can be used. Further, one of the two chrominance components, which is more robust to the variation of the illuminance, can be used. For example, in the case of YCgCo, the color data average value for at least one of Cg value and Co value can be used. In this case, the computer device can extract the average value of the Cg color data of the skin region as color data. (3) Further, the color data may be a value obtained by combining weights applied to at least one color component in various color systems such as RGB, YUV, HSV, YCbCr, YCgCo, and the like. In the case of combining color components, the color data may be a value obtained by adding different weighted values depending on the color system and the type of color component. Hereinafter, it is assumed that the computer device extracts and uses the Cg value in YCgCo. The computer apparatus can change the source image having the RGB color scheme to the YCgCo color scheme using Equation 1 below.

The computer device extracts the pulse wave signal based on the color change of the skin region. The pulse wave signal is the wavelength at which blood circulates in the heart. Pulse wave signals can be used to measure heart rate variability (HRV) and to assess current blood circulation and accumulated stress conditions. The computer device extracts a pulse wave signal that is a biological signal by applying a band-pass filter to the average value of Cg color data (140).

The computer device may determine the heart rate interval based on the pulse wave signal (150). The heartbeat interval can be detected by detecting the interval value between the peak and the peak in the pulse wave signal. This allows the computer to eventually measure the pulse. Measuring the pulse means, for example, that the pulse rate for a certain period of time can be determined.

Furthermore, the computer device can adjust the number of peaks constantly based on the heartbeat interval, and can grasp the improved pulse number. The threshold setting method based on the heart rate interval (RRI, RR-Interval) was calculated based on the heart rate at the time of the stable state for all ages presented by the Korean Medical Association, which is known as 60 to 130 times. The heart rate interval is about 1 second when the heart rate per minute is 60, and the heart rate interval is 0.4 seconds when the heart rate per minute is 130. The computer device can adjust the number of peaks to be constant by applying the above-described criteria to the heartbeat interval value calculated from the detected pulse wave signal as shown in Equation (2) below.

The computer device adds n number of peaks between intervals if the value of the heart rate interval (RRI) calculated from the pulse wave signal is more than n seconds and less than n + 1 seconds. n has a value of 0 or a positive integer. For example, when the heart rate interval (RRI) calculated from the pulse wave signal is less than 2 seconds and more than 1 second, the computer apparatus adds one peak number between intervals. The computer device adds two peaks between the intervals if the heart rate interval (RRI) calculated from the pulse wave signal is less than 2 seconds and less than 3 seconds.

The computer device can determine whether the heart rate interval falls within the critical range according to Equation (2) and adjust the number of peaks. The computer device may then measure the improved pulse based on the adjusted number of peaks (160).

FIG. 3 shows an example of a process of measuring a pulse according to the method of FIG. The user shoots a face image of a person using a smart device or a general camera (a process). The computer apparatus detects a face region displayed in a square in the source image including the face image (step b). The computer device detects the skin area in the face area (step c).

The computer device sets the skin area of interest (step d), converts the RGB color scheme of the ROI to the YCgCo color scheme, and extracts the average value of the Cg color data continuously (step e). The computer device applies a band-pass filter to the average value of the Cg color data continuously extracted (step f), and detects a pulse wave signal (step g). In the pulse signal detection process, g (1) is an example of a pulse wave signal detected from a face image with little peripheral illumination and environment change, and g (2) is a case of a pulse wave signal detected from a face image having a large peripheral illumination and environment change. . Finally, the computer device calculates the heart rate interval based on the pulse wave signal (step h). In the heart rate interval calculation process, h (1) represents a heart rate interval calculated from a pulse wave signal detected from a face image having a small change in ambient illuminance and environment, h (2) represents a pulse wave signal detected from a peripheral image, The heartbeat interval calculated by applying the procedure of FIG.

4 is another example of a flowchart for a method 200 for measuring a pulse using an image. First, the camera acquires the source image (210). The source image includes a skin area of interest of the user. The computer device detects the skin region from the source image captured by the camera (220). The algorithm by which the computer device detects the skin area can utilize various techniques known in the art.

The computer device extracts color data for the skin region (230). The computer device may extract an average of color data for the entire extracted skin area. In addition, the computer device may extract an average of color data for a specific skin region in the acquired image. The computer device continuously extracts the average of the color data from the continuous image (continuous frame). The computer device can change the RGB source image to the YCgCo color scheme and extract the Cg color data average value.

The computer device applies a short time fourier transform (STFT) to the continuous color data average value (240). The computer device can apply the STFT to the average value of the Cg color data. The computer device estimates the pulse using the frequency components of the plurality of intervals to which the STFT is applied (250). The computer device may limit the frequency range associated with the pulse at each interval. For example, a computer device can observe a signal within a frequency range of 0.6 Hz to 3.34 Hz, since the pulse rate per minute can be measured to about 40 to 200 depending on the stable or excited state of a normal person. The computer device can determine a frequency value having the largest power in a plurality of sections to which the STFT is applied and estimate a pulse from the average of the frequency values having the largest power in each analysis section.

Further, the computer device may apply an estimated pulse to a pre-established regression analysis function to produce an improved pulse (260). The process of preparing the regression analysis function will be described later.

5 is a diagram illustrating a process of measuring a pulse according to the method of FIG. The user shoots a face image of a person using a smart device or a general camera (a process). The computer apparatus detects a face region displayed in a square in the source image including the face image (step b). The computer device detects the skin area in the face area (step c). The computer device sets the skin area of interest (step d), converts the RGB color scheme of the ROI to the YCgCo color scheme, and extracts the average value of the Cg color data continuously (step e). The computer device applies STFT to the average value of the consecutively extracted Cg color data (step f). Since normal subjects can measure pulse rate per minute up to about 40 to 200 depending on their stable or excited state, signals can be observed within a frequency range of 0.6 Hz to 3.34 Hz. The computer determines a frequency value having the largest power in each section to which the STFT is applied, and calculates an average of frequency values having the largest power in the entire section (g process). The computer device estimates the pulse from the average value of the frequency values having the largest power (step h). The computer device applies the estimated pulse to a regression analysis function stored in a regression analysis DB. The computer device calculates an improved pulse with a formula calculated by applying an estimated pulse to a regression analysis function (i step).

6 shows an example of a procedure for preparing a regression analysis function. Regression analysis is an analytical method that measures fit between two variables for continuous variables. Regression analysis generates a function (formula) that calculates a specific value by using predetermined sample data in advance. The regression function is expressed as an equation representing a regression line or a regression curve. The computer device estimates the pulse (pulse estimated by applying the STFT) using the image, and at the same time measures the pulse of the user using the PPG measuring device. This process is repeated to generate a plurality of data, and the computer device generates a regression analysis function that calculates the improved second pulse by applying a pulse estimated from the STFT to a regression analysis function based on the generated plurality of data .

First, the process of preparing data for the second pulse will be described. The user shoots a face image of a person using a smart device or a general camera (a process). The computer apparatus detects a face region displayed in a square in the source image including the face image (step b). The computer device detects the skin area in the face area (step c). The computer device sets the skin area of interest (step d), converts the RGB color scheme of the ROI into the YCgCo color scheme, and extracts the average value of the Cg color data (step e). The computer device applies STFT to the average value of the consecutively extracted Cg color data (step f). Since normal subjects can measure pulse rate per minute up to about 40 to 200 depending on their stable or excited state, signals can be observed within a frequency range of 0.6 Hz to 3.34 Hz. The computer determines a frequency value having the largest power in each section to which the STFT is applied, and calculates an average of frequency values having the largest power in the entire section (g process). The computer device estimates the pulse from the average value of the frequency values having the largest power (step h). The computer device stores the estimated pulse using the STFT in the estimated pulse DB.

While simultaneously acquiring an image for estimating the second pulse, the pulse of the same user is measured by the PPG measuring device. The computer device stores the pulse measured from the PPG measuring instrument in the measuring device pulse DB.

The computer device generates a regression analysis function (i step) using the data of the pulse estimated by applying the STFT and the data of the pulse measured by the PPG measuring instrument at the same point in time (i step), and generates the regression analysis function by using the regression analysis DB . As described above, the regression analysis function means a regression line or a regression curve.

The computer device can derive a regression linear equation using the estimated pulse DB and the measuring device pulse DB for the same region. The regression linear equation is shown in Equation 3 below.

Where y is the improved pulse and x is the estimated pulse (pulse estimated by applying STFT). The results obtained by applying the actual data may vary with the values of a and b depending on the data used.

The regression curve is a curve obtained by representing a set of points on a correlation chart as a curve rather than a straight line. Regression curves show the relationship between two variables. The computer device can derive a regression curve equation using the estimated pulse DB and the measured pulse rate DB. The regression curve equation is shown in Equation (4) below.

Where y is the improved pulse and x is the estimated pulse (pulse estimated by applying STFT). The results obtained by applying the actual data may vary in the values of the constants a _i and c according to the data used.

Figure 7 is another example of a flowchart for a method 300 for measuring pulse using an image. A method 300 for measuring a pulse using an image corresponds to a technique using both the method described in FIG. 2 and the method described in FIG. 4 at the same time.

The computer device detects the skin region of interest in the source image captured by the camera (310). The algorithm by which the computer device detects the skin area can utilize various techniques known in the art. The computer device extracts a color data average value for the skin region (320). The computer device may extract an average of color data for the entire extracted skin area. In addition, the computer device may extract an average of color data for a specific skin region in the acquired image. The computer device continuously extracts the average of the color data from the continuous image (continuous frame). The computer device can change the RGB source image to the YCgCo color scheme and extract the Cg color data average value.

The computer device extracts the pulse wave signal based on the color change of the skin region. The computer device extracts a pulse wave signal which is a biological signal by applying a band-pass filter to the average value of Cg color data (330). The computer device may determine the heart rate interval based on the pulse wave signal (340). The heartbeat interval can be detected by detecting the interval value between the peak and the peak in the pulse wave signal. The computer device may adjust the number of peaks using the constant critical range described in equation (2) and determine a first pulse based on the adjusted number of peaks (350).

The computer device applies short time fourier transform (STFT) to the average value of Cg color data (360). The computer device estimates the pulse using the frequency components of the plurality of intervals to which the STFT is applied (370). The computer device may limit the frequency range associated with the pulse at each interval. For example, a computer device can observe a signal within a frequency range of 0.6 Hz to 3.34 Hz, since the pulse rate per minute can be measured to about 40 to 200 depending on the stable or excited state of a normal person. The computer device can estimate the pulse from the overall average of the frequency values having the largest power in each interval in which the STFT is applied. The computer device may also determine the second pulse 380 by applying an estimated pulse to the regression analysis function described in Figures 4 and 6.

Finally, the computer device may weight-average the first pulse and the second pulse to determine an improved pulse (390). At this time, the first weight given to the first pulse and the second weight given to the second pulse can be determined in advance through experiments. The computer device determines the final pulse by adding a value obtained by multiplying the first pulse by the first weight and a value obtained by multiplying the second pulse by the second weight.

The present embodiment and drawings attached hereto are only a part of the technical idea included in the above-described technology, and it is easy for a person skilled in the art to easily understand the technical idea included in the description of the above- It will be appreciated that variations that may be deduced and specific embodiments are included within the scope of the foregoing description.

50: Smart devices
51: Camera
52: Storage device
53:
54: Output device
81: Camera
85: Computer
91: User terminal
95: Server

Claims (16)

delete delete delete delete delete delete delete delete delete delete The computer device detecting a skin region in an image having a plurality of frames;
The computer device extracting color data of the skin region according to time;
Determining a first pulse by adjusting a number of peaks in an estimated pulse wave signal by applying a band pass filter in the color data;
Wherein the computer device determines a second pulse by applying a regression analysis method to a pulse estimated from a total average value of frequency values each having the largest power in a plurality of intervals in which STFT (Short Time Fourier Transform) is applied to the color data step; And
Wherein the computer device determines a final pulse from a weighted average of the first pulse and the second pulse. 12. The method of claim 11,
Wherein the computer device changes an RGB color of the skin region to a different color system and extracts an average value of at least one of chrominance components of the different color system as the color data. 12. The method of claim 11,
Wherein the computer device changes an RGB color scheme of the skin region to a YCgCo color scheme and extracts an average value of Cg color data as the color data. 12. The method of claim 11,
Wherein the computer device determines the first pulse based on a pulse wave signal to which n peaks are added when the heartbeat interval estimated based on the pulse wave signal is n seconds or more and less than n + 12. The method of claim 11,
Wherein the computer apparatus further comprises a step of measuring the second pulse by applying a pulse estimated by the STFT to a regression analysis function provided beforehand using a relationship between a frequency component and a pulse. 16. The method of claim 15,
Wherein the computer device further comprises: means for calculating a pulse rate estimated from an average value of frequency values having the largest powers in a plurality of intervals to which the STFT is applied, and a pulse rate measured from the subject with the PPG measuring device at the same time, A method of measuring a pulse using an image in which the regression analysis function is provided by applying a regression analysis method.

Images