KR102261526B1 - Non-contact system of measuring ppg signal and its way to working - Google Patents

Abstract

The present invention provides a non-contact PPG signal measurement system using a camera and a driving method thereof, comprising: a camera unit 100 for capturing an image of a subject to be measured; a central processing unit 200 for calculating the PPG signal of the subject to be measured from the image taken by the camera unit 100; and a display unit 300 that displays the PPG signal calculated by the central processing unit 200 and the heart rate extracted from the PPG signal on a display screen, wherein the central processing unit 200 includes the camera unit 100 a region selection unit 210 for selecting a target region from which an optical signal is to be extracted from the photographed image of the subject; an optical signal detector 220 that detects an optical signal reflected from the target region selected by the region selector 210 for each frame of the image captured by the camera unit 100; a photoelectric converter 230 for converting the optical signal detected by the optical signal detector 220 into electric charge; a calculation unit 240 for calculating a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge by the photoelectric conversion unit 230; a frequency signal converter 250 for converting the signal calculated by the calculator 240 into a frequency signal and outputting it as a PPG signal; a control unit 260 for processing the PPG signal converted by the frequency signal converting unit 250; and a heart rate determination unit 270 for determining the heart rate of the subject to be measured from the PPG signal processed by the control unit 260 .
According to the non-contact PPG signal measuring system using the camera and the driving method thereof proposed in the present invention, there is a correlation between the intensity of the optical signal in the image taken through the camera and the subcutaneous reflectance according to the blood flow change of the heart rate measurement subject. By using the feature, it is possible to reduce the inconvenience that the heart rate measurement subject must wear the device for conventional body-contact heart rate measurement, measure the heart rate very easily and conveniently in an unconstrained state, and the heart rate measurement subject briefly in front of the camera You can measure your heart rate just by standing, reducing the hassle of using a heart rate monitoring system.
In addition, according to the non-contact PPG signal measuring system using a camera and the driving method thereof proposed in the present invention, a target area from which an optical signal is to be extracted from a photographed image is determined and the optical signal intensity in the area is measured on the x,y coordinates. By calculating pixel values, filtering and amplifying them, it is affected from noise components caused by the movement of the heart rate target's face or noise components due to environmental influences such as changes in the frequency component of light irradiated to the face. can receive less, so it is possible to measure the heart rate of the subject of heart rate measurement more accurately.

Description

Non-contact PPG signal measurement system using a camera and its driving method {NON-CONTACT SYSTEM OF MEASURING PPG SIGNAL AND ITS WAY TO WORKING}

The present invention relates to a non-contact PPG (photoplethysmography) signal measuring system and a driving method thereof, and more particularly, to a non-contact PPG signal measuring system using a camera and a driving method thereof.

Heart rate refers to the rate of the heart, and due to the flow of blood, the arteries expand and relax repeatedly, which is called the pulse. The heart rate differs slightly depending on the state of the heart and blood vessels. Generally, the heart rate for adults is 60 to 80 beats per minute, and the younger the age, the higher the heart rate, so the rate for newborns is about 120 to 140 beats per minute. Heart rate usually increases by about 8 beats per 1℃ increase in body temperature, and is also affected by respiratory movement, adrenaline, carbon dioxide, and the like.

Recently, interest in sports, physical exercise, and health is continuously increasing, and heart rate measurement is increasingly important for preventing accidents or monitoring body changes. Heart rate measurement is the most important health information that is the basis for diagnosing stress, physical strength, and cardiovascular system. , the number of regular households to measure their heart rate is increasing.

Non-invasive methods for detecting a heartbeat include an electrical detection method, a mechanical detection method, and an optical detection method. The electrical detection method is a method of measuring a change in the electrical impedance or admittance of a tissue by attaching an electrode on the skin, but it is inconvenient to constantly measure an individual's heartbeat due to the characteristics of the device. The mechanical detection method detects the pulsation transmitted through the skin as a heartbeat through a pressure conversion sensor such as a piezoelectric element. ), which makes it difficult to use in various ways.

The optical detection method is a method of detecting by irradiating light having a good correlation with hemoglobin in blood to arterial blood close to the skin. 1 is a diagram illustrating a system for measuring a conventional contact PPG signal, and FIG. 2 is a diagram illustrating a principle of a system for measuring a conventional contact type PPG signal. As shown in FIGS. 1 and 2 , the conventional optical detection method is a body contact method, and thus there are many limitations in heart rate measurement. Accordingly, there is a need for an optical detection method for detecting a non-contact heartbeat.

On the other hand, as prior art related to the present invention, Korean Patent Application Laid-Open No. 10-2012-0057813 (title of the invention: heart rate measurement method using photoplethysmography, announcement date: June 07, 2012) and the like has been disclosed.

The present invention has been proposed to solve the above problems of the previously proposed methods, and there is a correlation between the intensity of an optical signal in an image captured by a camera and a subcutaneous reflectance according to a change in blood flow of a subject for heart rate measurement. By using the feature, it is possible to reduce the inconvenience that the heart rate measurement subject must wear the device for conventional body-contact heart rate measurement, measure the heart rate very easily and conveniently in an unconstrained state, and the heart rate measurement subject briefly in front of the camera An object of the present invention is to provide a non-contact PPG signal measuring system using a camera and a driving method thereof, which can measure the heart rate just by standing, thereby reducing the hassle of using the heart rate measuring system.

In addition, the present invention determines a target region from which an optical signal is to be extracted from a photographed image, calculates the optical signal intensity in the region as pixel values on the x and y coordinates, and filters and amplifies it by going through a processing process, The heart rate measurement target can be measured more accurately because it can be less affected by noise components caused by the movement of the heart rate target's face or noise components caused by environmental influences such as changes in the frequency component of the light irradiated to the face. Another object of the present invention is to provide a non-contact PPG signal measuring system using a camera and a driving method thereof.

A non-contact PPG signal measurement system using a camera according to a feature of the present invention for achieving the above object,

A non-contact PPG signal measurement system using a camera, comprising:

a camera unit for capturing an image of a heart rate measurement target (hereinafter, referred to as a measurement target);

a central processing unit for calculating the PPG signal of the subject to be measured from the image taken by the camera unit; and

and a display unit for displaying the PPG signal calculated by the central processing unit and the heart rate extracted from the PPG signal on a display screen,

The central processing unit,

a region selection unit for selecting a target region from which an optical signal is to be extracted from the image of the subject to be measured taken by the camera unit;

an optical signal detector configured to detect an optical signal reflected from the target region selected by the region selector for each frame of the image captured by the camera unit;

a photoelectric conversion unit converting the optical signal detected by the optical signal detecting unit into electric charge;

a calculation unit for calculating a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge by the photoelectric conversion unit;

a frequency signal converter converting the signal calculated by the calculator into a frequency signal and outputting it as a PPG signal;

a control unit for processing the PPG signal converted by the frequency signal converting unit; and

It is characterized in that it comprises a heart rate determination unit for determining the heart rate of the measurement target from the PPG signal processed by the control unit.

Preferably, the region selection unit,

a face recognition unit for recognizing the face of the subject to be measured from the image taken by the camera unit;

a feature point detection unit for detecting all the feature points from the face recognized by the face recognition unit;

a feature point extraction unit for extracting only effective feature points effective for face recognition and optical signal extraction of a subject to be measured from among the feature points detected by the feature point detection unit; and

and a region determining unit that determines a target region from which an optical signal is to be extracted from the effective key points extracted by the key point extraction unit.

More preferably, the region selection unit,

When it is inappropriate to recognize the face of the measurement subject from the image taken by the camera unit, or to determine the target region from which the optical signal is to be extracted from the face of the measurement subject, re-acquisition of the face image of the measurement subject It may be configured to further include an image re-requesting unit.

Preferably, the optical signal detection unit,

When optical signals of a plurality of colors are detected from the image photographed by the camera unit, each optical signal data value separated for each color channel may be detected for each frame.

More preferably, the optical signal detection unit,

It is possible to detect an optical signal data value in the green (G) channel, which is a wavelength having the highest light absorption rate of hemoglobin and oxidized hemoglobin.

Preferably, the calculation unit,

The change in the subcutaneous blood flow of the subject to be measured may be calculated by using the feature that there is a correlation between the intensity of the optical signal detected by the optical signal detector and the subcutaneous reflectance according to the change in the blood flow of the subject to be measured.

More preferably, the calculation unit,

The photoelectric conversion unit stores the intensity of the signal converted into electric charge as pixel values on the x and y coordinates, and divides the target region into small regions in which a predetermined blood flow rate is constant, and in each of the divided small regions By calculating the average value of the pixel values on the x and y coordinates and then tracking the change in the average value over time, the change in the subcutaneous blood flow of the subject to be measured may be calculated.

Preferably, the control unit,

It may be configured to include a filter unit for filtering noise or noise from the PPG signal converted by the frequency signal converting unit, and determining a frequency range of the output PPG signal.

More preferably, the control unit,

The filter unit may further include an amplifying unit for amplifying the PPG signal output by filtering the noise or noise to an appropriate gain value.

Preferably,

The control unit may further include a stress calculator that calculates a stress index by using the heart rate variability index extracted from the processed PPG signal and a static correlation between the autonomic nervous system activation due to stress.

A non-contact PPG signal measurement method using a camera according to a feature of the present invention for achieving the above object,

A non-contact PPG signal measurement method using a camera, comprising:

(1) taking an image of the subject to be measured;

(2) calculating the PPG signal of the subject to be measured from the image taken in step (1); and

(3) displaying the PPG signal calculated in step (2) and the heart rate extracted from the PPG signal on a display screen,

The step (2) is,

(2-1) selecting a target region from which an optical signal is to be extracted from the image of the subject to be measured taken in step (1);

(2-2) detecting the optical signal reflected from the target area selected in step (2-1) for each frame of the image captured in step (1);

(2-3) converting the optical signal detected in step (2-2) into electric charge;

(2-4) calculating a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge in the step (2-3);

(2-5) converting the signal calculated in step (2-4) into a frequency signal and outputting it as a PPG signal;

(2-6) processing the PPG signal converted in the step (2-5); and

(2-7) It is characterized in that it comprises the step of determining the heart rate of the subject to be measured from the PPG signal processed in the step (2-6).

Preferably, the step (2-1) is

(2-1-1) recognizing the face of the subject to be measured from the image captured in step (1);

(2-1-2) detecting all the feature points from the face recognized in the step (2-1-1);

(2-1-3) extracting only effective feature points effective for optical signal extraction among the feature points detected in step (2-1-2); and

(2-1-4) may be implemented including the step of determining a target region from which an optical signal is to be extracted from the effective feature points extracted in the step (2-1-3).

More preferably, the step (2-1) is

(2-1-5) In case of inappropriate for recognizing the face of the subject to be measured from the image taken in step (1), or inappropriate for determining the target region from which the optical signal is to be extracted from the face of the subject to be measured , it may be implemented by further comprising the step of requesting the re-acquisition of the face image of the subject to be measured.

Preferably, in the step (2-2),

When optical signals of a plurality of colors are detected in the image photographed in step (1), each optical signal data value separated for each color channel may be detected for each frame.

More preferably, in the step (2-2),

It is possible to detect an optical signal data value in the green (G) channel, which is a wavelength having the highest light absorption rate of hemoglobin and oxidized hemoglobin.

Preferably, in step (2-4),

The change in the subcutaneous blood flow of the subject to be measured may be calculated using the feature that there is a correlation between the intensity of the optical signal detected in step (2-2) and the subcutaneous reflectance according to the change in blood flow of the subject to be measured.

More preferably, in the step (2-4),

The intensity of the signal converted into electric charge in step (2-3) is stored as pixel values on the x and y coordinates, and the target area is divided into small areas in which a predetermined blood flow can be considered constant, and each divided After calculating the average value of the pixel values on the x and y coordinates in the region, the change in the average value over time is tracked to calculate the change in the subcutaneous blood flow of the subject to be measured.

Preferably, the step (2-6) is

(2-6-1) filtering noise or noise from the PPG signal converted in step (2-5) and determining a frequency range of the output PPG signal may be implemented.

More preferably, the step (2-6) is

(2-6-2) The step of amplifying the PPG signal output after the noise or noise has been filtered in the step (2-6-1) to an appropriate gain value may be further included.

Preferably,

(4) It may be implemented by further comprising the step of calculating a stress index using the positive correlation between the heart rate variability index extracted from the PPG signal processed in step (2-6) and the autonomic nervous system activation due to stress.

According to the non-contact PPG signal measuring system using the camera and the driving method thereof proposed in the present invention, there is a correlation between the intensity of the optical signal in the image taken through the camera and the subcutaneous reflectance according to the blood flow change of the heart rate measurement subject. By using the feature, it is possible to reduce the inconvenience that the heart rate measurement subject must wear the device for conventional body-contact heart rate measurement, measure the heart rate very easily and conveniently in an unconstrained state, and the heart rate measurement subject briefly in front of the camera You can measure your heart rate just by standing, reducing the hassle of using a heart rate monitoring system.

In addition, according to the non-contact PPG signal measuring system using a camera and the driving method thereof proposed in the present invention, a target area from which an optical signal is to be extracted from a photographed image is determined and the optical signal intensity in the area is measured on the x,y coordinates. By calculating pixel values, filtering and amplifying them, it is affected from noise components caused by the movement of the heart rate target's face or noise components due to environmental influences such as changes in the frequency component of light irradiated to the face. can receive less, so it is possible to measure the heart rate of the subject of heart rate measurement more accurately.

1 is a view showing a system for measuring a conventional contact PPG signal.
2 is a view showing the principle of a system for measuring a conventional contact PPG signal.
3 is a view showing the configuration of a non-contact PPG signal measurement system using a camera according to an embodiment of the present invention as a functional block.
4 is a view showing the configuration of the area selection unit of the non-contact PPG signal measurement system using a camera according to an embodiment of the present invention as a functional block.
5 is a diagram illustrating a method of selecting a target area from which an optical signal is to be extracted in a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention.
6 is a diagram illustrating a method of detecting separated optical signal data values for each color channel for each frame in a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention.
7 is a diagram illustrating a principle of using a correlation between the intensity of a detected optical signal and a subcutaneous reflectance according to a change in blood flow in the non-contact PPG signal measuring system using a camera according to an embodiment of the present invention.
8 is a view showing light reflection in hemoglobin of a non-contact PPG signal measurement system using a camera according to an embodiment of the present invention.
9 is a diagram illustrating a flow of a method of driving a non-contact PPG signal measurement system using a camera according to an embodiment of the present invention.
10 is a diagram illustrating a flow of a method of calculating a PPG signal in a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention.

Hereinafter, preferred embodiments will be described in detail so that those of ordinary skill in the art can easily practice the present invention with reference to the accompanying drawings. However, in describing the preferred embodiment of the present invention in detail, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the same reference numerals are used throughout the drawings for parts having similar functions and functions.

In addition, throughout the specification, when it is said that a part is connected to another part, this includes not only the case where it is directly connected, but also the case where it is indirectly connected with another element interposed therebetween. In addition, the inclusion of any component means that other components may be further included, rather than excluding other components, unless otherwise stated.

PPG measurement is a method of estimating the heartbeat state by measuring the amount of blood flow flowing through a blood vessel using the optical characteristics of a living tissue. When blood flows to the peripheral blood vessels, the blood flow slows between the diastolic and systolic of the heart, causing a change in intravascular transparency, and sensing this change in transparency is the basic principle of PPG measurement. Such a shape can be reliably measured on the face, fingers, ear lobe, etc. of peripheral tissues of the body. In the case of PPG measurement, which is generally used, it is measured using a finger. If the same principle is applied to the face, the blood flow in the blood vessels of the facial skin will also change according to the heartbeat state, so the present invention measures the heartbeat state by measuring and analyzing the change in blood flow in the blood vessels in the facial skin through a camera. .

3 is a diagram illustrating the configuration of a non-contact PPG signal measurement system using a camera according to an embodiment of the present invention as functional blocks. As shown in Figure 3, the non-contact PPG signal measurement system using a camera according to an embodiment of the present invention, the camera unit 100 for taking an image of the subject to be measured; a central processing unit 200 for calculating the PPG signal of the subject to be measured from the image taken by the camera unit 100; and a display unit 300 that displays the PPG signal calculated by the central processing unit 200 and the heart rate extracted from the PPG signal on a display screen, wherein the central processing unit 200 includes the camera unit 100 a region selection unit 210 for selecting a target region from which an optical signal is to be extracted from the photographed image of the subject; an optical signal detector 220 that detects an optical signal reflected from the target region selected by the region selector 210 for each frame of the image captured by the camera unit 100; a photoelectric converter 230 for converting the optical signal detected by the optical signal detector 220 into electric charge; a calculation unit 240 for calculating a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge by the photoelectric conversion unit 230; a frequency signal converter 250 for converting the signal calculated by the calculator 240 into a frequency signal and outputting it as a PPG signal; a control unit 260 for processing the PPG signal converted by the frequency signal converting unit 250; and a heart rate determiner 270 for determining the heart rate of the subject to be measured from the PPG signal processed by the controller 260 .

The region selection unit 210, as a device constituting the central processing unit 200, selects an optimal target region from which an optical signal is extracted from the image of the measurement subject photographed by the camera unit 100 to select the heart rate of the measurement subject. The measurement accuracy can be improved. Hereinafter, the region selection unit 210 will be described in more detail with reference to FIG. 4 .

4 is a block diagram illustrating the configuration of an area selection unit of a non-contact PPG signal measurement system using a camera according to an embodiment of the present invention. As shown in FIG. 4 , the region selection unit 210 includes a face recognition unit 211 for recognizing a face of a subject to be measured from an image captured by the camera unit 100 , and a face recognized by the face recognition unit 211 . A keypoint detector 212 that detects all the keypoints in , a keypoint extraction unit 213 that extracts only effective keypoints effective for optical signal extraction among the keypoints detected by the keypoint detection unit 212, and the effective extracted by the keypoint extraction unit 213 It may be configured to include a region determiner 214 that determines a target region from which an optical signal is to be extracted from the feature point. The heart rate measurement target is generally, but is not limited to, a human being, and may be any living organism having a measurable heart.

5 is a diagram illustrating a method of selecting a target area from which an optical signal is to be extracted in a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention. As shown in FIG. 5 , the face recognition unit 211 may recognize the face of the measurement target for each frame of the image captured by the camera unit 100 . As the face recognition method, any existing face recognition method may be used. When the face recognition unit 211 determines that a face is present in the frame, it may determine that the face is the face of the subject to be measured.

As shown in FIG. 5 , the feature point detector 212 may detect all feature points suitable for determining a face region suitable for optical signal measurement in the face recognized by the face recognition unit 211 . More specifically, it is possible to detect not only facial organs located in the face region of the subject to be measured, but also all feature points capable of determining the face region along the contour of the face of the subject to be measured. Facial organs may be eyes, nose, mouth, and ears, but are not limited thereto, and may include various body parts positioned on the face, and it is preferable to broadly interpret including glasses or sunglasses worn on the face.

As shown in FIG. 5 , the key point extraction unit 213 may extract only effective key points necessary or effective for recognizing the face region of the subject to be measured and extracting an optical signal among the key points detected by the key point detection unit 212 . have.

As shown in FIG. 5 , the region determiner 214 may determine a target region from which an optical signal is to be extracted by connecting the effective feature points extracted by the feature point extraction unit 213 . When determining the target region from which the optical signal is to be extracted, a region in which capillaries are dense and the hemoglobin concentration changes according to the heartbeat is appropriate, but preferably the forehead or the cheek, but is not limited thereto. The target area is suitable for heart rate measurement because it is a region in the face where movement by expression is relatively small, and since it is located in the center of the target face, the influence on heart rate measurement can be reduced even if the face moves within the frame. According to the embodiment of determining the target region shown in FIG. 5 , the target region is a triangular region surrounded by a straight line connecting the effective feature point located on one side of the nose, the effective feature point located at the lower end of the ear, and the effective feature point located on the one side of the mouth. can be

The image re-requester 215 is another device constituting the region selector 210 and may request re-acquisition of a face image of a subject to be measured. More specifically, when it is inappropriate to recognize the face of the measurement subject from the image taken by the camera unit 100 or to determine the target region from which the optical signal is to be extracted from the face of the measurement subject, the camera unit 100 Re-acquisition of the subject's face image may be requested.

6 is a diagram illustrating a method of detecting separated optical signal data values for each color channel for each frame in a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention. As shown in FIG. 6 , when the optical signal detector 220 detects optical signals of a plurality of colors from the image photographed by the camera unit 100, each optical signal data value separated for each color channel for each frame can be detected. That is, in the case of the RGB color space system, each optical signal data value separated for each red (R) component channel, green (G) component channel, and blue (B) component channel can be detected for each frame in the captured image.

In particular, the optical signal detector 220 may detect optical signal data of a green (G) component channel. That is, it is easier to measure the change in the hemoglobin concentration according to the heartbeat by using the green (G) component channel, which has the highest light absorption rate of hemoglobin and oxidized hemoglobin, which are the color elements.

The photoelectric conversion unit 230 may convert the optical signal detected by the optical signal detection unit 220 into electric charge. More specifically, by using a photoelectric conversion element, which is an element that converts light into electric charge, when light is transmitted to the photoelectric conversion element, the light may be converted into electric charge. Preferably, a photodiode may be used as the photoelectric conversion element.

The calculating unit 240 may calculate a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge by the photoelectric conversion unit 230 . 7 is a diagram illustrating a principle of using the correlation between the intensity of a detected optical signal and the subcutaneous reflectance according to a change in blood flow in the non-contact PPG signal measurement system using a camera according to an embodiment of the present invention, and FIG. It is a view showing light reflection in hemoglobin of a non-contact PPG signal measurement system using a camera according to an embodiment of the present invention. 7 to 8 , in the process of calculating the change in blood flow, the calculator 240 correlates the intensity of the optical signal detected by the optical signal detector 220 and the subcutaneous reflectance according to the change in blood flow of the subject to be measured. It can be calculated using the characteristic that there is a relationship.

More specifically, the operation unit 240 stores the intensity of the signal converted into electric charge by the photoelectric conversion unit 230 as pixel values on the x and y coordinates, and the target region is a small region in which a predetermined blood flow rate is constant. After calculating the average value of the pixel values on the x and y coordinates in each divided small area by , it is possible to calculate the change in the subcutaneous blood flow of the subject to be measured by tracking the change in the average value over time. Hereinafter, a method of calculating a change in subcutaneous blood flow using an equation will be described in more detail.

For each frame of the image captured by the camera unit 100, the intensity of the optical signal reflected from the target area of the subject's face may be stored as pixel values in the x and y coordinates, respectively, and this value is stored as V(x,y). , t). At this time, V(x,y,t) is, as can be seen from [Equation 1] below, the intensity of illumination I(x,y,t) and the reflectance of the skin R(x,y,t) are two factors. It can be composed of the product of

The intensity of illumination I(x, y, t) may mean all the light irradiated on the target area of the face of the measurement target. In this case, a constant intensity of illumination is required only for the target area in which the PPG is measured. The reflectance R(x,y,t) of the skin indicates the intensity of the light reflected from the skin, which can be composed of two types of reflection: the reflection of the skin surface and the reflection of the subcutaneous tissue.

The target region of the subject's face is divided into regions of interests (ROIs) small enough to say that blood flow is constant, and the V(x,y,t) value in each divided small region can be called R. , where y _i (t) is _{the average value of ROI R i} pixel values according to time t, and I{1,2,… ,n} is the index value of R. When a constant light intensity in ROI R _i is expressed _{as I i , y i} (t) can be expressed as in [Equation 2] below.

In [Equation 2] above, I _i represents the light intensity of ROI R _{i , a i} represents the blood flow intensity, b _i represents the reflectance on the skin surface of the face, and q _i (t) is the camera quantization represents noise. When the light of I _i is illuminated on ROI R _i , a large amount of reflection ( b _i ) occurs on the skin surface, and in this case, it is a value that is not affected by changes in blood flow at all. However, a portion of light penetrates the skin surface and the amount of light reflection may vary depending on changes in blood flow, and this value is expressed as p(t). Here, the subcutaneous reflectance can be calculated from _{the two factors ai} and p(t) that contribute to the reflection of light below the skin surface. Accordingly, the calculator 240 may calculate the _{change in the subcutaneous blood flow of the subject by using the characteristic that the ROI R i} value is affected by the subcutaneous reflectance according to the change in the blood flow of the subject to be measured.

The frequency signal converter 250 may convert the signal calculated by the calculator 240 into a frequency signal. That is, the subcutaneous blood flow change signal of the subject to be measured calculated by the calculator 240 may be converted into a frequency signal in the frequency domain.

The controller 260 may process the signal converted by the frequency signal converter 250 . That is, the controller 260 may include a filter unit 261 that filters noise or noise from the PPG signal converted by the frequency signal converter 250 and determines a frequency range of the output PPG signal. Affects the measured optical signal due to changes in the environment, such as the movement of the face or the light illuminating the target area of the subject's face However, such noise may interfere with the detection of heartbeat components. Therefore, it is desirable to remove or reduce the noise component. In general, the noise component is composed of a frequency component lower than the heartbeat component, but does not necessarily have a lower frequency, and may be higher than or equal to the heartbeat component. The control unit 260 may determine a frequency range of the output PPG signal by using a filter such as a bandpass filter, and extract an accurate PPG signal from which a noise component is removed.

The amplifying unit 262 may amplify the PPG signal output after noise or noise has been filtered to an appropriate gain value. That is, the amplification unit 262 can adjust the amplification factor according to the PPG signal of various waveforms obtained by each person by using a variable resistor. In the case of a person with a small waveform, the PPG signal can be amplified to an appropriate gain value.

The heart rate determiner 270 determines the heart rate of the subject from the PPG signal processed by the controller 260 . That is, the number corresponding to the peak value in the waveform of the PPG signal can be determined as the heart rate. For example, the number of peak values in the waveform of the PPG signal in 1 minute can be calculated to obtain the heart rate in 1 minute. have.

The display unit 300 may display the PPG signal calculated by the central processing unit 200 and the heart rate extracted from the PPG signal on the display screen. That is, the display unit 300 may be any device that shows the measurement target to know their PPG signal or heart rate, for example, a liquid crystal display of a television, a liquid crystal display of a computer, or a liquid crystal display of a mobile phone. can be

The stress calculator 400 may calculate the stress index by using the heart rate variability index extracted from the PPG signal processed by the controller 260 and the static correlation between the autonomic nervous system activation due to stress. Here, as the heart rate variability index, SDNN (standard deviation of all RR interval: standard deviation of continuous R peak interval of PPG signal), RMSSD (square root of the mean squared differences of successive normal sinus intervals: corresponding interval of PPG signal) R peak interval continuous root mean square), LF (low frequency: low frequency component), HF (high frequency: high frequency component), or LF/HF (ratio of low frequency component to high frequency component) can be used.

For example, in the SDNN value, the wider the interval, the greater the gradient. In general, the greater the gradient, the healthier the body can be used to calculate the physical fatigue, and the higher the RMSSD value, the greater the stability of the heart. It can be used to know the activity of the parasympathetic nerve. The LF value reflects the sympathetic nervous system activity, and it is affected by mental stress and fatigue. In particular, the acute stress level can be evaluated, and it is correlated with depression or anger, and the HF value reflects the parasympathetic nervous system activity. It is closely related to activity and correlates with long-term stress, anxiety, or fear. The LF/HF value reflects the overall balance of the autonomic nervous system. In general, healthy people have higher LF values than HF values while awake. Therefore, by using the positive correlation between the heart rate variability index extracted from the PPG signal and the autonomic nervous system activation due to stress, the stress index can be measured using the non-contact PPG signal measurement system using the camera of the present invention, and the measured stress index may be displayed on the screen of the display unit 300 .

9 is a diagram illustrating a flow of a method for driving a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention, and FIG. 10 is a PPG in a non-contact PPG signal measuring system using a camera according to an embodiment of the present invention. It is a diagram showing the flow of a method for calculating a signal. 9 to 10 , the non-contact PPG signal measurement method using a camera according to an embodiment of the present invention includes the steps of capturing an image of the subject to be measured (S100), and the subject to be measured from the image captured in step S100. and calculating the PPG signal of (S200), and displaying the PPG signal calculated in the step S200 and the heart rate extracted from the PPG signal on the display screen (S300), wherein the step S200 is the step S100 Step of selecting a target region from which an optical signal is to be extracted from the image of the subject to be measured (S210), and detecting the optical signal reflected from the target region selected in step S210 for each frame of the image taken in step S100 (S220), converting the optical signal detected in step S220 into electric charge (S230), calculating a change in subcutaneous blood flow of the subject to be measured from the signal converted into electric charge in step S230 (S240), the signal calculated in step S240 is converted into a frequency signal and output as a PPG signal (S250), processing the PPG signal converted in step S250 (S260), and determining the heart rate of the subject to be measured from the PPG signal processed in step S260 (S270) ) can be implemented.

Since the details related to each step have been sufficiently described in relation to the non-contact PPG signal measurement system using a camera according to an embodiment of the present invention, the detailed description will be omitted.

Various modifications and applications of the present invention described above are possible by those skilled in the art to which the present invention pertains, and the scope of the technical idea according to the present invention should be defined by the following claims.

100: camera unit
200: central processing unit
210: area selection unit
211: face recognition unit
212: feature point detection unit
213: feature point extraction unit
214: area determining unit
215: video request unit
220: optical signal detection unit
230: photoelectric conversion unit
240: arithmetic unit
250: frequency signal conversion unit
260: control unit
270: heart rate determination unit
300: display unit
400: stress calculator
S100: Step of taking an image of the subject to be measured
S200: Calculating the PPG signal of the subject to be measured from the captured image
S210: Step of selecting a target area to extract an optical signal from the photographed image of the subject to be measured
S220: Detecting the optical signal reflected from the selected target area for each frame of the image
S230: Converting the detected optical signal into electric charge
S240: Calculating the change in the subcutaneous blood flow of the subject to be measured from the signal converted to electric
S250: converting the calculated signal into a frequency signal and outputting it as a PPG signal
S260: processing the converted PPG signal
S270: Determining the heart rate of the subject from the processed PPG signal
S300: displaying the calculated PPG signal and the heart rate extracted from the PPG signal on a display screen

Claims (20)

In the non-contact PPG signal measurement system using a camera,
a camera unit 100 for capturing an image of a heart rate measurement target (hereinafter, referred to as a measurement target);
a central processing unit 200 for calculating the PPG signal of the subject to be measured from the image taken by the camera unit 100; and
and a display unit 300 for displaying the PPG signal calculated by the central processing unit 200 and the heart rate extracted from the PPG signal on a display screen,
The central processing unit 200,
a region selection unit 210 for selecting a target region from which an optical signal is to be extracted from the image of the subject to be measured taken by the camera unit 100;
an optical signal detection unit 220 for detecting the optical signal reflected from the target area selected by the area selection unit 210 for each frame of the image photographed by the camera unit 100;
a photoelectric conversion unit 230 for converting the optical signal detected by the optical signal detection unit 220 into electric charge;
a calculation unit 240 for calculating a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge by the photoelectric conversion unit 230;
a frequency signal converter 250 for converting the signal calculated by the calculator 240 into a frequency signal and outputting it as a PPG signal;
a control unit 260 for processing the PPG signal converted by the frequency signal converting unit 250; and
and a heart rate determination unit 270 for determining the heart rate of the subject to be measured from the PPG signal processed by the control unit 260,
The region selection unit 210,
a face recognition unit 211 for recognizing the face of the subject to be measured from the image taken by the camera unit 100;
a feature point detection unit 212 for detecting all the feature points from the face recognized by the face recognition unit 211;
a feature point extracting unit 213 for extracting only effective feature points effective for face recognition and optical signal extraction of a measurement target from among the feature points detected by the feature point detection unit 212;
a region determining unit 214 for determining a target region from which an optical signal is to be extracted from the effective key points extracted by the key point extraction unit 213; and
When it is inappropriate to recognize the face of the subject to be measured from the image taken by the camera unit 100 or to determine the target region from which the optical signal is to be extracted from the face of the subject, the face of the subject to be measured and an image re-requesting unit 215 for requesting image re-acquisition,
The region determining unit 214 is
Considering the density of capillaries, the degree of movement by facial expressions, and the effect of facial movement within the frame, the effective feature point located on one side of the nose, the effective feature point located at the bottom of the ear, and the effective feature point located on one side of the mouth are connected. Determining the area of a triangle surrounded by a straight line as the target area,
The calculation unit 240,
Calculate the change in the subcutaneous blood flow of the subject to be measured using the feature that there is a correlation between the intensity of the optical signal detected by the optical signal detector 220 and the subcutaneous reflectance according to the change in the blood flow of the subject to be measured,
The calculation unit 240,
The photoelectric conversion unit 230 stores the intensity of the signal converted into electric charge as pixel values on the x and y coordinates, and divides the target region into small regions in which a predetermined blood flow rate can be considered constant, and each divided After calculating the average value of the pixel values on the x, y coordinates in the region, the change in the average value over time is tracked to calculate the change in the subcutaneous blood flow of the subject to be measured,
Further comprising a stress calculator 400 for calculating a stress index by using the heart rate variability index extracted from the PPG signal processed by the controller 260 and the static correlation of the autonomic nervous system activation due to stress,
The heart rate variability indicator is
SDNN (standard deviation of all RR interval) value, RMSSD (square root of the mean squared differences of successive normal sinus intervals) value, LF (low frequency) value, HF (high frequency) value, or LF / HF value A non-contact PPG signal measurement system using a camera. delete delete According to claim 1, wherein the optical signal detection unit 220,
Non-contact PPG signal using a camera, characterized in that when optical signals of a plurality of colors are detected in the image photographed by the camera unit 100, each optical signal data value separated for each color channel is detected for each frame measuring system. The method of claim 4, wherein the optical signal detection unit 220,
A non-contact PPG signal measuring system using a camera, characterized in that the optical signal data value is detected in the green (G) channel, which has the highest light absorption rate of hemoglobin and oxidized hemoglobin. delete delete According to claim 1, wherein the control unit 260,
Non-contact PPG signal using a camera, characterized in that it includes a filter unit 261 that filters noise or noise from the PPG signal converted by the frequency signal converter 250 and determines a frequency range of the output PPG signal. measuring system. The method of claim 8, wherein the control unit 260,
Non-contact PPG signal measurement system using a camera, characterized in that it further comprises an amplifier (262) for amplifying the PPG signal output after the noise or noise has been filtered by the filter unit (261) to an appropriate gain value. delete A non-contact PPG signal measurement method using a camera, comprising:
(1) taking an image of the subject to be measured;
(2) calculating the PPG signal of the subject to be measured from the image taken in step (1); and
(3) displaying the PPG signal calculated in step (2) and the heart rate extracted from the PPG signal on a display screen,
The step (2) is,
(2-1) selecting a target region from which an optical signal is to be extracted from the image of the subject to be measured taken in step (1);
(2-2) detecting the optical signal reflected from the target area selected in step (2-1) for each frame of the image captured in step (1);
(2-3) converting the optical signal detected in step (2-2) into electric charge;
(2-4) calculating a change in the subcutaneous blood flow of the subject to be measured from the signal converted into electric charge in the step (2-3);
(2-5) converting the signal calculated in step (2-4) into a frequency signal and outputting it as a PPG signal;
(2-6) processing the PPG signal converted in the step (2-5); and
(2-7) comprising the step of determining the heart rate of the subject to be measured from the PPG signal processed in the step (2-6),
The step (2-1) is,
(2-1-1) recognizing the face of the subject to be measured from the image captured in step (1);
(2-1-2) detecting all the feature points from the face recognized in the step (2-1-1);
(2-1-3) extracting only effective feature points effective for optical signal extraction among the feature points detected in step (2-1-2);
(2-1-4) determining a target region from which an optical signal is to be extracted from the effective feature points extracted in the step (2-1-3); and
(2-1-5) In case of inappropriate for recognizing the face of the subject to be measured from the image taken in step (1), or inappropriate for determining the target region from which the optical signal is to be extracted from the face of the subject to be measured , comprising the step of requesting re-acquisition of the subject's face image,
In the above step (2-1-4),
Considering the density of capillaries, the degree of movement by facial expressions, and the effect of facial movement within the frame, the effective feature point located on one side of the nose, the effective feature point located at the bottom of the ear, and the effective feature point located on one side of the mouth are connected. Determining the area of a triangle surrounded by a straight line as the target area,
In the above step (2-4),
Calculate the change in the subcutaneous blood flow of the subject to be measured using the feature that there is a correlation between the intensity of the optical signal detected in the step (2-2) and the subcutaneous reflectance according to the change in the blood flow of the subject to be measured,
In the above step (2-4),
The intensity of the signal converted into electric charge in step (2-3) is stored as pixel values on the x and y coordinates, and the target area is divided into small areas in which a predetermined blood flow can be considered constant, and each divided After calculating the average value of the pixel values on the x, y coordinates in the region, the change in the average value over time is tracked to calculate the change in the subcutaneous blood flow of the subject to be measured,
(4) calculating the stress index using the positive correlation between the heart rate variability index extracted from the PPG signal processed in the step (2-6) and the autonomic nervous system activation due to stress,
The heart rate variability indicator is
SDNN (standard deviation of all RR interval) value, RMSSD (square root of the mean squared differences of successive normal sinus intervals) value, LF (low frequency) value, HF (high frequency) value, or LF / HF value A non-contact PPG signal measurement method using a camera. delete delete The method of claim 11, wherein in step (2-2),
Non-contact PPG signal measurement using a camera, characterized in that when optical signals of a plurality of colors are detected in the image taken in step (1), each optical signal data value separated for each color channel is detected for each frame Way. 15. The method of claim 14, wherein in step (2-2),
A method for measuring a non-contact PPG signal using a camera, characterized in that the optical signal data value is detected in the green (G) channel, which is a wavelength having the highest light absorption of hemoglobin and oxidized hemoglobin. delete delete According to claim 11, wherein the step (2-6),
(2-6-1) Filtering the noise or noise from the PPG signal converted in the step (2-5) to determine the frequency range of the output PPG signal, characterized in that it includes a filtering step. Non-contact PPG signal measurement method. The method of claim 18, wherein step (2-6) comprises:
(2-6-2) Non-contact PPG using a camera, characterized in that it further comprises the step of amplifying the PPG signal output after the noise or noise has been filtered in the step (2-6-1) to an appropriate gain value. How to measure the signal. delete

Images