KR20190035312A - Device and method for heart rate measuring based on cntactless sensing - Google Patents

Abstract

The present invention relates to a device for measuring a heart rate and a method thereof and, more specifically, to a device for measuring a heart rate based on contactless sensing by analyzing in a frequency domain region pixel information around blood vessels from obtained image data and a method thereof. The present invention can detect a subtle color change in an obtained image occurring according to a periodic blood flow volume using at least one camera without adding an additional measuring sensor and analyze the detected data in a frequency domain region, thereby accurately measuring a heart rate. In addition, the present invention integrates data obtained from a camera added depending on environments and data obtained through an existing camera at a frequency domain region and can increase accuracy of heart rate measurement using the integrated data.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a non-contact sensing type heart rate measuring apparatus and method,

The present invention relates to an apparatus and method for measuring heart rate, and more particularly, to a heart rate measuring apparatus and method for analyzing pixel information in a frequency domain region around blood vessels in image data acquired based on non-contact type sensing, .

Generally, heart rate measurement data is used in various fields such as psychotherapy, interview, and polygraph detection. Conventionally, a measurement sensor is worn or attached to a part of the body in order to acquire heart rate measurement data, resulting in psychological change and inconvenience of the user.

The background art of the present invention is disclosed in Korean Patent Registration No. 10-1352479 (registered on Apr. 10, 2014).

The present invention provides an apparatus and method for measuring heart rate more accurately by analyzing pixel information around blood vessels in a frequency domain region in an image acquired using a plurality of non-contact type sensing methods.

According to one aspect of the present invention, an apparatus for measuring heart rate is provided.

The apparatus for measuring heart rate according to an embodiment of the present invention includes an image acquiring unit for acquiring a face image by adding a camera of a different type to a reference camera in accordance with a single camera or environment at a position distant from a target object, A data extracting unit for detecting a skin region and extracting a signal necessary for heart rate measurement by detecting skin color information in the detected skin region; a data extracting unit for radially amplifying skin color information detected from RGB data in a Cb-Cr plane, A skin color amplifying unit for amplifying the skin color information detected by the image pickup unit by a bit-shift operation, and a skin color amplifying unit for, when acquiring an image using one or more cameras, And outputs the sum of the amplified signals as a sum (Fourier) may include conversion to detect a frequency band corresponding to the heart rate, and the strongest heart rate per minute to measure the heart rate of the signal in the band parts.

According to another aspect of the present invention, a heart rate measuring method and a computer program for executing the method are provided.

A handwriting recognition method and a computer program for executing the same according to an embodiment of the present invention include performing coordinate calibration of a thermal imaging camera on the basis of an RGB camera at a position distant from a target object, Acquiring RGB data and column image data including a face region of a user by using the HSI or YCbCr color space, separating the RGB data into HSI or YCbCr color space, Detecting a skin region of the thermal image data by using a skin region of the RGB data, detecting skin color information by a pixel value corresponding to a skin region of the RGB data and the thermal image data, Amplifying the skin color information of the data radially in the Cb-Cr plane, Amplifying the skin color information of the thermal image data through a bit shift operation; performing band-pass filtering on the skin color information amplified in each of the RGB data and the thermal image data to extract a signal of a frequency band; Generating a signal by applying a weight to the filtered signals, synthesizing the filtered signals into a single signal, detecting a frequency band corresponding to a heart rate by Fourier transforming the synthesized signal, And measuring the temperature of the liquid.

The present invention can detect fine color changes occurring according to a periodic blood flow in an image acquired using at least one camera without attaching a separate measurement sensor and accurately measure the heart rate by analyzing the sensed data in the frequency domain region .

In addition, data acquired by adding a camera according to the environment and data acquired through an existing camera can be integrated in the frequency domain domain, and the accuracy of heart rate measurement can be improved by using the integrated frequency.

1 to 3 are views for explaining a heart rate measuring apparatus according to an embodiment of the present invention.
4 is a view for explaining a method of measuring a heart rate using an RGB camera and a thermal imaging camera according to an embodiment of the present invention.
5 is a view for explaining a method of calibrating an RGB camera and a thermal imaging camera according to an embodiment of the present invention.
6 and 7 are views illustrating a method of measuring skin color information in RGB data by a heart rate measuring apparatus according to an embodiment of the present invention.
8 is a diagram illustrating a method of measuring skin color information in thermal image data according to an embodiment of the present invention.
9 is a diagram illustrating an example in which a heart rate measuring apparatus according to an embodiment of the present invention converts skin color information of thermal image data into a heart rate signal.
10 to 11 are views for explaining a method of measuring a heart rate using a heart rate signal and a thermal image heart rate signal extracted from RGB data by a heart rate measuring apparatus according to an embodiment of the present invention.
12 is a view for explaining a method of measuring a heart rate using a plurality of cameras according to an embodiment of the present invention.

While the present invention has been described in connection with certain exemplary embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and similarities. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, wherein like reference numerals refer to like or corresponding components throughout. .

1 to 3 are views for explaining a heart rate measuring apparatus according to an embodiment of the present invention.

1, the heart rate measuring apparatus 100 acquires a face image of a subject using one or more cameras such as a thermal image camera 11, an RGB camera 12, and an infrared camera 13, It is possible to extract the skin color information from the facial image acquired through the facial image. The heart rate measuring apparatus 100 can accurately measure the heart rate by synthesizing and analyzing the extracted one or more pieces of skin color information in the frequency domain region.

2, the heart rate measuring apparatus 100 includes an image acquiring unit 110, a data extracting unit 120, a data filtering unit 130, a data synthesizing unit 140, and a heart rate measuring unit 150 .

The image acquiring unit 110 acquires a face image by adding a camera of a different type to the reference camera depending on a single camera or environment at a certain distance from the object. At this time, the image acquisition unit 110 acquires the face image of the object after performing the calibration according to the number of cameras. For example, when using a single camera, the image acquiring unit 110 omits the calibration process. When one or more cameras are used, the coordinates of other cameras are AFFINE converted based on one camera, ) Can be performed. The image obtaining unit 110 according to an exemplary embodiment of the present invention performs coordinate calibration of a thermal image camera based on an RGB camera when measuring heart rate using an RGB camera and a thermal image camera, The angle of view can be matched.

The data extracting unit 120 detects a skin region in the acquired facial image, detects skin color information in the detected skin region, and extracts signals necessary for heart rate measurement. The data extracting unit 120 according to an embodiment of the present invention detects a skin region in a face image acquired using an RGB camera and extracts a face image obtained using a thermal image camera through pixel positions of the skin region It is possible to extract a signal necessary for measuring the heart rate in the skin region.

Referring to FIG. 3, the data extracting unit 120 includes a skin color detecting unit 121 and a skin color amplifying unit 122.

The skin color detection unit 121 separates the RGB data into HSI or YCbCr color space to specify a skin region when the acquired facial image is RGB data, and detects skin color information from pixel values corresponding to the designated skin region. For example, the skin color detecting unit 121 separates RGB data into HSI color space, and detects a hue (HUE) in the range of 0? H? 50. Or RGB data into a YCbCr color space and detects a range corresponding to 77? Cb? 127 and 133? Cr? 173 clustered near the center in the Cb-Cr plane as a skin region. Generally, human skin color varies depending on the illumination or race, but has a characteristic of forming a cluster close to the center of the plane when expressed in the Cb-Cr plane except for the illumination component (Y) in the YCbCr color space. Using this characteristic, the skin color detecting unit 121 can detect the skin region from the acquired face image, and can detect the skin color information from the pixel value of the corresponding region.

In addition, when the acquired face image is RGB data and thermal image data, the skin color detection unit 121 designates a skin region from RGB data by the above-described method, and uses the pixel position of the designated skin region to determine skin region . At this time, the skin color detecting unit 121 can calculate the average temperature using the pixel value corresponding to the skin region of the thermal image data, thereby more accurately designating the skin region. The average temperature may be -4 ° C to + 4 ° C. The skin color detecting unit 121 detects skin color information as a pixel temperature value in the skin region of the thermal image data.

The skin color amplifying unit 122 scales the detected skin color information in both directions by separating the RGB data into the HSI color space or separates the detected skin color information into the YCbCr color space and radially amplifies the detected skin color information in the Cb- The skin color information detected from the thermal image data is amplified by a bit-shift operation.

Referring again to FIG. 2, the data filtering unit 130 performs band pass filtering on the amplified skin color information to extract a signal of a frequency band from which noises have been removed.

When the image is acquired using one or more cameras, the data synthesis unit 140 synthesizes the signals of the skin region extracted from the respective images by applying a weight according to the image acquisition environment and combining the signals into one signal. For example, in case of uneven illumination in the image acquisition environment, the accuracy is high when infrared data is used rather than heart rate estimation through RGB data. Accordingly, the data combining unit 140 may synthesize signals of the skin region extracted from each image by applying different weights according to illumination, distance, and camera performance. At this time, the signal of the skin region is phase-matched so that the correlation coefficient between the signals becomes high because a time difference occurs according to each camera.

The heart rate measuring unit 150 performs a Fourier transform on the sum of the amplified signals to detect a frequency band corresponding to the heart rate and measures the heart rate per minute of the strongest signal in the corresponding band.

4 is a view for explaining a method of measuring a heart rate using an RGB camera and a thermal imaging camera according to an embodiment of the present invention.

Referring to FIG. 4, in step S405, the heart rate measuring apparatus 100 performs coordinate calibration of the thermal imaging camera based on the RGB camera at a position distant from the target object.

In step S410, the heart rate measuring apparatus 100 acquires RGB data and thermal image data including the face region of the user using the calibrated RGB camera and the thermal image camera.

In step S415, the heart rate measuring apparatus 100 separates the RGB data into the HSI or YCbCr color space, designates the skin area, and detects skin color information from pixel values corresponding to the designated skin area. In this case, the skin region separates the RGB data into the HSI color space and detects the color HUE in the range of 0? H? 50 or separates the RGB data into the YCbCr color space. Cb? 127 and 133? Cr? 173 can be detected and specified.

In step S420, the heart rate measuring apparatus 100 specifies a skin region of the thermal image data using the skin region of RGB data. At this time, the skin temperature can be more precisely calculated by calculating the average temperature using the pixel value corresponding to the skin region of the thermal image data. The average temperature may be -4 ° C to + 4 ° C.

In step S425, the heart rate measuring apparatus 100 detects skin color information using pixel values corresponding to skin regions of RGB data and thermal image data.

In step S430, the heart rate measuring apparatus radially amplifies the skin color information of the RGB data in the Cb-Cr plane.

In step S435, the heart rate measuring apparatus 100 amplifies the skin color information of the thermal image data through a bit shift operation.

In step S440, the heart rate measuring apparatus 100 performs band-pass filtering on the skin color information amplified in each of the RGB data and the thermal image data to extract a frequency band signal.

In step S445, the heart rate measuring apparatus 100 applies a weight to the filtered signals and combines them into one signal.

In step S450, the heart rate measuring apparatus 100 performs a Fourier transform on the synthesized signal to detect a frequency band corresponding to the heart rate.

In step S450, the heart rate measuring apparatus 100 measures the heart rate per minute of the strongest signal in the detected frequency band.

FIG. 5 is a view for explaining a method of calibrating an RGB camera and a thermal imaging camera according to an embodiment of the present invention.

Referring to FIG. 5, the heart rate measuring apparatus 100 sets four coordinate points 510 to 540 in the image obtained by the RGB camera, and displays the coordinate points obtained by the thermal imaging camera through the AFFINE transformation The resolution and angle of view of the RGB camera and the thermal imaging camera can be matched by setting coordinate points such as 550 to 580 in FIG.

6 and 7 are views illustrating a method of measuring skin color information in RGB data by the heart rate measuring apparatus according to an embodiment of the present invention.

Referring to FIG. 6, the heart rate measuring apparatus 100 can acquire RGB data including a face area as shown in FIG. 6 (A).

Referring to Fig. 7, the heart rate measuring apparatus 100 can detect the clustered region in the vicinity of the center in the Cb-Cr plane after separating the acquired RGB data into the YCbCr color space as shown in Fig. 7 (A). The heart rate measuring apparatus 100 can radially amplify the region detected in the Cb-Cr plane.

Referring again to FIG. 6, the heart rate measuring apparatus 100 detects a pixel value corresponding to a color corresponding to a region detected in the Cb-Cr plane from RGB data, and detects a skin region as shown in FIG. 6 (B) can do. The detected skin area is used to detect a skin area more quickly, excluding an unnecessary area in the thermal image data.

FIG. 8 is a diagram illustrating a method of measuring skin color information in a thermal image data according to an embodiment of the present invention.

8, the heart rate measuring apparatus 100 acquires thermal image data as shown in FIG. 8 (A), specifies a skin region at the same pixel position as the skin region in RGB data from the acquired thermal image data, The skin color information is detected by the pixel temperature value of the area. The heart rate measuring apparatus 100 can acquire thermal image data as shown in FIG. 8B by amplifying the detected skin color information through a bit-shift operation. At this time, the heart rate measuring apparatus 100 acquires only data corresponding to the temperature close to the skin color based on the amplified skin color information. Since the skin color differs depending on the surrounding environment, it is possible to specify the range by calculating the average temperature of the face region rather than the fixed temperature range. Wherein the average temperature may be -4 to +4.

9 is a diagram illustrating an example of converting skin color information of thermal image data into a heart rate signal according to an embodiment of the present invention.

Referring to FIG. 9, the heart rate measuring apparatus 100 detects skin color information in real time as shown in FIG. 9A and band-pass filters the detected skin color information to remove noise The heartbeat signal can be obtained.

10 to 11 are diagrams for explaining a method of measuring a heart rate using an RGB heartbeat signal and a thermal imaging heartbeat signal according to an embodiment of the present invention.

Referring to FIG. 10, the heart rate measuring apparatus 100 calculates a correlation between a heartbeat signal extracted using an RGB camera and a heartbeat signal extracted using a thermal imaging camera, and synthesizes the correlation into a single signal. In this case, when a time-series signal corresponding to a heartbeat is calculated using each camera, a time difference between signals occurs. Therefore, when the correlation between signals having a large time difference is calculated as shown in FIG. 10 (A) do. In order to solve this problem, it is possible to synthesize the signals by matching the phases of the two signals as shown in FIG. 10 (B) by modifying the camera signals other than the reference camera signal to move leftward and rightward while increasing the correlation coefficient.

Referring to FIG. 11, the heart rate measuring apparatus 100 performs a Fourier transform as shown in FIG. 11 (B) to obtain a synthesized signal whose phases coincide with each other in FIG. 11 (A) It is possible to detect the frequency band in the range of 60 to 100 Hz and measure the frequency of occurrence of the strongest signal in the band to measure the heart rate per minute.

12 is a view for explaining a method of measuring the heart rate using one or more cameras according to an embodiment of the present invention.

Referring to FIG. 12, the heart rate measuring apparatus 100 extracts skin color information from data acquired using a plurality of cameras in addition to RGB cameras and RGB cameras, applies a band-pass filter to a plurality of extracted skin color information, And the heart rate can be measured by performing the Fourier transform.

The method of measuring heart rate according to an embodiment of the present invention may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer readable medium. The computer readable medium may include program instructions, data files, data structures, and the like, alone or in combination. Program instructions to be recorded on a computer-readable medium may be those specially designed and constructed for the present invention or may be available to those skilled in the computer software arts. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Includes hardware devices specifically configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory, and the like. The above-mentioned medium may also be a transmission medium such as a light or metal wire, wave guide, etc., including a carrier wave for transmitting a signal designating a program command, a data structure and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

The embodiments of the present invention have been described above. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

100: Heart rate measuring device
110:
120: Data extraction unit
130: Data filtering unit
140:
150: heart rate measuring unit

Claims (10)

A heart rate measuring apparatus comprising:
An image acquiring unit for acquiring a face image by adding a camera of a different type to a reference camera according to a single camera or environment at a distance from the object;
A data extracting unit for detecting a skin region in the acquired facial image, detecting skin color information in the detected skin region, and extracting signals necessary for heart rate measurement;
A data filtering unit amplifying the skin color information radially in a Cb-Cr plane or amplifying the data by a bit-shift operation;
A data synthesizer for synthesizing a signal of a skin region extracted from each image by applying a weight according to an image acquisition environment to acquire an image using one or more cameras; And
And a heart rate measuring unit for Fourier-transforming the sum of the amplified signals to detect a frequency band in a range of 60 to 100 Hz corresponding to a heart rate, and to measure a heart rate per minute of the strongest signal in the corresponding band.
The apparatus of claim 1, wherein the image obtaining unit
In case of using a single camera, the calibration process is omitted, and when one or more cameras are used, the coordinates of other cameras are AFFINE converted based on one camera to perform calibration, A heart rate measuring device for matching the angle of view.
The apparatus of claim 1, wherein the data extracting unit
A skin color detection unit for separating RGB data into HSI or YCbCr color space to designate a skin area and detecting skin color information from pixel values corresponding to a designated skin area when the acquired facial image is RGB data; And
And a skin color amplifying unit amplifying the skin color information detected in the RGB data radially in the Cb-Cr plane or amplifying the skin color information detected in the thermal image data through a bit-shift operation, .
The image processing apparatus according to claim 3, wherein the skin color detecting section
When the acquired facial image is RGB data and column image data, the skin region of the column image data is designated by using the pixel position of the skin region designated in the RGB data
Heart rate measuring device.
The image processing apparatus according to claim 3, wherein the skin color detecting section
After separating the RGB data into the HSI color space, the color HUE in the range of 0? H? 50 is detected, or the RGB data is separated into the YCbCr color space, Cb? 127 and 133? Cr? 173 clustered in the skin region.
The image processing apparatus according to claim 3, wherein the skin color detecting section
Calculating an average temperature of the pixel temperature corresponding to the skin region of the thermal image data to specify the skin region more accurately, wherein the average temperature is -4 to +4 DEG C.
A method for measuring heart rate using a heart rate measuring device,
Performing coordinate calibration of the thermal imaging camera based on the RGB camera at a position a certain distance from the object;
Acquiring RGB data and thermal image data including a face region of a user using the calibrated RGB camera and thermal image camera;
Separating the RGB data into HSI or YCbCr color space, and detecting a clustered region in the vicinity of the center in the Cb-Cr plane as a skin region of RGB data;
Designating a skin region of the thermal image data using a skin region of the RGB data;
Detecting skin color information using a pixel value corresponding to a skin region of the RGB data and the thermal image data;
Amplifying skin color information of the RGB data radially in a Cb-Cr plane;
Amplifying skin color information of the thermal image data through a bit shift operation;
Performing band-pass filtering on the skin color information amplified in each of the RGB data and the thermal image data to extract a frequency band signal;
Applying the weights to the filtered signals and synthesizing them into one signal;
Detecting a frequency band in a range of 60 to 100 Hz corresponding to a heart rate by Fourier transforming the synthesized signal; And
Measuring a heart rate per minute of the strongest signal in the detected frequency band.
The method as claimed in claim 7, wherein the step of separating the RGB data into HSI or YCbCr color space and then detecting a clustered region in the vicinity of the center in the Cb-Cr plane as a skin region of RGB data
After separating the RGB data into the HSI color space, the color HUE in the range of 0? H? 50 is detected, or the RGB data is separated into the YCbCr color space, Cb? 127 and 133? Cr? 173 clustered in the skin region.
The method of claim 7, wherein the skin region designation of the thermal image data using the skin region of the RGB data comprises:
Wherein a skin region is more accurately specified using an average temperature by using pixel values corresponding to a skin region of the thermal image data, and the average temperature is in a range of -4 to +4 [deg.] C.
A computer program for executing the method for measuring heart rate according to any one of claims 7 to 9 and recorded on a computer-readable recording medium.

Images