KR102018853B1 - Device and method for heart rate measuring based on cntactless sensing - Google Patents

Abstract

The present invention relates to an apparatus and method for measuring heart rate, and more particularly, to an apparatus and method for measuring heart rate by analyzing pixel information around a blood vessel in a frequency domain region from image data acquired based on contactless sensing. It is about. According to the present invention, the heart rate can be accurately measured by detecting minute color changes occurring according to periodic blood flow in an image acquired using at least one camera without attaching a separate measuring sensor, and analyzing the detected data in a frequency domain region. .
In addition, according to the present invention, data acquired by adding a camera and data acquired through an existing camera may be integrated in a frequency domain region, and heart rate measurement accuracy may be improved by using the integrated frequency.

Description

Non-contact sensing based heart rate measuring device and method {DEVICE AND METHOD FOR HEART RATE MEASURING BASED ON CNTACTLESS SENSING}

The present invention relates to an apparatus and method for measuring heart rate, and more particularly, to an apparatus and method for measuring heart rate by analyzing pixel information around a blood vessel in a frequency domain region from image data acquired based on contactless sensing. It is about.

In general, heart rate measurement data is used in various fields such as psychotherapy, interviews, and lie detection. Conventionally, since a measurement sensor is worn or attached to a part of the body in order to acquire heart rate measurement data, psychological changes and inconveniences of a user are caused.

Background art of the present invention is disclosed in Republic of Korea Patent Publication No. 10-1352479 (registered on January 10, 2014).

The present invention provides a heart rate measuring apparatus and method for more accurately measuring heart rate by analyzing pixel information around blood vessels in a frequency domain region in images acquired using a plurality of non-contact sensing.

According to one aspect of the invention, a heart rate measuring apparatus is provided.

An apparatus for measuring heart rate according to an embodiment of the present invention includes an image acquisition unit for acquiring a face image by adding a camera different from a reference camera according to a single camera or an environment at a predetermined distance from an object, A data extractor which detects a skin region, detects skin color information in the detected skin region, and extracts a signal necessary for measuring heart rate, and amplifies the skin color information detected in RGB data in a Cb-Cr plane radially or by thermal image data. Skin amplification unit for amplifying the skin color information detected by using a bit-shift operation, when the image is obtained by using at least one camera, the signal of the skin region extracted from each image is weighted according to the image acquisition environment Solve the sum of the data synthesizer and the amplified signals synthesized into a single signal by applying (Fourier) may include conversion to detect a frequency band corresponding to the heart rate, and the strongest heart rate per minute to measure the heart rate of the signal in the band parts.

According to another aspect of the present invention, a heart rate measuring method and a computer program for executing the same is provided.

A handwriting recognition method and a computer program for executing the same according to an embodiment of the present invention may include performing coordinate calibration of a thermal imaging camera based on an RGB camera at a distance away from an object, a calibrated RGB camera and a thermal imaging camera. Acquiring RGB data and thermal image data including a face region of a user using the same; separating the RGB data into an HSI or YCbCr color space, and then clustering an area clustered near a center in a Cb-Cr plane into a skin region of RGB data. Detecting skin information using the skin region of the RGB data; detecting skin color information with corresponding pixel values in the RGB data and the skin region of the thermal image data; Amplifying the skin color information of the data radially in the Cb-Cr plane, Amplifying the skin color information of the thermal image data through a bit shift operation, and performing band-pass filtering on the skin color information amplified from each of the RGB data and the thermal image data to extract a signal of a frequency band Synthesizing the filtered signals into a single signal by weighting the filtered signals; Fourier transforming the synthesized signals to detect a frequency band corresponding to a heart rate; and heart rate per minute of the strongest signal in the detected frequency band. It may include the step of measuring.

According to the present invention, the heart rate can be accurately measured by detecting minute color changes occurring according to periodic blood flow in an image acquired using at least one camera without attaching a separate measuring sensor, and analyzing the detected data in a frequency domain region. .

In addition, according to the environment, data acquired by adding a camera and data acquired through an existing camera may be integrated in the frequency domain region, and the integrated frequency may be used to improve heart rate measurement accuracy.

1 to 3 are views for explaining a heart rate measuring apparatus according to an embodiment of the present invention.
4 is a view for explaining a method of measuring the heart rate using an RGB camera and a thermal imaging camera in the heart rate measuring apparatus according to an embodiment of the present invention.
5 is a view for explaining a method of calibrating an RGB camera and a thermal imaging camera by a heart rate measuring apparatus according to an exemplary embodiment.
6 and 7 illustrate a method in which a heart rate measuring apparatus according to an embodiment of the present invention detects skin color information from RGB data.
8 is a diagram illustrating a method of detecting skin color information from thermal image data by a heart rate measuring apparatus according to an exemplary embodiment.
9 is a diagram illustrating an example in which the heart rate measuring apparatus according to an embodiment of the present invention converts skin color information of thermal image data into a heart rate signal.
10 to 11 are diagrams for describing a method of measuring a heart rate using a heart rate signal and a thermal image heart rate signal extracted from RGB data by a heart rate measuring apparatus according to an embodiment of the present invention.
12 is a diagram for describing a method of measuring a heart rate by using a plurality of cameras in a heart rate measuring apparatus according to an exemplary embodiment;

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In describing the present invention, when it is determined that the detailed description of the related known technology may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components are given the same reference numerals and redundant description thereof will be omitted. Shall be.

1 to 3 are views for explaining a heart rate measuring apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the heart rate measuring apparatus 100 acquires an image of a subject's face by using one or more cameras, such as a thermal imaging camera 11, an RGB camera 12, an infrared camera 13, and the like. Skin color information may be extracted from the acquired face image through. The heart rate measuring apparatus 100 may accurately measure the heart rate by synthesizing the extracted one or more skin color information in the frequency domain region.

2, the heart rate measuring apparatus 100 includes an image acquirer 110, a data extractor 120, a data filter 130, a data synthesizer 140, and a heart rate measurer 150. .

The image acquisition unit 110 acquires a face image by adding a camera different from the reference camera according to a single camera or environment at a position away from the object. In this case, the image acquisition unit 110 acquires a face image of the object after performing calibration according to the number of cameras. For example, the image acquisition unit 110 omits the calibration process when using a single camera, and when using one or more cameras, performs calibration by affine transforming coordinates of other cameras based on one camera. ) Can be performed. The image acquisition unit 110 according to an embodiment of the present invention performs coordinate calibration of a thermal imaging camera based on an RGB camera when measuring a heart rate using an RGB camera and a thermal imaging camera, and determines the position, resolution, and resolution of the camera. You can match the angle of view.

The data extractor 120 detects a skin region in the acquired face image, and extracts a signal for measuring heart rate by detecting skin color information in the detected skin region. The data extractor 120 according to an embodiment of the present invention detects a skin region from a face image acquired using an RGB camera, and uses a thermal image camera to detect a skin region through a pixel position of the detected skin region. The skin region may be detected at, and a signal for measuring heart rate may be extracted from the skin region.

Referring to FIG. 3, the data extractor 120 includes a skin color detector 121 and a skin color amplifier 122.

When the acquired face image is RGB data, the skin color detector 121 separates the RGB data into an HSI or YCbCr color space to designate a skin region, and detects skin color information from pixel values corresponding to the specified skin region. For example, the skin color detector 121 separates RGB data into an HSI color space to detect a color HUE in a range of 0 ≦ H ≦ 50. Alternatively, the RGB data is separated into the YCbCr color space, and then the ranges corresponding to 77≤Cb≤127 and 133≤Cr≤173 clustered near the center in the Cb-Cr plane are detected as the skin region. In general, the skin color of a person varies depending on the lighting and race, but when expressed in the Cb-Cr plane excluding the lighting component (Y) in the YCbCr color space, there is a characteristic of forming a cluster near the center of the plane. The skin color detector 121 may detect the skin region from the acquired face image and detect the skin color information from the pixel value of the corresponding region by using this characteristic.

In addition, when the acquired face image is RGB data and thermal image data, the skin color detection unit 121 designates a skin region in the RGB data by the above-described method, and uses the pixel position of the designated skin region to skin the skin region of the thermal image data. Specify. In this case, the skin color detector 121 may more accurately designate the skin region by calculating an average temperature using a pixel value corresponding to the skin region of the thermal image data. Herein, the average temperature may be -4 ° C to + 4 ° C. The skin color detector 121 detects skin color information as a pixel temperature value in the skin region of the thermal image data.

The skin color amplifying unit 122 scales the skin color information detected by separating the RGB data into the HSI color space in both directions, or amplifies the skin color information detected by separating the YCbCr color space in the Cb-Cr plane in a radial manner. Skin color information detected in the thermal image data is amplified by bit-shift operation.

Referring back to FIG. 2, the data filtering unit 130 may perform band pass filtering on the amplified skin color information to extract a signal of a frequency band from which noise is removed.

When the image synthesizing unit 140 acquires an image using one or more cameras, the data synthesis unit 140 synthesizes a signal of the skin region extracted from each image into one signal by applying weights according to an image acquisition environment. For example, if the lighting is uneven in the image acquisition environment, the accuracy is higher when using infrared data rather than heart rate estimation using RGB data. Accordingly, the data synthesizing unit 140 may synthesize signals of skin regions extracted from each image by applying different weights according to illumination, distance, and camera performance. At this time, the signal in the skin region is phased so that the correlation coefficient between signals is high because time difference occurs according to each camera.

The heart rate measuring unit 150 performs Fourier transform on the sum of the amplified signals to detect a frequency band corresponding to the heart rate, and measures the heart rate per minute of the strongest signal in the band.

4 is a diagram for describing a method of measuring a heart rate using an RGB camera and a thermal imaging camera according to an exemplary embodiment.

Referring to FIG. 4, in operation S405, the heart rate measuring apparatus 100 performs coordinate calibration of a thermal image camera based on an RGB camera at a position away from the object.

In operation S410, the heart rate measuring apparatus 100 acquires RGB data and thermal image data including a face region of the user using a calibrated RGB camera and a thermal image camera.

In operation S415, the heart rate measuring apparatus 100 separates RGB data into an HSI or YCbCr color space, specifies a skin region, and detects skin color information from pixel values corresponding to the designated skin region. Here, the skin region separates the RGB data into the HSI color space to detect a color (HUE) in a range of 0≤H≤50, or separates the RGB data into the YCbCr color space and then clusters 77≤ in the Cb-Cr plane near the center The ranges Cb≤127 and 133≤Cr≤173 can be detected and specified.

In operation S420, the heart rate measuring apparatus 100 designates the skin region of the thermal image data by using the skin region of the RGB data. In this case, the skin temperature may be more accurately specified by calculating an average temperature using a pixel value corresponding to the skin region of the thermal image data. Herein, the average temperature may be -4 ° C to + 4 ° C.

In operation S425, the heart rate measuring apparatus 100 detects skin color information using corresponding pixel values in skin regions of RGB data and thermal image data.

In operation S430, the heart rate measuring apparatus amplifies the skin color information of the RGB data radially in the Cb-Cr plane.

In operation S435, the heart rate measuring apparatus 100 amplifies skin color information of the thermal image data through a bit shift operation.

In operation S440, the heart rate measuring apparatus 100 performs band-pass filtering on the skin color information amplified from the RGB data and the thermal image data, and extracts a signal of a frequency band.

In operation S445, the heart rate measuring apparatus 100 synthesizes the filtered signals into one signal by applying weights to the filtered signals.

In operation S450, the heart rate measuring apparatus 100 detects a frequency band corresponding to the heart rate by performing Fourier transform on the synthesized signal.

In operation S450, the heart rate measuring apparatus 100 measures the heart rate per minute of the strongest signal in the detected frequency band.

5 is a diagram for describing a method of calibrating an RGB camera and a thermal image camera by a heart rate measuring apparatus according to an exemplary embodiment.

Referring to FIG. 5, the heart rate measuring apparatus 100 sets four coordinate points, such as 510 to 540, in an image obtained by an RGB camera, and obtains the image by the thermal imaging camera through an affine transformation. By setting the coordinate points as 550 to 580 at, the resolution and angle of view of the RGB camera and the thermal imaging camera can be matched.

6 and 7 illustrate a method of detecting skin color information from RGB data by a heart rate measuring apparatus according to an exemplary embodiment.

Referring to FIG. 6, the heart rate measuring apparatus 100 may acquire RGB data including a face region as illustrated in FIG. 6A.

Referring to FIG. 7, the heart rate measuring apparatus 100 may separate the acquired RGB data into the YCbCr color space as shown in FIG. 7A, and then detect areas clustered near the center in the Cb-Cr plane. The heart rate measuring apparatus 100 may radially amplify the area detected in the Cb-Cr plane.

Referring back to FIG. 6, the heart rate measuring apparatus 100 detects a pixel value corresponding to a color corresponding to an area detected in the Cb-Cr plane from RGB data, and detects a skin region as shown in FIG. 6B. can do. The detected skin region is used to detect the skin region faster than the unnecessary region in the thermal image data.

8 is a diagram illustrating a method of detecting skin color information from thermal image data by a heart rate measuring apparatus according to an exemplary embodiment.

Referring to FIG. 8, the heart rate measuring apparatus 100 acquires thermal image data as shown in FIG. 8A, and designates the skin region at the same pixel position as the skin region in the RGB data from the acquired thermal image data. The skin color information is detected by the pixel temperature value of the region. The heart rate measuring apparatus 100 may acquire thermal image data as shown in FIG. 8B by amplifying the detected skin color information through a bit-shift operation. In this case, the heart rate measuring apparatus 100 acquires only data corresponding to a temperature close to the skin color based on the amplified skin color information. Since the skin color varies depending on the surrounding environment, the range can be specified by calculating the average temperature of the face area rather than the fixed temperature range. Here, the average temperature may be -4 to +4.

9 is a diagram illustrating an example of converting skin color information of thermal image data into a heart rate signal according to an exemplary embodiment.

Referring to FIG. 9, the heart rate measuring apparatus 100 detects skin color information in real time as shown in FIG. 9A, and performs band pass filtering on the detected skin color information to remove noise as illustrated in FIG. 9B. Heart rate signal can be obtained.

10 to 11 are diagrams for describing a method of measuring a heart rate using an RGB heartbeat signal and a thermal image heartbeat signal according to an exemplary embodiment.

Referring to FIG. 10, the heart rate measuring apparatus 100 calculates a correlation between a heartbeat signal extracted by using an RGB camera and a heartbeat signal extracted by using a thermal imager, and synthesizes the signal into a single signal. In this case, when the time series signals corresponding to the heart rate are calculated using each camera, the time difference between the signals is generated. Therefore, when the correlation between the signals having the large time difference is calculated as shown in FIG. do. In order to solve this problem, it is possible to synthesize signals by matching the phases between the two signals as shown in FIG.

Referring to FIG. 11, the heart rate measuring apparatus 100 performs Fourier transform on a synthesized signal having the same phase between two signals of FIG. 11A as shown in FIG. 11B, corresponding to a heart rate. The heart rate per minute may be measured by detecting a frequency band in the range of 60 to 100 hz and measuring the number of occurrences of the strongest signal in the corresponding band.

FIG. 12 is a diagram for describing a method of measuring a heart rate by using one or more cameras, according to an embodiment of the present invention.

Referring to FIG. 12, the heart rate measuring apparatus 100 extracts skin color information from data acquired using a plurality of cameras in addition to an RGB camera and an RGB camera, and applies the extracted heartbeat signal to a plurality of extracted skin color information by applying a band pass filter. The heart rate can be measured by synthesizing and performing a Fourier transform.

Heart rate measurement method according to an embodiment of the present invention is implemented in the form of program instructions that can be executed by various computer means may be recorded on a computer readable medium. Computer-readable media may include, alone or in combination with the program instructions, data files, data structures, and the like. The program instructions recorded on the computer readable medium may be those specially designed and constructed for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs, DVDs, and magnetic disks, such as floppy disks. Hardware devices specially configured to store and execute program instructions such as magneto-optical media and ROM, RAM, flash memory and the like. In addition, the above-described medium may be a transmission medium such as an optical or metal wire, a waveguide, or the like including a carrier wave for transmitting a signal specifying a program command, a data structure, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

So far, the present invention has been described with reference to the embodiments. Those skilled in the art will appreciate that the present invention can be implemented in a modified form without departing from the essential features of the present invention. Therefore, the disclosed embodiments should be considered in descriptive sense only and not for purposes of limitation. The scope of the present invention is shown in the claims rather than the foregoing description, and all differences within the scope will be construed as being included in the present invention.

100: heart rate measuring device
110: image acquisition unit
120: data extraction unit
130: data filtering unit
140: data synthesis unit
150: heart rate measurement unit

Claims (10)

delete In the heart rate measuring device,
The heart rate measuring device,
An image acquisition unit for acquiring a face image by adding a camera different from the reference camera according to a single camera or environment at a distance away from the object;
A data extracting unit detecting a skin region in the acquired face image, extracting a signal for measuring heart rate by detecting skin color information in the detected skin region;
A data filtering unit for amplifying the skin color information radially in the Cb-Cr plane or through a bit-shift operation;
A data synthesizer for synthesizing a signal of a skin region extracted from each image into a single signal by applying weights according to an image acquisition environment when an image is acquired using at least one camera; And
Heart rate measurement unit for Fourier (Fourier) conversion of the sum of the amplified signals to detect a frequency band of 60 to 100hz range corresponding to the heart rate, and measures the heart rate per minute of the strongest signal in the band;
If a single camera is used, the calibration process is skipped, and if more than one camera is used, a calibration is performed by affineing the coordinates of other cameras based on one camera to perform the calibration. Heart rate measuring device to match the angle of view.
In the heart rate measuring device,
The heart rate measuring device,
An image acquisition unit for acquiring a face image by adding a camera different from the reference camera according to a single camera or environment at a distance away from the object;
A data extracting unit detecting a skin region in the acquired face image, extracting a signal for measuring heart rate by detecting skin color information in the detected skin region;
A data filtering unit for amplifying the skin color information radially in the Cb-Cr plane or through a bit-shift operation;
A data synthesizer for synthesizing a signal of a skin region extracted from each image into a single signal by applying weights according to an image acquisition environment when an image is acquired using at least one camera; And
Heart rate measurement unit for Fourier (Fourier) conversion of the sum of the amplified signals to detect a frequency band of 60 to 100hz range corresponding to the heart rate, and measures the heart rate per minute of the strongest signal in the band;
The data extraction unit
A skin color detector which separates RGB data into an HSI or YCbCr color space to designate a skin region and detects skin color information from pixel values corresponding to the designated skin region when the acquired face image is RGB data; And
And a skin color amplifying unit for amplifying the skin color information detected from the RGB data in a Cb-Cr plane radially or amplifying the skin color information detected from the thermal image data through a bit-shift operation. .
According to claim 3, wherein the skin color detection unit
When the acquired face image is RGB data and thermal image data, the skin region of the thermal image data is designated using the pixel position of the skin region specified in the RGB data.
Heart rate measuring device.
According to claim 3, wherein the skin color detection unit
After separating RGB data into HSI color space, detecting color (HUE) in the range 0≤H≤50, or separating RGB data into YCbCr color space and then near the center of the Cb-Cr plane except illumination component (Y) A heart rate measuring apparatus for detecting the areas of 77≤Cb≤127 and 133≤Cr≤173 clustered in the skin region.
According to claim 3, wherein the skin color detection unit
A device for designating a skin region more accurately by calculating an average temperature of pixel temperatures corresponding to a skin region of thermal image data, wherein the average temperature is -4 ° C to + 4 ° C.
In the method of measuring the heart rate using a heart rate measuring device,
Performing a coordinate calibration of the thermal imaging camera with respect to the RGB camera at a position away from the object by the heart rate measuring apparatus;
Acquiring, by the heart rate measuring apparatus, RGB data and thermal image data including a face region of a user using a calibrated RGB camera and a thermal image camera;
Separating the RGB data into the HSI or YCbCr color space by the heart rate measuring apparatus and detecting a clustered area near the center of the Cb-Cr plane as a skin region of the RGB data;
Designating, by the apparatus for measuring heart rate, a skin region of the thermal image data using the skin region of the RGB data;
Detecting, by the apparatus for measuring heart rate, skin color information using pixel values corresponding to skin regions of the RGB data and the thermal image data;
Amplifying the skin color information of the RGB data radially in a Cb-Cr plane by the heart rate measuring apparatus;
Amplifying the skin color information of the thermal image data by a bit shift operation by the heart rate measuring apparatus;
Extracting a frequency band signal by performing a band-pass filtering on skin color information amplified from the RGB data and the thermal image data by the heart rate measuring apparatus;
Synthesizing the signal into a single signal by applying weights to the filtered signals;
Detecting a frequency band in a range of 60 to 100 hz corresponding to a heart rate by Fourier transforming the synthesized signal by the heart rate measuring apparatus; And
And measuring the heart rate per minute of the strongest signal in the detected frequency band by the heart rate measuring apparatus.
The method of claim 7, wherein after separating the RGB data into the HSI or YCbCr color space, detecting a clustered area near the center of the Cb-Cr plane as a skin region of the RGB data.
The heart rate measuring device separates RGB data into an HSI color space and then detects a color (HUE) in a range of 0≤H≤50, or separates RGB data into a YCbCr color space and then removes Cb- excluding illumination component (Y). A method of measuring heart rate that detects the range of 77≤Cb≤127 and 133≤Cr≤173 clustered near the center in the Cr plane as the skin region.
The method of claim 7, wherein the step of designating the skin region of the thermal image data using the skin region of the RGB data,
The heart rate measuring apparatus designates a skin region using an average temperature using a pixel value corresponding to a skin region of thermal image data, wherein the average temperature is in a range of -4 ° C to + 4 ° C.
A computer program which executes the method for measuring heart rate according to any one of claims 7 to 9 and recorded in a computer-readable recording medium.

Images