KR102468648B1 - Method for calculating heart rate using rPPG signal of serial image and system thereof - Google Patents

Abstract

One embodiment of the present invention provides a method for calculating a heart rate of a person using a remote photoplethysmogram signal for an image, which includes the steps of: obtaining an image; setting and extracting a first region including a human face from the obtained image; converting the extracted first region into a mesh structure and setting a second region in which a part of the converted mesh structure is removed; and calculating a heart rate of a person included in the image based on a processing result of a remote PPG signal for the set second region.

Description

[Method for calculating heart rate using rPPG signal of serial image and system thereof}

The present invention relates to a method for calculating the heart rate of a person included in an image, and more specifically, to a method for calculating the heart rate of a person by analyzing characteristics of a specific part of a person in an image, and based on the result of the analysis, and It is about a system for implementing the method.

As the resolution of digital cameras gradually increases, information that can be obtained by analyzing images acquired through digital cameras is also diversifying. As a recently known technique, a method of estimating a person's heart rate based on the tone of a person's face included in an image has also been introduced.

Republic of Korea Patent Publication No. 10-2019-0137893 (published on December 11, 2019)

A technical problem to be solved by the present invention is to provide a method for accurately identifying a person's face in an image and calculating the person's heart rate, and a system for implementing the method.

A method according to an embodiment of the present invention for solving the above technical problem includes obtaining an image; setting and extracting a first region including a human face from the obtained image; converting the extracted first region into a mesh structure, and setting a second region by removing a part of the converted mesh structure; and calculating a heart rate of a person included in the image based on a processing result of a remote PPG signal for the set second region.

In the above method, the image may be a continuous image composed of at least two or more frames.

In the above method, the setting of the second area may include: converting the extracted first area into a mesh structure having triangles as a basic unit; and setting the second region by removing some triangles from the converted mesh structure.

In the method, the converting to the mesh structure may include: capturing a predetermined number of feature points in the extracted first region; and performing Delaunay triangulation based on the feature points.

System according to another embodiment of the present invention for solving the above technical problem, the communication unit for obtaining an image from an external device; and a processing unit, wherein the processing unit sets and extracts a first region including a human face from the obtained image, converts the extracted first region into a mesh structure, and converts a portion of the converted mesh structure. A second area from which is removed is set, and a heart rate of a person included in the image is calculated based on a processing result of a remote PPG signal for the set second area.

An embodiment of the present invention may provide a computer readable recording medium storing a program for executing the method.

According to the present invention, the heart rate of a person included in an image can be accurately calculated.

1 is a block diagram showing modules constituting a system according to the present invention.
FIG. 2 is a block diagram illustrating another example of the processing unit described in FIG. 1 .
3 is a diagram for explaining an example of a process of setting a first area by a first area setting unit.
4 is a diagram for explaining an example of a process of converting a first region into a mesh structure by a second region setting unit.
5 is a diagram schematically illustrating an example of a process of setting a second area by a second area setting unit.
6 is a flowchart illustrating an example of a method according to the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and a method of achieving them will become clear with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are given the same reference numerals, and overlapping descriptions thereof will be omitted. .

In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning.

In the following embodiments, singular expressions include plural expressions unless the context clearly indicates otherwise.

In the following embodiments, terms such as include or have mean that features or elements described in the specification exist, and do not preclude the possibility that one or more other features or elements may be added.

When an embodiment is otherwise embodied, a specific process sequence may be performed differently from the described sequence. For example, two processes described in succession may be performed substantially simultaneously, or may be performed in an order reverse to the order described.

1 is a block diagram showing modules constituting a system according to the present invention.

Referring to FIG. 1 , a system 100 for calculating a person's heart rate using a remote photoplethysmogram signal for an image according to the present invention includes a database 110, a communication unit 130, a processing unit 150, and an output unit ( 170). Hereinafter, the system 100 for calculating a person's heart rate by using a remote photoplethysmogram of an image according to the present invention will be abbreviated as a heart rate calculation system 100.

The heart rate calculation system 100 according to an embodiment of the present invention may correspond to one or more processors or include one or more processors. Accordingly, the heart rate calculation system 100 and the communication unit 130, the processing unit 150, and the output unit 170 included in the heart rate calculation system 100 are included in a hardware device such as a microprocessor or a general-purpose computer system. can be driven by

The name of each module included in the heart rate calculation system 100 shown in FIG. 1 is arbitrarily named to intuitively describe the representative function performed by each module, and when the heart rate calculation system 100 is actually implemented, Each module may be given a name different from that described in FIG. 1 .

In addition, the number of modules included in the heart rate calculation system 100 of FIG. 1 may vary depending on the embodiment. More specifically, the heart rate calculation system 100 of FIG. 1 includes a total of four modules, but according to embodiments, at least two or more modules are integrated into one module, or at least one or more modules are combined into two or more modules. It may be implemented in a separate form.

The database 110 stores various data necessary for the heart rate calculation system 100 to operate. As an example, the database 110 stores an integrated management program for controlling the operation of the heart rate calculation system 100, and the database 110 may receive and store data received by the communication unit 130.

The communication unit 130 may perform communication with an internal storage device such as the database 110 or an external storage device and obtain an image. The image acquired by the communication unit 130 may basically be a photographed image of a person's face, but is not limited thereto. For example, the heart rate calculation system 100 according to the present invention may apply the process according to the characteristics of the present invention to all acquired images. The communication unit 130 may include a module for accessing and authenticating a communication network in order to use various wired and wireless communication networks such as a data network, a mobile communication network, and the Internet.

The processing unit 150 processes data received by the communication unit 130 and data to be transmitted. As an example, the processing unit 150 may analyze an image received from any one of the database 110 and the communication unit 130 to calculate the heart rate of a person included in the image. As another example, a process in which the processing unit 150 detects a face region of a person in an image received from any one of the database 110 and the communication unit 130 and excludes some skin regions from the detected face region. can also handle The image analyzed by the processing unit 150 may be a serial image composed of a plurality of frames. Hereinafter, unless otherwise stated, the image is not a still image that is fixed over time, but a time-dependent image. It is regarded as a video in which the output result changes according to the flow. For example, the video processing unit 150 receives from the database 110 or the communication unit 130 has an extension of mp4, m4v, avi, wmv, mwa, asf, mpg, mpeg, ts, mkv, mov, 3gp, 3g2 , and may be a file that is any one of webm, but is not limited thereto. In addition to the above-described embodiments, the processing unit 150 performs a function of overall data processing necessary to implement the method according to the present invention.

The output unit 170 receives a command from the processing unit 150 and performs a function of calculating and outputting various data. As an example, the output unit 170 may output result data processed by the processing unit 150 and transmit it to the communication unit 130 .

FIG. 2 is a block diagram illustrating another example of the processing unit described in FIG. 1 .

Referring to FIG. 2 , it can be seen that the processing unit 150 includes an image acquisition unit 151, a first region setting unit 153, a second region setting unit 155, and a heart rate calculator 157. Since the image acquisition unit 151, the first region setting unit 153, the second region setting unit 155, and the heart rate calculation unit 157 are sub-modules included in the processing unit 150, at least one or more processors as described above. , or may be a module capable of independent operation including at least one processor. In addition, in FIG. 2, the processing unit 150 is illustrated as including a total of four sub-modules, but according to embodiments, at least one or more modules shown in FIG. 2 are included in another module, or at least two or more modules are included in one module. It may be implemented in an integrated form as a module of. Hereinafter, it will be described with reference to FIGS. 1 and 2 .

The image acquisition unit 151 performs a function of acquiring images from the database 110, the communication unit 130 and other external devices. The image acquisition unit 151 not only performs a function of simply receiving an image, but also analyzes the image, such as basic information data on a dummy section and image format, which do not substantially contain content in the acquired image. You can perform a function to remove data that is not necessary for

As another example, the image acquisition unit 151 may perform a function of dividing the obtained image into frame units and transmitting the obtained image to the first region setting unit 153, and the image acquisition unit 151 may If the acquired image is an image of a type (format) that cannot be analyzed, it may wait until another image is acquired without dividing it into frame units.

The first region setting unit 153 may receive an image from the image acquisition unit 151 and extract a first region including a human face from the image.

As an example, the first region setting unit 153 may set and extract the first region from the image. Here, the first domain (1st domain) is derived by a boundary line for setting a part of a frame constituting an image, and may have a certain area within the resolution of the image. For example, if the resolution of an image is 1028*720, the first area of the image may be a 300*200 rectangular area or a 600*600 square area. The size and shape of the first region described above are illustrative, and may vary according to embodiments. The first region extracted by the first region setting unit 153 cannot exceed the resolution of the image. For example, when a 1028*768 size image composed of 740160 pixels is obtained, the total number of pixels of the first areas extracted from the image cannot exceed 740160, and the number of pixels in the first area The number of horizontal and vertical pixels is limited to 1028 and 768 respectively.

The first region setting unit 153 may determine a region estimated to be a human face in the image, set a size of the first region based on the size of the determined region, and extract the region. In the process of setting and extracting the first region by the first region setting unit 153, color, location, and body structure characteristics may be considered, and information on color, location, and body structure for determining a person's face is stored in the database 110 or stored in the form of a library (DLL) of the first area setting unit 153. In addition, the first region setting unit 153 may detect a part of a person's face in an image using OpenCV (OPENsource Computer Vision library), and distinguish the detected face from non-face skin.

As another example, the first region setting unit 153 may divide the acquired image into frames, learn all of the separated frames, and then set the first region based on the learning result. The first region setting unit 153 may include a learning model capable of learning and discriminating a person's shape, a person's face, and a person's posture included in the image. At this time, the learning model refers to a model capable of repeating learning through a machine learning algorithm set inside the model and distinguishing a person and a person's body included in the image, and the first region setting unit 153 physically Or it can be implemented logically. The learning model may include an algorithm such as an Artificial Neural Network (ANN), a Recurrent Neural Network (RNN), a Deep Neural Network (DNN), an autoencoder, etc., but is not limited thereto. In addition, the learning model of the first area setting unit 153 is used to set and extract a first area including a human face, to distinguish a domain of skin other than the human face. may contain logic. A body part that is similar in color to a person's face but distinguished may be a person's neck, a person's clavicle, a person's arm, a person's leg, a person's stomach, and the like.

The first region setting unit 153 extracts a first region including a human face from the image in the same manner as above, transfers the extracted first region to the second region setting unit 155, and extracts the first region. The number of is the same as the number of frames constituting the image.

The second area setting unit 155 converts the first area extracted by the first area setting unit 153 into a mesh structure and sets a second area obtained by removing a part of the converted mesh structure. . Here, the mesh structure means a structure in which polygons constituting basic units form a mesh, and the mesh structure in the present invention may be composed of one type of polygon. For example, the second area setting unit 155 may convert the first area into a mesh shape composed of a plurality of triangles or a mesh shape composed of a plurality of hexagons. In the present invention, the reason why the second area setter 155 converts the first area into a mesh structure is to accurately specify a part of a person's face in the image, particularly a part where the heart rate can be well detected.

Since the 2nd domain is set by removing some polygons from the first domain converted to the mesh structure, it includes less pixel information than the first domain. In other words, the second region may be understood as a face part necessary for calculating a heart rate with a remote photoplethysmography signal, among the human face.

As an embodiment, the second area setting unit 155 may convert the first area into a mesh structure using triangles as a basic unit, and may set the second area by removing some triangles from the converted mesh structure. As another embodiment, the second region setting unit 155 may convert the first region into a mesh structure having rectangles as a basic unit, and may set the second region by removing some rectangles from the converted mesh structure. In addition to the above-described embodiment, if each number of polygons included in the mesh structure is unified into one, the method according to the present invention can accurately calculate the heart rate of a person in an image. A detailed operation of the second area setting unit 155 will be described later with reference to FIGS. 3 to 5 .

The heart rate calculation unit 157 determines the heart rate of a person included in the image based on the processing result of the remote PPG signal for the second area set by the second area setting unit 155. yields More specifically, the heart rate calculation unit 157 calculates a remote photoplethysmogram (PDP) signal for each pixel included in the second region, and processes the calculated remote photoplethysmogram (PDP) signal, which is an index related to respiration. HRV (Heart Rate Variability), Oxygen Saturation (SpO2), HR (Heart Rate), etc. can be calculated indirectly. In this process, the remote photoplethysmogram signal of the second region calculated for each pixel may be converted into the frequency domain rather than the time domain and processed in order to improve the accuracy and speed of the analysis. , which will be described later.

In the present invention, the method of calculating the remote optical pulse wave signal for the second region follows the conventional method. As an example, the technology disclosed in Korean Patent Registration No. 10-2225557, remote optical pulse wave signal in the face image. A method of obtaining a heart rate by extracting a pulse wave signal may be applied to the present invention, but is not limited thereto.

3 is a diagram for explaining an example of a process of setting a first area by a first area setting unit.

Referring to FIG. 3 , it can be seen that the first area setting unit 153 has created a detection box in order to set and extract the first area. In FIG. 3, the shape of the detection box is square. Not limited. For example, a rectangular detection box with a longer length may be created for a slim face, and a rectangular detection box with a longer width may be created for a round face. The first area setting unit 153 may determine the number and distribution of pixels containing information on human facial skin, and adjust the shape and size of the detection box in consideration of the identified number and distribution of pixels.

The first area setting unit 153 detects information included in one frame based on internally stored information about face color, information about face contour, and information about inflection points of each part of the face, while detecting information included in the face. It is possible to determine the coordinates of landmarks constituting . As an example, the number of landmarks may be 68, but is not limited thereto.

4 is a diagram for explaining an example of a process of converting a first region into a mesh structure by a second region setting unit.

Referring to FIG. 4 , when the first area is set and extracted as shown in FIG. 3 , the second area setter 155 converts the first area into triangles as a basic unit based on feature points formed in the process of extracting the first area. It can be seen that it is converted to a mesh structure that Since the feature points formed by the first region setting unit 153 are located only in the portion corresponding to the human facial skin in the first region, the mesh structure formed by connecting the feature points is generally applied to the human facial skin except for the eyeball. is formed As an example, as shown in FIG. 4 , if the basic unit of the mesh structure is determined as a triangle, all feature points may be connected as triangles through a Delaunay triangulation technique. When the first area is converted into a mesh structure through the Delaunay triangulation technique, each triangle constituting the mesh structure becomes the closest to an equilateral triangle, so as described later, a portion of the polygons forming the mesh structure is removed from the entire mesh structure When this is done, the remaining facial area has a certain regularity, so it may be easy to calculate an accurate heart rate. This will be described later with reference to FIG. 5 .

In FIG. 4, for convenience of explanation, a triangle is described as a mesh structure that is a basic unit, but it can be similarly applied to a mesh structure of a polygon other than a triangle, and instead of a Delaunay triangulation technique, a dual relationship with the Delaunay triangulation technique A Voronoi Diagram algorithm may be applied.

5 is a diagram schematically illustrating an example of a process of setting a second area by a second area setting unit.

FIG. 5 is a diagram for explaining a process of setting a second area by removing some polygons forming the mesh structure from the mesh structure of the first area generated in FIG. 4 . The second area setting unit 155 may measure mathematical indices of triangles constituting the mesh structure, respectively, and reduce the mesh structure based on the measured mathematical indices of polygons constituting the mesh structure.

As an example, the second area setter 155 may remove triangles having a side length less than the first reference value from among the triangles forming the mesh structure. Through the partial removal process as described above, the second area setting unit 155 can remove triangles for human eyes and lips from the entire mesh structure.

As another example, the second area setting unit 155 may remove triangles having an area less than the second reference value from among triangles forming the mesh structure. Through the partial removal process as described above, the second region setting unit 155 can leave only the most necessary part (arterial blood) for calculating the heart rate in the entire face.

The process of removing some polygons based on the first reference value and the second reference value, which is executed by the second area setting unit 155, may be performed sequentially or simultaneously, and the face angle of the person in the image exceeds the preset reference angle. In one case, if the length of a person's face exceeds a predetermined length, either process may be omitted or a weight may be applied and amplified.

As an optional embodiment, the second region setting unit 155 may identify and remove polygon numbers only when they are removed based on the basic unit forming the mesh structure, the estimated angle estimated by the curvature of a person's face in the image. For example, if the basic unit forming the mesh structure is a triangle and the estimated angle estimated by the curvature of a person's face in the image is 20 degrees to 30 degrees, the second area setting unit 155 determines from the upper left corner to the lower right corner Of the plurality of labeled polygons, only polygons numbered 16 to 30 and polygons 45 to 51 can be removed. In this optional embodiment, the data referred to by the second region setting unit 155 is stored in the second region setting unit 155 in the form of a table, and is stored in the database 110 according to the embodiment. It could be. According to this optional embodiment, the second region setting unit 155 can quickly remove from the mesh structure a polygon that calculates an accurate heart rate when it is removed mathematically, empirically, or statistically, and is a mesh parameter. If only the polygon number information to be removed according to the basic unit forming the structure and the estimated angle of the human face is updated, the heart rate of the person in the image can be calculated with high accuracy without improving the entire algorithm of the heart rate calculation system 100. there will be

As a result, as shown in FIG. 5 , the second region setting unit 155 can set a second region in which regions less necessary for calculating the heart rate of a person in the image are removed. The heart rate calculation unit 157 can accurately calculate the heart rate of a person included in the image by selectively processing the rPPG signal only for the second region shown in FIG. 5 .

6 is a flowchart illustrating an example of a method according to the present invention.

Since the method according to FIG. 6 can be implemented by the heart rate calculation system 100 described in FIGS. 1 and 2 or sub-modules included in the heart rate calculation system 100, hereinafter, referring to FIGS. 1 and 2 description, and redundant description will be omitted.

The image acquisition unit 151 acquires an image (S610).

The first region setting unit 153 may set and extract a first region including a human face in the acquired image (S630).

The second area setting unit 155 converts the extracted first area into a mesh structure and sets a second area from which a part of the mesh structure is removed (S650).

The heart rate calculation unit 157 may calculate the heart rate of the person included in the image based on the processing result of the rPPG signal for the second region (S670).

Embodiments according to the present invention described above may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the medium is a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD-ROM and a DVD, a magneto-optical medium such as a floptical disk, and a ROM hardware devices specially configured to store and execute program instructions, such as RAM, flash memory, and the like.

Meanwhile, the computer program may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. An example of a computer program may include not only machine language code generated by a compiler but also high-level language code that can be executed by a computer using an interpreter or the like.

Specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For brevity of the specification, description of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection of lines or connecting members between the components shown in the drawings are examples of functional connections and / or physical or circuit connections, which can be replaced in actual devices or additional various functional connections, physical connection, or circuit connections. In addition, if there is no specific reference such as “essential” or “important”, it may not be a component necessarily required for the application of the present invention.

In the specification of the present invention (especially in the claims), the use of the term “above” and similar indicating terms may correspond to both singular and plural. In addition, when a range is described in the present invention, it includes an invention in which individual values belonging to the range are applied (unless there is a description to the contrary), and each individual value constituting the range is described in the detailed description of the invention Same as Finally, unless an order is explicitly stated or stated to the contrary for the steps constituting the method according to the present invention, the steps may be performed in any suitable order. The present invention is not necessarily limited according to the order of description of the steps. The use of all examples or exemplary terms (eg, etc.) in the present invention is simply to explain the present invention in detail, and the scope of the present invention is limited due to the examples or exemplary terms unless limited by the claims. it is not going to be In addition, those skilled in the art can appreciate that various modifications, combinations and changes can be made according to design conditions and factors within the scope of the appended claims or equivalents thereof.

100: heart rate calculation system
110: database
130: Ministry of Communication
150: processing unit
151: image acquisition unit
153: first area setting unit
155: second area setting unit
157: heart rate calculation unit
170: output unit

Claims (5)

Acquiring an image by an image acquisition unit;
setting and extracting a first region including a human face from the acquired image by a first region setting unit;
converting the extracted first region into a mesh structure by a second region setting unit and setting a second region by removing a part of the converted mesh structure; and
Calculating, by a heart rate calculator, a heart rate of a person included in the image based on a processing result of a remote PPG signal for the set second region;
Setting the second area,
Removing polygons having at least one side length less than a first reference value from among the polygons forming the transformed mesh structure, and removing polygons having a width less than a second reference value from among the polygons, A method for calculating human heart rate using a pulse wave signal. According to claim 1,
The video,
A method for calculating human heart rate using a remote photoplethysmogram signal for an image, which is a continuous image composed of at least two or more frames. According to claim 1,
The step of setting the second area,
converting the extracted first region into a mesh structure in which triangles are basic units; and
and setting the second area by removing some triangles from the converted mesh structure, wherein the heart rate of a person is calculated using a remote photoplethysmogram signal for an image. According to claim 3,
The step of converting to the mesh structure,
stamping a preset number of feature points on the extracted first region; and
A method of calculating a human heart rate using a remote photoplethysmogram signal for an image, comprising performing a Delaunay triangulation based on the feature points. a communication unit that acquires an image from an external device; and
Including a processing unit,
The processing unit,
Setting and extracting a first region including a human face in the obtained image;
Converting the extracted first region into a mesh structure and setting a second region in which a part of the converted mesh structure is removed;
Calculating a heart rate of a person included in the image based on a processing result of a remote PPG signal for the set second region;
When setting the second area from which a part of the transformed mesh structure is removed, polygons having at least one side length less than the first reference value are removed from among the polygons forming the transformed mesh structure, and polygons having a width less than the second reference value among the polygons A system for calculating a person's heart rate using a remote photoplethysmogram signal for an image, which eliminates

Images