KR20240035290A - Method, apparatus and computer program for measuring heart rate using mobile facial video based on deep learning in real time - Google Patents

Abstract

One embodiment of the present invention provides a heart rate measuring method using a heart rate measuring device equipped with a camera, a display unit, and a processor, comprising: obtaining user information; A process of acquiring a user's mobile face image captured by the camera in real time; extracting a first predetermined number of image frames from the mobile face image by the processor; Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; It is possible to provide a heart rate measurement method using a real-time deep learning-based mobile face image, including a process of acquiring the user's heart rate based on the predicted rPPG signal by the processor.

Description

Method, apparatus and computer program for measuring heart rate using mobile facial video based on deep learning in real time}

The present invention relates to a heart rate measurement device and method using real-time deep learning-based mobile face images. In detail, the present invention relates to a real-time deep learning-based mobile face that extracts rPPG signals on the mobile terminal itself using mobile face images captured in real time on a mobile terminal. It relates to a heart rate measurement device and method using images.

There is research to identify human diseases by sensing biological signals with a PPG sensor in a non-invasive manner. However, due to the current COVID-19 pandemic, technology that can remotely monitor health in a non-invasive manner has become very important.

Countries are encouraged to use telehealth strategies where possible to reduce the risk of COVID-19 in healthcare settings. Therefore, a new method is required rather than the existing method of monitoring biometric information using sensors that require contact with the body.

Remote health monitoring technologies are based on communication systems such as mobile phones or online health portals. These remote health monitoring technologies may very well be required for continuous patient monitoring even after a pandemic such as COVID-19 is over.

Conventional remote photoplethysmography (rPPG) estimation methods collect RGB video, separate facial regions of interest (ROIs), and use signal processing approaches to analyze subtle color changes in the face.

Republic of Korea Patent No. 10-2096617 (2020.03.27.)

One embodiment of the present invention is a real-time deep learning-based mobile face image that can measure the user's heart rate using the user's mobile face image captured using a camera installed in a mobile terminal such as a smartphone without an additional sensor for heart rate measurement. A heart rate measuring device and method using can be provided.

One embodiment of the present invention can provide a heart rate measurement device and method using a real-time deep learning-based mobile facial image that can measure heart rate non-contactly, such as in newborns, driving, or exercise situations.

One embodiment of the present invention provides a heart rate measurement device and method using a real-time deep learning-based mobile facial image that can measure the user's heart rate in real time based on an AI model mounted inside a mobile device such as a smartphone. You can.

One embodiment of the present invention provides a heart rate measurement device and method using a real-time deep learning-based mobile face image that measures the user's heart rate based on an AI model without separating the region of interest (ROI) from the mobile face image. can be provided.

One embodiment of the present invention uses a real-time deep learning-based mobile facial image that can measure the user's heart rate in real time without network delay or calculation delay of an external server because it uses its own processor without a connection to an external server. Measuring devices and methods can be provided.

One embodiment of the present invention can provide a heart rate measurement device and method using a real-time deep learning-based mobile facial image that can measure the user's heart rate based on remote photoplethysmography (rPPG).

One embodiment of the present invention provides a heart rate measuring method using a heart rate measuring device equipped with a camera, a display unit, and a processor, comprising: obtaining user information; A process of acquiring a user's mobile face image captured by the camera in real time; extracting a first predetermined number of image frames from the mobile face image by the processor; Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; It is possible to provide a heart rate measurement method using a real-time deep learning-based mobile face image, including a process of acquiring the user's heart rate based on the predicted rPPG signal by the processor.

In one embodiment of the present invention, the process of acquiring the user's mobile face image captured by the camera in real time includes: When the user's face image captured in real time by the camera and the face guide area match, the first predetermined A heart rate measurement method using real-time deep learning-based mobile face images can be provided, which includes the process of acquiring a plurality of mobile face images by photographing the user at time intervals.

In one embodiment of the present invention, the process of predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames includes applying the predictor model to each of the plurality of mobile face images. A heart rate measurement method using a real-time deep learning-based mobile face image can be provided, including the process of acquiring a plurality of predicted rPPG signals.

In one embodiment of the present invention, the process of acquiring the user's heart rate based on the predicted rPPG signal by the processor includes calculating the user's heart rate value based on the plurality of predicted rPPG signals, and calculating a second predetermined value. A heart rate measurement method using a real-time deep learning-based mobile face image can be provided, which includes a process of obtaining an average heart rate value by averaging a number of heart rate values.

In one embodiment of the present invention, the predictor model is composed of a predictor model learned with other learning data based on the user information, or the heart rate is a heart rate value to which the weight of the user information is applied. A heart rate measurement method using a mobile face image can be provided.

In one embodiment of the present invention, the predictor model is an artificial intelligence model learned using the subject's face image and the subject's actual heart rate signal measured simultaneously while shooting the face image as learning data. A heart rate measurement method using a running-based mobile face image can be provided.

One embodiment of the present invention is a real-time deep learning-based mobile face image, characterized in that the loss function of the predictor model is learned in a way to minimize the difference between the heart rate signal estimated from the subject's face image and the actual heart rate signal. A heart rate measurement method using can be provided.

One embodiment of the present invention provides a heart rate measuring device equipped with a camera, a display unit, and a processor, wherein the processor includes: acquiring user information; A process of acquiring a user's mobile face image captured by the camera in real time; extracting a first predetermined number of image frames from the mobile face image by the processor; Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; And a heart rate measurement device using a real-time deep learning-based mobile face image can be provided, in which the processor performs a process of acquiring the user's heart rate based on the predicted rPPG signal.

In one embodiment of the present invention, the process of acquiring the user's mobile face image captured by the camera in real time includes: When the user's face image captured in real time by the camera and the face guide area match, the first predetermined A heart rate measuring device using real-time deep learning-based mobile face images can be provided, including a process of acquiring a plurality of mobile face images by photographing the user at time intervals.

In one embodiment of the present invention, the process of predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames includes applying the predictor model to each of the plurality of mobile face images. A heart rate measurement device using a real-time deep learning-based mobile face image can be provided, including a process of acquiring a plurality of predicted rPPG signals.

In one embodiment of the present invention, the process of acquiring the user's heart rate based on the predicted rPPG signal by the processor includes calculating the user's heart rate value based on the plurality of predicted rPPG signals, and calculating a second predetermined value. A heart rate measurement device using a real-time deep learning-based mobile face image can be provided, including a process of obtaining an average heart rate value by averaging a number of heart rate values.

In one embodiment of the present invention, the predictor model is composed of a predictor model learned with other learning data based on the user information, or the heart rate is a heart rate value to which the weight of the user information is applied. A heart rate measurement device using a based mobile face image can be provided.

In one embodiment of the present invention, the predictor model is an artificial intelligence model learned using the subject's face image and the subject's actual heart rate signal measured simultaneously while shooting the face image as learning data. A heart rate measurement device using a running-based mobile face image can be provided.

One embodiment of the present invention is a real-time deep learning-based mobile face image, characterized in that the loss function of the predictor model is learned in a way to minimize the difference between the heart rate signal estimated from the subject's face image and the actual heart rate signal. A heart rate measuring device using can be provided.

One embodiment of the present invention is combined with a computing device, a process of obtaining user information; A process of acquiring a user's mobile face image captured by a camera in real time; Extracting a first predetermined number of image frames from the mobile face image by a processor; Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; and a heart rate measurement computer program using a real-time deep learning-based mobile facial image stored in a computer-readable recording medium to execute a process of acquiring the user's heart rate based on the predicted rPPG signal by the processor. there is.

One embodiment of the present invention is a real-time deep learning-based mobile face image that can measure the user's heart rate using the user's mobile face image captured using a camera installed in a mobile terminal such as a smartphone without an additional sensor for heart rate measurement. It has the effect of providing a heart rate measuring device and method using.

One embodiment of the present invention has the effect of providing a heart rate measurement device and method using a real-time deep learning-based mobile facial image that can measure heart rate non-contactly, such as in newborns, driving, or exercise situations.

One embodiment of the present invention provides a heart rate measurement device and method using a real-time deep learning-based mobile facial image that can measure the user's heart rate in real time based on an AI model mounted inside a mobile device such as a smartphone. It has an effect.

One embodiment of the present invention provides a heart rate measurement device and method using a real-time deep learning-based mobile face image that measures the user's heart rate based on an AI model without separating the region of interest (ROI) from the mobile face image. It has the effect of providing

One embodiment of the present invention uses a real-time deep learning-based mobile facial image that can measure the user's heart rate in real time without network delay or calculation delay of an external server because it uses its own processor without a connection to an external server. It has the effect of providing measuring devices and methods.

One embodiment of the present invention has the effect of providing a heart rate measurement device and method using a real-time deep learning-based mobile facial image that can measure the user's heart rate based on remote photoplethysmography (rPPG).

Figure 1 is a hardware configuration diagram of a heart rate measurement device using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 2 is a flowchart of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 3 shows an example screen of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 4 shows a flowchart of the inference process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 5 is a conceptual diagram of the inference process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 6 shows a conceptual diagram of the inference process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 7 is a flowchart of the learning process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 8 shows a conceptual diagram of the learning process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.
Figure 9 shows a graph according to the learning process of the loss function in the learning process of the heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the concept of the present invention disclosed in this specification are merely illustrative for the purpose of explaining the embodiments according to the concept of the present invention, and the embodiments according to the concept of the present invention are It may be implemented in various forms and is not limited to the embodiments described herein.

Since the embodiments according to the concept of the present invention can make various changes and have various forms, the embodiments will be illustrated in the drawings and described in detail in this specification. However, this is not intended to limit the embodiments according to the concept of the present invention to specific disclosed forms, and includes all changes, equivalents, or substitutes included in the spirit and technical scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The above terms are used only for the purpose of distinguishing one component from another component, for example, without departing from the scope of rights according to the concept of the present invention, a first component may be named a second component and similarly a second component A component may also be named a first component.

When a component is said to be “connected” or “connected” to another component, it is understood that it may be directly connected to or connected to that other component, but that other components may also exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between. Other expressions that describe the relationship between components, such as "between" and "immediately between" or "neighboring" and "directly adjacent to" should be interpreted similarly.

Technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this specification, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in this specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, are generally understood by a person of ordinary skill in the field to which the technology disclosed in this specification belongs, unless specifically defined in a different way in this specification. It has the same meaning as understood as. Terms as defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings they have in the context of the related technology, and unless clearly defined in this specification, should not be interpreted in an idealized or overly formal sense. It shouldn't be.

The suffixes “module” and “part” for components used in this specification are given or used interchangeably only considering the ease of preparing the specification, and do not have distinct meanings or roles in themselves, and do not have distinct meanings or roles in this specification. It may mean a functional or structural combination of hardware for performing a method according to an embodiment of the invention or software that can drive the hardware.

Below, with reference to the attached drawings, a heart rate measuring device and method using a real-time deep learning-based mobile face image according to an embodiment of the present invention will be described.

Figure 1 is a hardware configuration diagram of a heart rate measurement device using a real-time deep learning-based mobile face image according to an embodiment of the present invention.

Referring to FIG. 1, a heart rate measurement device 100 (hereinafter referred to as 'computing device') using a real-time deep learning-based mobile face image according to an embodiment of the present invention will be described.

The computing device 100 can measure the user's heart rate using the user's mobile face image without a sensor physically contacting the user. This is called remote heart rate measurement technology or remote photoplethysmography technology based on vision computing.

Remote photoplethysmography (rPPG) is a technology that extracts heart rate signals by capturing subtle changes in skin color captured from blood flow changes due to the cardiac cycle.

Heart rate is the number of heart beats per unit time and reflects a person's physical and mental state under the influence of the autonomic nervous system, which consists of the sympathetic and parasympathetic nervous systems.

Typically, this heart rate can be monitored using an electrocardiogram (ECG), which records the heart's electrical activity, or a photoplethysmogram (PPG), which uses an optical sensor. However, these methods generally obtain signals by attaching a sensor to the skin surface, so they have limitations in that physical contact with the target is essential or movement is restricted.

To solve this problem, the computing device 100 according to an embodiment of the present invention proposes a method of measuring the user's rPPG signal and heart rate using a mobile face image in real time based on deep learning.

Referring to FIG. 1, in one embodiment of the present invention, the computing device 100 includes one or more processors 110, a memory 120 that loads a computer program 141 executed by the processor 110, It may include a storage 140 that stores a communication interface 130 and a computer program 141.

In various embodiments, the computing device 100 refers to all types of hardware devices including at least one processor 110, and depending on the embodiment, it is understood to also encompass software configurations that operate on the hardware device. It can be.

The computing device 100 may be understood as including, but is not limited to, a smartphone, tablet PC, desktop, laptop, and user clients and applications running on each device.

Each process described in this specification may be performed by the computing device 100. Additionally, the subject of each process is not limited to being performed by one computing device 100, and depending on the embodiment, at least part of each process may be performed by different computing devices.

The processor 110 controls the overall operation of each component of the computing device 100. The processor 110 includes a Central Processing Unit (CPU), Micro Processor Unit (MPU), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), or any other type of processor well known in the art of the present invention. It can be.

Additionally, the processor 110 may perform operations on at least one application or program for executing methods according to embodiments of the present invention, and the computing device 100 may include one or more processors.

In addition, the processor 110 includes random access memory (RAM) (not shown) and read-only memory (ROM) that temporarily and/or permanently store signals (or data) processed within the processor 110. , not shown) may be further included. Additionally, the processor 110 may be implemented in the form of a system on chip (SoC) that includes at least one of a graphics processing unit, RAM, and ROM.

The processor 110 may perform methods/processes according to various embodiments of the present invention. As an example, the processor 110 may perform a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention and a combination of at least one process forming the method.

Memory 120 stores various data, commands and/or information. Memory 120 may load a computer program 141 from storage 140 to execute methods/operations according to various embodiments of the present invention. When the computer program 141 is loaded into the memory 120, the processor 110 may perform the method/operation by executing one or more instructions constituting the computer program 141. The memory 120 may be implemented as a volatile memory such as RAM, but the technical scope of the present disclosure is not limited thereto.

The bus (BUS) provides communication functions between components of the computing device 100. A BUS can be implemented as various types of buses, such as an address bus, data bus, and control bus.

The communication interface 130 supports wired and wireless Internet communication of the computing device 100. Additionally, the communication interface 130 may support various communication methods other than Internet communication. For example, the communication interface 130 may support at least one of short-range communication, mobile communication, and broadcast communication methods. To this end, the communication interface 130 may be configured to include a communication module well known in the technical field of the present invention. In some embodiments, communication interface 130 may be omitted.

The communication interface 130 may be connected to the heart rate monitor 200 through wired or wireless communication. The communication interface 130 can communicate with the heart rate monitor 200 in real time and receive the cellular heart rate signal measured by the heart rate monitor 200 in real time.

Storage 140 may store the computer program 141 non-temporarily. The storage 140 is a non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or a device well known in the art to which the present invention pertains. It may be configured to include any known type of computer-readable recording medium.

The computer program 141, when loaded into the memory 120, may include one or more instructions that cause the processor 110 to perform methods/processes according to various embodiments of the present invention. That is, the processor 110 can perform methods/processes according to various embodiments of the present invention by executing the one or more instructions.

The method or algorithmic processes described in relation to embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

In one embodiment, the computer program 141 is combined with the computing device 100 and may be a computer program stored in a computer-readable recording medium to execute a heart rate measurement method using a real-time deep learning-based mobile facial image. .

The components of the present invention may be implemented as a program (or application) and stored in a medium in order to be executed in conjunction with a hardware computer. Components of the invention may be implemented as software programming or software elements, and similarly, embodiments may include various algorithms implemented as combinations of data structures, processes, routines or other programming constructs, such as C, C++, , may be implemented in a programming or scripting language such as Java, assembler, etc. Functional aspects may be implemented as algorithms running on one or more processors.

Furthermore, the computing device 100 may further include at least one of an output unit 150, an A/V input unit 170, and a user input unit 180.

The output unit 150 may output at least one of a video signal and an audio signal generated or executed by an executing program (or application). The output unit 150 may include at least one of a display unit 151 that outputs a video (image) signal and an audio output unit 152 that outputs an audio signal.

The display unit 151 may be provided on the front of the computing device 100.

The A/V (Audio/Video) input unit 170 may include a camera 171 for inputting a color (RGB) video signal from the outside to the computing device 100.

The camera 171 may be provided on the front or back of the computing device 100. The camera 171 may be an RGB camera that captures RGB images, and can obtain an image frame composed of RGB images, such as a still image or video, through an image sensor in shooting mode. The RGB camera 171 can capture a user's mobile face image by photographing the user.

The user's mobile face image captured by the camera 171 may be composed of a plurality of image frames for the user and may include information on the shooting point.

The user input unit 180 refers to a means for a user to input a user input signal or data to control the computing device 100. The user input unit 180 includes a touch pad (for example, a contact capacitive type, a pressure resistance film type, an infrared detection type, a surface ultrasonic conduction type, an integral tension measurement type, a piezo effect type, etc.), and a key pad (key pad), power button, volume button, etc., but is not limited thereto.

The computing device 100 according to an embodiment of the present invention may perform a heart rate measurement method using a real-time deep learning-based mobile face image, which will be described later, and a combination of at least one process constituting the method.

The heart rate monitor 200 may communicate with the computing device 100 wired or wirelessly to transmit the user's heart rate information. The heart rate monitor 200 can measure the user's heart rate in real time, which is temporally paired with the user's mobile face image captured in real time by the camera 171 of the computing device 100. The user's heart rate information transmitted from the heart rate monitor 200 to the computing device 100 may include a heart rate signal and information on a time when the heart rate signal was measured.

Figure 2 is a flowchart of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention. Figure 3 shows an example screen of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention. Figure 4 shows a flowchart of the inference process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention. Figure 5 shows a conceptual diagram of the inference process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention. Figure 6 shows a conceptual diagram of the inference process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.

2 to 6, a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention will be described.

In the S100 process, the computing device 100 may obtain user information.

Referring to FIG. 3(a), the computing device 100 may provide a user information query screen SC1 to the user through the display unit 151.

The user information query screen (SC1) provides an information input space where at least one of the user's name, age, gender, and current status information can be entered.

The computing device 100 may obtain at least one of the user's name, age, gender, and current status information entered by the user through the user information query screen and store it in the storage 140.

The user's name can be used as the main identification information to distinguish the user. Current status information may include information about whether the device is in a rest state or an active state.

In process S200, the computing device 100 may acquire the user's mobile face image.

The user's mobile face image is a collection of a plurality of image frames captured by an image sensor that senses light in the visible light band, such as color video or RGB video captured by the camera 171 provided in the computing device 100. You can.

Referring to FIG. 3(b), the computing device 100 may provide a face image capture screen SC2 to the user through the display unit 151.

The display unit 151 is provided on the front of the computing device 100, and a camera 171 may be provided on the same front.

The face image capture screen (SC2) is located inside the real-time image display area (SC2-A1) where the real-time image captured using the camera 171 is displayed, and the real-time image display area (SC2-A1) to align the user's face. It may include a provided face guide area (SC2-A2).

The real-time image display area (SC2-A1) is located on the display unit 151 and can output real-time image captured through the camera 171.

The face guide area (SC2-A2) is located inside the real-time video display area (SC2-A1) and provides a guide to adjust the user's face position. The face guide area (SC2-A2) may be composed of a vertically long oval or the shape of a human face. The user can adjust the distance between the camera 171 and his or her face so that the size of the face image captured in real time through the camera 171 matches the size of the face guide area (SC2-A2).

The face image capturing screen (SC2) may include a progress display area (SC2-A3) that shows the progress of predicting the user's heart rate.

In various embodiments, the progress display area SC2-A3 is configured longitudinally as shown in FIG. 3(b) and may be displayed as a plurality of heart shapes that are empty or filled with a specific color. Alternatively, the progress display area (SC2-A3) can be displayed in one of the following shapes: an hourglass shape, an increase or decrease in numbers expressed as percentages, or a bar shape that is longitudinally empty or filled with a specific color. Alternatively, the progress display area (SC2-A3) provides letters/sentences/figures to the effect of adjusting the user's face position to match the face guide area (SC2-A2), or predicts heart rate for the user's face image. You can provide letters/sentences/figures that indicate the level of progress.

In various embodiments, the progress display area (SC2-A3) monitors the heart rate using the user's face image and size matching information regarding whether the size of the user's face image matches the size of the face guide area (SC2-A2). It may include predicted progress information.

For example, the size matching information blinks an empty heart shape, shows an hourglass shape, or shows a 0% number when the size of the user's face image and the size of the face guide area (SC2-A2) do not match. It may show an empty bar, show the text "Adjust face position", or contain arrows to adjust the face position.

In addition, the progress information may display the degree of the heart rate measurement process by the predictor model (PM) or processor of the computing device 100, with some hearts among a plurality of heart shapes filled with a specific color, or showing the degree of progress. It can display a percentage number, a bar with a portion filled with a specific color, or the text "Measuring heart rate."

In process S200, if the area where the user's real-time face image shown in the real-time image display area (SC2-A1) and the face guide area (SC2-A2) intersect do not match, the computing device 100 determines the user's Size matching information can be provided through the progress display area (SC2-A3) so that the size and position of the face image can be adjusted. In addition, when the area where the user's face image and the face guide area (SC2-A2) intersect coincide, the computing device 100 displays information on the progress of predicting the heart rate for the user's face image in the progress display area ( It can be provided through SC2-A3).

In the S200 process, the computing device 100 is a process of checking whether the user's face image and the face guide area (SC2-A2) match. (1) Captured in real time through the camera 171 and displayed in the real-time image display area (SC2-A2). -A process of determining that the area where the user's face image displayed in A1) and the face guide area (SC2-A2) intersect does not match if it is less than a certain area; (2) the user's face image is judged to be inconsistent with the face guide area (SC2-A2); Process of determining that it does not match if it deviates from A2); (3) Even if the user's face image is located inside the face guide area (SC2-A2), it is judged not to match if the area of the user's face image is less than a certain area Process, (4) At least two feature points among both eyes, both eyebrows, nose, mouth, and both ears of the user's face image are not located inside the face guide area (SC2-A2) or are between at least two feature points. If the distance is less than a predetermined distance, the process of determining that it does not match can be performed by any one or a combination of two or more.

The face image capture screen (SC2) may include a heart rate display area (SC2-A4) that displays the user's heart rate information.

The heart rate display area (SC2-A4) may include characters indicating that the heart rate prediction process is in progress, such as “Measuring heart rate” if the heart rate is being predicted. Additionally, the heart rate display area (SC2-A4) may include letters and numbers indicating the predicted heart rate, such as “average bpm: 88” if the heart rate prediction has been completed.

In various embodiments, the computing device 100 starts capturing the user's face image through the camera 171 when the user's face image and the face guide area (SC2-A2) match each other, thereby capturing the user's mobile face image. can be obtained. The mobile face image is captured through an image sensor included in the camera 171 of the computing device 100 and stored in the storage 140 as a video file.

The computing device 100 may capture a user's mobile face image for a first predetermined period of time after the capture begins and store the user's mobile face image. In other words, the computing device 100 may create and store a video file having a length of a first predetermined time after the start of filming. Alternatively, the computing device 100 may generate and store a facial image obtained by photographing the user's mobile face at first predetermined time intervals. Here, the first predetermined time can be set between 3 and 7 seconds, and can preferably be set at 5 seconds.

The computing device 100 may acquire a plurality of mobile face images by capturing and storing a plurality of mobile face images at first predetermined time intervals. Here, the plurality of mobile face images include a first mobile face image captured between 0 and 5 seconds (first cycle) after shooting starts, a second mobile face image captured between 5 and 10 seconds (second cycle), It may include a third mobile face image captured between 10 and 15 seconds (third cycle), a fourth mobile face image captured between 15 and 20 seconds (fourth cycle), etc. In other words, the computing device 100 may capture and store a plurality of mobile face images for each cycle after the capture begins.

In the S300 process, the computing device 100 may predict heart rate information using a predictor model (PM). The computing device 100 may measure heart rate information based on the user's mobile face image using a predictor model (PM).

The computing device 100 may predict the user's heart rate value (rPPG signal) based on a predictor model (PM) using at least one of the mobile face images.

As an example, the predictor model (PM) may be a model learned from facial images and heart rate data captured regardless of the user's age, gender, or current condition. In this case, the computing device 100 may predict the user's heart rate value (rPPG signal) by applying a predictor model (PM) to the captured mobile face image of the user regardless of user information.

In this case, the computing device 100 may calculate the weighted heart rate value (wb) by applying a weight according to user information to the predicted user's heart rate value (pb). That is, the computing device 100 calculates the heart rate value by applying the age weight (aw) corresponding to the user's age range, the gender weight (gw) corresponding to the user's gender, and the state weight (sw) corresponding to the user's current state. You can. For example, the average value of the predicted heart rate value multiplied by the age, gender, and status weights can be used as the weighted heart rate value (wb = pb x (aw+gw+sw)/3).

In various embodiments, there may be a plurality of predictor models (PM) depending on the age of the user. For example, there may be a 20s predictor model (PM), a 30s predictor model (PM), a 40s predictor model (PM), etc. The age-specific predictor model (PM) may be a model learned using mobile face images and heart rate data of users corresponding to that age group.

In various embodiments, there may be a plurality of predictor models (PM) depending on the gender of the user. For example, there may be a predictor model (PM) for men and a predictor model (PM) for women. The gender predictor model (PM) may be a model learned using the mobile face image and heart rate data of the user corresponding to the gender.

In various embodiments, there may be a plurality of predictor models (PM) depending on the current state of the user. For example, there may be a predictor model (PM) for activity and a predictor model (PM) for rest. The state-specific predictor model (PM) may be a model learned using the user's mobile face image and heart rate data corresponding to the state.

Here, the computing device 100 may use different predictor models (PMs) based on user information. In other words, the computing device 100 calculates the user's heart rate value (rPPG signal) using one of the age predictor model (PM), gender predictor model (PM), and state predictor model (PM) based on user information. You can predict or use all models in parallel to predict the average or median value of the output values as the user's heart rate value (rPPG signal).

The computing device 100 may obtain an average heart rate value by calculating an average value of at least a second predetermined number of heart rate values and provide the average heart rate value to the user. The second predetermined number may be 2 to 4 times, and preferably 3 times. In this case, the average heart rate value may be the average of three continuously predicted heart rate values.

The predictor model (PM) can receive the user's mobile face image as input and output heart rate information. Details about this will be described later.

In process S400, the computing device 100 may display heart rate information on the display unit 151.

Referring to FIG. 3(c), in various embodiments, the computing device 100 displays the predicted heart rate in the heart rate display area (SC2-A4) on the face image capture screen (SC2) to the user through the display unit 151. Values can be displayed. The predicted heart rate value may be an average value of heart rate values predicted continuously a predetermined number of times.

Referring to FIG. 3(d), in various embodiments, the computing device 100 may provide a heart rate tracking management screen SC3 that provides statistical information of the predicted heart rate through the display unit 151.

The heart rate tracking management screen (SC3) can display and provide a history of heart rate measured at different times, a history of daily average heart rate, monthly average heart rate history, and graphs thereof.

Referring to FIGS. 4, 5, and 6, a method for predicting heart rate information using a mobile face image will be described in detail.

In process S310, the computing device 100 may preprocess a mobile face image captured through its own camera 171.

The computing device 100 may acquire a mobile face image stored in the storage 140. Here, the mobile face image stored in the storage 140 is the user's mobile face image captured by the camera 171 provided in the computing device 100.

The computing device 100 may extract a plurality of image frames from the user's mobile face image. The computing device 100 may extract a first predetermined number of image frames from a plurality of image frames.

The first predetermined number may be between 90 frames and 160 frames, and may be preferably set to 90 frames. The mobile face image composed of a first predetermined number of image frames may be the same as the mobile face image captured at the first predetermined time.

To increase the accuracy of rPPG signal prediction, more image frames than 160 frames can be used, but the problem of slow processing speed occurs. To solve this problem, the verification loss can be minimized while increasing the computation processing speed by using a first predetermined number of image frames.

The first predetermined number of image frames may be image frames located consecutively to each other in the mobile face image.

In one embodiment of the present invention, by minimizing the size of the processing target, the rPPG signal can be predicted in real time from the mobile computing device 100 itself, such as a smartphone, rather than predicting the rPPG signal by sending the face image to a server or cloud. .

This eliminates network support or data processing time at the server that occurs when transmitting data to the server, and performs calculations to predict the rPPG signal through the own processor of the computing device 100 and stores the heart rate information in its own database. Real-time processing is possible and the problem of external transmission of sensitive data can be solved.

The computing device 100 may convert a plurality of image frames into a plurality of bitmap images. Thereafter, the computing device 100 may change the plurality of bitmap images into a face image tensor.

Here, the face image tensor may be a four-dimensional tensor containing the number of image frames (time), H is the vertical size of the image frame, W is the horizontal size of the image frame, and C is the number of channels in the video format.

Additionally, the face image tensor can be a 5-dimensional tensor transformed from a 4-dimensional tensor using torch.jit.script with torch.stack.

In process S320, the computing device 100 can predict the rPPG signal using an estimator.

The predictor model (PM) can predict the rPPG signal using the user's mobile face image. In detail, the predictor model (PM) can predict the rPPG signal using multiple image frames of the mobile face image. Alternatively, the predictor model (PM) can predict the rPPG signal using the user's face image tensor.

The predictor model (PM) may receive at least one of the user's mobile face image, a plurality of image frames of the face image, and the user's face tensor and output the user's rPPG signal.

Referring to FIG. 6, the predictor model (PM) may be composed of a 3D convolutional neural network that extracts 3D spatiotemporal features from volume-shaped data. The predictor model (PM) extracts rPPG features from a face image composed of a plurality of image frames having a volume shape, and can extract the rPPG signal based on the rPPG features.

rPPG features may include characteristic information of multiple image frames rather than one image frame. Since a plurality of consecutive image frames that constitute the mobile face image are used, the rPPG features include temporal information of the mobile face image.

The rPPG signal must utilize time-series video sequences from face images to accurately measure the user's current heart rate. The 3D convolutional neural network obtains the current output value using a plurality of consecutive image frames in time series, so it is suitable for measuring an accurate rPPG signal from a face image.

In process S330, the computing device 100 may obtain heart rate information using the rPPG signal.

The computing device 100 may obtain heart rate information including heart rate (bpm) from the rPPG signal. The computing device 100 may detect the peak value of the rPPG signal and calculate the user's heart rate based on the peak value.

Heart rate information may include the user's real-time heart rate calculated in real time and information at the time when the real-time heart rate was calculated. Heart rate information may include real-time heart rate and the user's past heart rate calculated at a past time.

Figure 7 is a flowchart of the learning process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention. Figure 8 shows a conceptual diagram of the learning process of a heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention. Figure 9 shows a graph according to the learning process of the loss function in the learning process of the heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention.

7 to 9, the learning process of the loss function in the learning process of the heart rate measurement method using a real-time deep learning-based mobile face image according to an embodiment of the present invention will be described.

In process S500, the computing device 100 may acquire the subject's face image and the subject's heart rate data.

The subject's face image may include the subject's identification information, including subject person information, scenario information, and source name information, and the identification information may be expressed as 'P1/V1/Source1'.

The heart rate data may include the subject's actual heart rate signal measured in real time when capturing the subject's face image and the subject's identification information. The actual heart rate signal measured in real time may include BVP (Blood Volume Pulse) signals corresponding to each image frame of the face image.

In process S600, the computing device 100 may generate learning data using facial images and heart rate data. Additionally, the computing device 100 may generate test data using facial images and heart rate data.

The computing device 100 may extract a plurality of image frames from a face image. Additionally, the computing device 100 may extract the heart rate signal corresponding to each image frame by synchronizing the heart rate signal corresponding to each of the plurality of image frames.

Additionally, the computing device 100 may convert and extract the heart rate signal into a PPG signal.

The computing device 100 may generate a plurality of image frames and a heart rate signal corresponding to each image frame as learning data or test data. Alternatively, the computing device 100 may generate a plurality of image frames and a PPG signal corresponding to each image frame as learning data or test data.

In the S700 process, the computing device 100 may learn a predictor model (PM) using training data. Additionally, the computing device 100 may evaluate a learned predictor model (PM) using test data.

The computing device 100 may train a predictor model (PM) using facial images and heart rate signals as training data. The predictor model (PM) can learn a loss function to minimize the difference between the predicted heart rate signal for the face image and the actual heart rate signal for the face image.

Additionally, the computing device 100 may learn a predictor model (PM) using facial images and actual PPG signals as training data. The predictor model (PM) can learn a loss function to minimize the difference between the predicted rPPG signal for a face image and the actual PPG signal corresponding to the face image.

Here, the loss function is used to minimize the difference between the heart rate signal (or estimated rPPG signal) estimated from the face image and the actual heart rate signal (or actual PPG signal). The actual heart rate signal can use the BVP signal.

For example, the loss function (Loss) is set to a negative Pearson correlation coefficient because the estimated rPPG signal and the actual PPG signal must be learned in a direction that shows a positive linear relationship close to +1. Through this, the loss is learned so that it approaches 0 during the learning process (see Figure 9).

Here, in the loss function (Loss), T is the total signal length (frame length), x is the estimated rPPG signal predicted by the predictor model (PM), and y is the actual PPG signal (or actual BVP signal).

The above-described learning process (process S500 to S700) may be performed together in the computing device 100 where the inference process (process S100 to S400) is performed, or may be performed in another computing device (not shown).

In the case where the electronic learning process and reasoning process are performed in one computing device 100, the computing device 100 uses the user's mobile face image captured by its own camera 171 and a heart rate monitor (200) connected wired or wirelessly. After performing a learning process of learning a predictor model (PM) using the heart rate signal measured in real time while shooting a face image as training data, the user's mobile face image is captured in real time using the learned predictor model (PM) From this, the user's rPPG signal can be predicted, heart rate calculated, and provided to the user.

In the case where the latter learning process and inference process are performed in another computing device, the computing device 100 receives and stores the learned predictor model (PM) from the other computing device (not shown) and then stores the learned predictor model (PM). ) can be used to predict the user's rPPG signal from the user's mobile face image captured in real time, calculate the heart rate, and provide it to the user.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached registration claims.

Computing devices: 100
Heart rate monitor: 200

Claims (15)

In the heart rate measurement method using a heart rate measuring device equipped with a camera, a display unit, and a processor,
Process of obtaining user information;
A process of acquiring a user's mobile face image captured by the camera in real time;
extracting a first predetermined number of image frames from the mobile face image by the processor;
Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; and
Including the process of acquiring the user's heart rate based on the predicted rPPG signal by the processor,
Heart rate measurement method using real-time deep learning-based mobile face images. According to paragraph 1,
The process of acquiring the user's mobile face image captured by the camera in real time,
When the user's face image captured in real time by the camera matches the face guide area, acquiring a plurality of mobile face images by photographing the user at first predetermined time intervals,
Heart rate measurement method using real-time deep learning-based mobile face images. According to paragraph 2,
The process of predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames includes:
Including the process of obtaining a plurality of predicted rPPG signals by applying a predictor model to each of the plurality of mobile face images,
Heart rate measurement method using real-time deep learning-based mobile face images. According to paragraph 3,
The process of acquiring the user's heart rate based on the predicted rPPG signal by the processor,
Comprising the process of calculating the user's heart rate value based on the plurality of predicted rPPG signals and obtaining an average heart rate value obtained by averaging a second predetermined number of heart rate values,
Heart rate measurement method using real-time deep learning-based mobile face images. According to paragraph 1,
The predictor model is composed of a predictor model learned with different learning data based on the user information, or
Characterized in that the heart rate is a heart rate value to which the weight of the user information is applied.
Heart rate measurement method using real-time deep learning-based mobile face images. According to paragraph 1,
The predictor model is an artificial intelligence model learned using the subject's face image and the subject's actual heart rate signal measured simultaneously while shooting the face image as learning data,
Heart rate measurement method using real-time deep learning-based mobile face images. According to clause 6,
The loss function of the predictor model is characterized in that it is learned in a way that minimizes the difference between the heart rate signal estimated from the subject's face image and the actual heart rate signal,
Heart rate measurement method using real-time deep learning-based mobile face images. In the heart rate measuring device equipped with a camera, a display unit, and a processor,
The processor,
Process of obtaining user information;
A process of acquiring a user's mobile face image captured by the camera in real time;
extracting a first predetermined number of image frames from the mobile face image by the processor;
Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; and
Performing a process of acquiring the user's heart rate based on the predicted rPPG signal by the processor,
Heart rate measurement device using real-time deep learning-based mobile face images. According to clause 8,
The process of acquiring the user's mobile face image captured by the camera in real time,
When the user's face image captured in real time by the camera matches the face guide area, capturing the user at first predetermined time intervals to obtain a plurality of mobile face images,
Heart rate measurement device using real-time deep learning-based mobile face images. According to clause 9,
The process of predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames includes:
Including the process of obtaining a plurality of predicted rPPG signals by applying a predictor model to each of the plurality of mobile face images,
Heart rate measurement device using real-time deep learning-based mobile face images. According to clause 10,
The process of acquiring the user's heart rate based on the predicted rPPG signal by the processor,
Comprising the process of calculating the user's heart rate value based on the plurality of predicted rPPG signals and obtaining an average heart rate value obtained by averaging a second predetermined number of heart rate values,
Heart rate measurement device using real-time deep learning-based mobile face images. According to clause 8,
The predictor model is composed of a predictor model learned with different learning data based on the user information, or
Characterized in that the heart rate is a heart rate value to which the weight of the user information is applied.
Heart rate measurement device using real-time deep learning-based mobile face images. According to clause 8,
The predictor model is an artificial intelligence model learned using the subject's face image and the subject's actual heart rate signal measured simultaneously while shooting the face image as learning data,
Heart rate measurement device using real-time deep learning-based mobile face images. According to clause 13,
The loss function of the predictor model is characterized in that it is learned in a way that minimizes the difference between the heart rate signal estimated from the subject's face image and the actual heart rate signal,
Heart rate measurement device using real-time deep learning-based mobile face images. Combined with a computing device,
Process of obtaining user information;
A process of acquiring a user's mobile face image captured by a camera in real time;
Extracting a first predetermined number of image frames from the mobile face image by a processor;
Predicting an rPPG signal based on a predictor model composed of a 3D convolutional neural network using the first predetermined number of image frames; and
Stored on a computer-readable recording medium to execute a process of acquiring the user's heart rate based on the predicted rPPG signal by the processor,
A heart rate measurement computer program using real-time deep learning-based mobile face images.