JP7308564B2 - Non-contact bioindex measurement method - Google Patents

Description

Plethysmography measures and analyzes the changes in shape and form that naturally occur when the volume of tissues such as human organs and blood vessels changes due to the flow of blood vessels. It is a method related technology.

Photoplethysmography (PPG) is the most common technique for measuring photoplethysmography (PPG) using light. and the Beer-Lambert law proportional to the thickness of the absorbing layer. According to this law, the change in transmitted light results in a signal proportional to the change in the volume of the substance through which it is transmitted. can do.

Recently, a technology using rPPG (remote photoplethysmography) has appeared, one step ahead of the technology using PPG. As the most popular technology for detecting signals related to heartbeat using PPG, a device with a camera and lighting attached at a short distance, such as a smartphone, is brought into direct contact with the human body to irradiate light and transmit light immediately. If there is a technology to measure and obtain PPG, recently, technology related to rPPG (remote photoplethysmography), which grasps changes in blood vessel volume from signals obtained from images taken by a camera, has been continuously researched and developed. It is a situation where

Since the technology using rPPG does not require contact between the target object and the measuring device, it can be applied to various devices and places equipped with cameras, such as airport immigration offices and telemedicine.

However, the technology related to rPPG has a large impact on the signal due to ambient light and noise generated by the movement of the object during the process of photographing the object with the camera, so only the signal related to the volume change of the object to be measured is extracted from the captured image. It can be seen that the technology to measure biosignals using rPPG is the core technology.

A problem to be solved according to one embodiment is to obtain a biometric index in a non-contact manner.

A problem to be solved by another embodiment is to reduce noise due to the subject's movement in order to acquire the biometric index.

Another problem to be solved by another embodiment is to reduce noise due to changes in the intensity of external light in order to obtain a biometric index.

A problem to be solved by another embodiment is to obtain various biometrics at the same time.

A problem to be solved by another embodiment is to acquire biometric information based on various biometric indices.

A problem to be solved by another embodiment is to acquire various biometric indices in a related manner at the same time.

A problem to be solved by another embodiment is detecting drowsiness based on the heart rate of the subject and the LF/HF of the heartbeat signal.

A problem to be solved by another embodiment relates to a smart mirror device that acquires at least two or more relevant biometrics.

A problem to be solved by another embodiment relates to a method of operating a smart mirror device that obtains at least two or more relevant biometrics.

A problem to be solved by another embodiment relates to a smart mirror device including an opening and closing device.

A non-contact method for measuring a bioindex according to one embodiment comprises:
acquiring a plurality of image frames of the subject;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
calculating a first difference value and a second difference value based on the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; and wherein the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is a difference value between the first color channel value and the second color channel value for the same image frame. indicating a difference value of the three color channel values;
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; obtaining a first characteristic value based on the average value of the difference values;
obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group; determining a biometric index for the subject based on the first characteristic value and the second characteristic value;
The first color channel value is the average pixel value of the first color channel for one image frame, the second color channel value is the average pixel value of the second color channel for one image frame, and the third color channel value is the average pixel value of the second color channel for one image frame. A channel value can indicate the average pixel value of the third color channel for one image frame.

A method for measuring a biometric index using an infrared camera according to one embodiment includes:
obtaining a plurality of image frames of the subject using an infrared camera;
obtaining a first area value, a second area value, and a third area value for at least one image frame included in the plurality of image frames; and acquiring at least one image frame included in the plurality of image frames. calculating a first difference value and a second difference value based on the first region value, the second region value and the third region value for
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; obtaining a first characteristic value based on the average value of the difference values;
obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group; determining a biometric index for the subject based on the first characteristic value and the second characteristic value;
The first region value is an average pixel value for a first region of interest in one image frame, the second region value is an average pixel value for a second region of interest in one image frame, and the third region value is an average pixel value for a third region of interest in one image frame, the first difference value being a difference value between the first region value and the second region value for the same image frame, and the second difference value being the same image frame is the difference value between the first region value and the third region value for

A biometric index acquisition device according to one embodiment includes:
an image acquisition unit for acquiring an image frame of the person to be measured; and a control unit for acquiring a biometric index using the image frame,
the control unit acquires a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of acquired image frames;
calculating a first difference value and a second difference value based on the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; death,
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; Obtain the first characteristic value based on the average value of the difference values,
obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group;
determining a biometric index for the subject based on the first characteristic value and the second characteristic value;
The first color channel value is the average pixel value for the first color channel in one image frame, the second color channel value is the average pixel value for the second color channel in one image frame, and the third color channel value is the average pixel value for the second color channel in one image frame. channel value is the average pixel value for the third color channel in one image frame,
The first difference value is the difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value and the third color channel value for the same image frame. is the difference between

A non-contact biometric method according to one embodiment comprises:
acquiring a plurality of image frames of the subject;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
Obtaining a first characteristic value based on the first, second and third color channel values for at least one image frame included in a first group of image frames acquired during a first preset time. stages and
Obtaining a second characteristic value based on the first, second and third color channel values for at least one image frame included in a second group of image frames acquired during a second preset time. and determining a biometric index based on the first characteristic value and the second characteristic value;
The first group of image frames and the second group of image frames are at least partially overlapped such that what is included in the first group of image frames is included in the second group of image frames to determine the biometric index. A first characteristic value for a first image frame that does not exist is used, a first characteristic and a second characteristic value for a second image frame included in the first image frame group and the second image frame group are used, and the second A second characteristic value for a third image frame included in the image frame group but not included in the first image frame group may be used.

A non-contact biometric method according to one embodiment comprises:
acquiring a plurality of image frames including a first group of image frames and a second group of image frames at least partially overlapping the first group of image frames;
obtaining a biometric index based on the first group of image frames;
outputting a biometric index at a first time point;
obtaining a first biometric index based on the second group of image frames;
and outputting the biometric index at a second time later than the first time,
the biometric index output at the first time point is the biometric index obtained based on the first image frame group;
the biometric index output at the second time point is the first biometric index if the difference between the first biometric index and the biometric index output at the first time point is equal to or less than a reference value;
If the difference between the first biometric index and the biometric index output at the first time point exceeds a reference value, the biometric index is corrected from the biometric index output at the first time point.

A bioindex measurement method according to one embodiment includes:
acquiring a plurality of image frames of the subject;
setting a first area and a second area for at least one image frame included in the plurality of image frames;
oxygen saturation of the subject according to a first feature obtained based on at least two color channel values among a first color channel value, a second color channel value and a third color channel value for the first region; and
determining the heart rate of the subject based on a second characteristic obtained based on a first difference value that is the difference between the first color channel value and the second color channel value for the first region; ,
A second difference value, which is a difference value between the first color channel value and the second color channel value for the second region, and a third feature obtained based on the first difference value determine the blood pressure of the subject. the step of deciding and
and outputting the oxygen saturation, heart rate and blood pressure;
the first feature is obtained based on a first set of image frames obtained during a first time period;
the second feature is obtained based on a second set of image frames obtained during a second time period;
the third feature is obtained based on a third set of image frames obtained during a third time period;
The first image frame group, the second image frame group, and the third image frame group commonly include a plurality of image frames so that the oxygen saturation, heart rate, and blood pressure are obtained in a correlated manner. can be done.

According to another embodiment, a bioindex measurement method comprises:
acquiring a plurality of image frames of the subject;
setting a first area, a second area, and a third area for at least one image frame included in the plurality of image frames;
Oxygen saturation of the subject according to a first feature obtained based on at least two color channel values among a first color channel value, a second color channel value and a third color channel value for the first region and
determining the heart rate of the subject based on a second characteristic obtained based on a first difference value that is the difference between the first color channel value and the second color channel value for the first region; ,
a second difference value that is a difference value between the first color channel value and the second color channel value for the second region and a difference value between the first color channel value and the second color channel value for the third region; determining the blood pressure of the subject according to a third characteristic obtained based on a third difference value; and outputting the oxygen saturation, heart rate and blood pressure;
the first feature is obtained based on a first set of image frames obtained during a first time period;
the second feature is obtained based on a second set of image frames obtained during a second time period;
the third feature is obtained based on a third set of image frames obtained during a third time period;
The first image frame group, the second image frame group, and the third image frame group commonly include a plurality of image frames so that the oxygen saturation, heart rate, and blood pressure are obtained in a correlated manner. can be done.

In another embodiment, a bioindex measurement method comprises:
acquiring a plurality of image frames of the subject;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
determining the oxygen saturation of the subject according to a first feature obtained based on at least two or more of the first, second and third color channel values; ,
Based on a first difference value that is a difference value between the first color channel value and the second color channel value and a second difference value that is a difference value between the first color channel value and the third color channel value. determining the heart rate of the subject according to a second feature;
determining the blood pressure of the subject by a third characteristic obtained based on the first difference value and the second difference value; and outputting the oxygen saturation, heart rate and blood pressure. ,
the first feature is obtained based on a first set of image frames obtained during a first time period;
the second feature is obtained based on a second set of image frames obtained during a second time period;
the third feature is obtained based on a third set of image frames obtained during a third time period;
The first image frame group, the second image frame group, and the third image frame group commonly include a plurality of image frames so that the oxygen saturation, heart rate, and blood pressure are obtained in a correlated manner. can be done.

In another embodiment, a bioindex measurement method comprises:
acquiring a plurality of image frames of the subject;
setting at least two or more regions for at least one image frame included in the plurality of image frames;
obtaining a first color channel value, a second color channel value, a third color channel value, a fourth color channel value and a fifth color channel value for at least one image frame included in the plurality of image frames; ,
determining the oxygen saturation of the subject according to a first feature obtained based on at least two of the first, second and third color channel values; ,
determining the heart rate of the subject according to a second characteristic obtained based on at least two or more of the first, second and third color channel values;
determining the blood pressure of the subject according to a third characteristic obtained based on at least two or more of the first color channel value, the second color channel value and the third color channel value;
determining the body temperature of the subject according to a fourth characteristic obtained based on the fourth color channel values; and outputting the oxygen saturation, heart rate, body temperature and blood pressure;
the first color channel value is a Green channel value;
the second color channel value is a Red channel value;
the third color channel value is a Blue channel value;
the fourth color channel value is a Saturation channel value;
The fifth color channel value is the Hue channel value.

In another embodiment, a bioindex measurement method comprises:
acquiring a plurality of image frames of the subject;
obtaining N preliminary heart rates based on at least one image frame included in the plurality of image frames;
obtaining M preliminary oxygen saturation levels based on at least one image frame included in the plurality of image frames;
obtaining K preliminary blood pressures based on at least one image frame included in the plurality of image frames;
obtaining a heart rate based on the N preliminary heart rates;
obtaining an oxygen saturation level based on the M preliminary oxygen saturation levels;
obtaining blood pressure based on the K preliminary blood pressures; and outputting the heart rate, oxygen saturation and blood pressure,
When the image frame acquired when the subject is in the first state is the first image frame, the first image frame includes the N preliminary heart rates, the M preliminary oxygen saturations and the It may be commonly included in the image frame for obtaining K blood pressures.

A biometric information acquisition method according to one embodiment includes:
acquiring a plurality of image frames of the subject;
obtaining a first biometric index based on a first group of image frames including at least one image frame included in the plurality of image frames;
obtaining a second biometric index based on a second group of image frames including at least one image frame included in the plurality of image frames;
acquiring biometric information based on at least one of the first biometric index and the second biometric index; and outputting the first biometric index, the second biometric index and the biometric information. ,
At least a part of the first image frame group and the second image frame group may be overlapped in order to obtain biometric information reflecting the specific state of the subject.

A method for detecting drowsiness based on heart rate according to one embodiment includes:
obtaining the subject's heart rate, which may be performed by at least one processor;
comparing the heart rate and a reference heart rate to obtain a comparison result;
comparing the heart rate and a reference heart rate to obtain a comparison result;
obtaining a duration of the duration in which the heart rate is equal to or lower than the reference heart rate based on the comparison result; and determining whether the duration reaches the reference duration. and a drowsiness sensing phase for determining a drowsiness state.

A method for detecting drowsiness based on heart rate and LF/HF according to one embodiment includes:
acquiring a heart rate for a subject, which may be performed by at least one processor;
comparing the heart rate and a reference heart rate to obtain a comparison result;
obtaining a first drowsiness parameter based on the comparison result;
obtaining an LF/HF (Low Frequency/High Frequency) value indicating a ratio of sympathetic nerve activity and sub-sympathetic nerve activity of the object;
obtaining a second drowsiness parameter based on the LF/HF values of the subject; and determining a drowsiness state of the subject using at least one of the first drowsiness parameter and the second drowsiness parameter. and a drowsiness sensing phase.

A smart mirror device according to one embodiment includes:
an image acquisition unit for acquiring a plurality of image frames of the reflective mirror surface and the subject;
a display unit disposed behind the reflective mirror surface to display visual information through the reflective mirror surface; a control unit for obtaining a
The control unit controls the display unit to display a first biometric index obtained based on a first image frame group included in the plurality of image frames at a first time point, and controls the display unit at a second time point to display the first biometric index obtained based on a first image frame group included in the plurality of image frames. controlling the display unit to display a second biometric index obtained based on a second image frame group included in a plurality of image frames;
the first image frame group and the second image frame group include at least one image frame in common so that the first biometric index and the second biometric index are associated with each other;
The at least one image frame commonly included in the first image frame group and the second image frame group is the subject observed by the subject through the reflecting mirror surface before the first and second time points. An image frame captured in a first state of the person may be included.

A method of operating a smart mirror device according to one embodiment includes obtaining an on-trigger;
acquiring a plurality of image frames of the subject;
obtaining a first biometric index based on a first group of image frames included in the plurality of image frames;
obtaining a second biometric index based on a second group of image frames included in the plurality of image frames;
displaying the first and second biometric indices;
acquiring an off-trigger; and ceasing acquisition of multiple image frames for the subject;
The first image frame group and the second image frame group include at least one image frame in common so that the first biometric index and the second biometric index are associated with each other;
The on-trigger can be obtained from at least one sensor, and the off-trigger can be obtained from an image sensor for capturing the plurality of image frames.

A smart mirror device according to another embodiment includes:
a reflective mirror surface, an image acquisition unit for acquiring a plurality of image frames;
a display unit positioned behind the reflective mirror surface for displaying visual information through the reflective mirror surface;
an opening/closing unit arranged in front of the image acquisition unit for opening and closing the field of view of the image acquisition unit; and a control unit for acquiring a biometric index by controlling operations of the image acquisition unit and the display unit. and including
The surface of the opening/closing part is formed of a reflective mirror, and when the opening/closing part is in an open state, the controller controls the display unit so that the biometric index and at least one or more visual information can be displayed through the display unit. and when the opening/closing unit is in a closed state, the control unit may control the display unit to display at least one or more visual information through the display unit.

According to one embodiment of the present invention, a method for obtaining a biometric index in a non-contact manner is provided.

Another embodiment of the present invention provides a method for reducing noise due to subject movement.

According to yet another embodiment of the present invention, a method for reducing noise due to intensity changes of external light is provided.

According to yet another embodiment of the present invention, a method for obtaining multiple bioindexes at the same time is provided.

According to another embodiment of the present invention, a method for obtaining biometric information based on various bioindexes is provided.

According to yet another embodiment of the present invention, a method for relatedly acquiring multiple biometrics at the same time is provided.

According to yet another embodiment of the present invention, a smart mirror device is provided for obtaining at least two or more relevant biometrics.

According to yet another embodiment of the present invention, a method of operating a smart mirror device for obtaining at least two or more relevant biometrics is provided.

Another embodiment of the present invention provides a method and apparatus for detecting drowsiness based on the heart rate of a subject and the LF/HF of the heartbeat signal.

According to yet another embodiment of the present invention, a smart mirror device including an opening and closing device is provided.

1 is a diagram of a biometric index and biometric information management system according to one embodiment; FIG. FIG. 10 is a diagram related to a biometric index and biometric information management system according to another embodiment; 1 is a diagram for explaining a biometric index acquisition device according to an embodiment; FIG. FIG. 4 is a flow chart illustrating a biometric index acquisition method according to one embodiment; FIG. FIG. 4 illustrates a biometric index acquisition method according to one embodiment; 4 is a flow chart showing a biometric information acquisition method according to one embodiment. FIG. 4 is a diagram for explaining a biometric acquisition method using a biometric acquisition model; 4 is a flowchart for explaining a heart rate measurement method according to one embodiment; 4 is a flowchart for explaining an oxygen saturation measurement method according to one embodiment; 4 is a flowchart for explaining an oxygen saturation measuring method according to another embodiment; 4 is a flowchart for explaining a blood pressure measurement method according to one embodiment; 9 is a flowchart for explaining a blood pressure measurement method according to another embodiment; 4 is a flowchart for explaining a method of measuring body temperature according to one embodiment; 4 is a flowchart for explaining a heart rate acquisition method according to one embodiment; 4 is a graph for color channel values according to one embodiment; 5 is a graph for explaining a noise reduction method according to one embodiment; FIG. 4 is a diagram showing the absorbance of hemoglobin and oxyhemoglobin in the visible light band; FIG. 4 is a diagram for explaining a characteristic value acquisition method according to an embodiment; FIG. 10 is a diagram for explaining a characteristic value acquisition method according to another embodiment; FIG. 4 is a diagram for explaining a method of using a plurality of characteristic values; It is the graph which extracted the frequency component from the graph with respect to the characteristic value. FIG. 4 is a diagram for explaining a heart rate acquisition method according to one embodiment; 4 is a flow chart illustrating a method for correcting an output heart rate according to one embodiment; FIG. 4 is a diagram for explaining a heartbeat signal extraction method according to an embodiment; FIG. 3 is a diagram for explaining a heart rate acquisition method using infrared rays according to one embodiment; FIG. 3 is a diagram for explaining a heart rate acquisition method using infrared rays according to one embodiment; 4 is a flow chart for explaining a biometric index acquisition method according to an embodiment; FIG. 4 is a diagram for explaining a method of acquiring a plurality of biometric indices and biometric information according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; FIG. 4 is a diagram for explaining a method of acquiring multiple related bioindexes according to one embodiment; FIG. 4 is a diagram for explaining a method for acquiring a plurality of biometric indices according to one embodiment; 1 is a block diagram of a device for detecting drowsiness based on heart rate; FIG. 1 is a flowchart of a method for detecting drowsiness based on heart rate; Fig. 3 is a graph of a subject's average heart rate based on the subject's heart rate; 4 is a graph related to the heart rate of a person to be measured for explaining a state in which the person to be measured is detected to be drowsy based on the heart rate; Fig. 3 is a graph of heart rate with noise; It is a graph for explaining the situation in which the person to be measured has recovered to a steady state. Fig. 3 is a flow chart of a method for detecting drowsiness based on LF/HF; 3 is a graph relating to LF/HF of a subject for explaining a state in which the subject is perceived to be sleepy based on LF/HF. 3 is a graph relating to LF/HF of a person to be measured for showing a state in which the person to be measured is recovering from drowsiness based on LF/HF. Fig. 3 is a flow chart of a method for detecting drowsiness based on heart rate and LF/HF; 1 is a diagram for explaining a smart mirror device according to one embodiment; FIG. 1 is a diagram for explaining a smart mirror device according to one embodiment; FIG. FIG. 4 is a diagram for explaining a smart mirror device outputting a guide area according to an embodiment; FIG. 4 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment; 1 is a diagram for explaining a smart mirror device according to one embodiment; FIG. FIG. 4 is a diagram illustrating a display device for measuring biometrics in real time according to an embodiment; FIG. 4 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment; FIG. 4 is a diagram for explaining a smart mirror device including an opening/closing device according to an embodiment; FIG. 4 is a diagram for explaining a smart mirror device arranged in a crate according to an embodiment; 4 is a flowchart for explaining a method of operating a smart mirror device according to one embodiment; FIG. 4 is a diagram for explaining the operation of a smart mirror device using a trigger signal according to one embodiment; FIG. 4 is a diagram for explaining a method of operating a smart mirror device according to an embodiment; It is a figure for demonstrating the operation|movement of the smart mirror apparatus by one Example. FIG. 4 is a diagram for explaining a method of operating a smart mirror device according to an embodiment; It is a figure for demonstrating the operation|movement of the smart mirror apparatus by one Example. FIG. 4 is a diagram for explaining a method of operating a smart mirror device according to an embodiment; FIG. 4 is a diagram for explaining the operation of a smart mirror according to one embodiment; 1 is a diagram for explaining a bioindex measuring device according to an embodiment; FIG. 1 is a diagram for explaining a bioindex measuring device according to an embodiment; FIG. 1 is a diagram for explaining a biometric measuring device arranged in an autonomous vehicle according to one embodiment; FIG. 4 is a flowchart for explaining a method of operating a biometric device according to one embodiment; 4 is a flowchart for explaining a method of operating a biometric device according to one embodiment; FIG. 4 is a diagram for explaining an operating method of a driving scheduling assistance device using a biometric measuring device according to an embodiment; 4 is a flowchart for explaining a method of operating a biometric device according to one embodiment; 4 is a flowchart for explaining an operation method of the driving index calculation device according to one embodiment; 1 is an illustration of an infant monitoring device according to one embodiment; FIG. FIG. 10 is a diagram for explaining occurrence of an event for an infant; 4 is a flow chart illustrating a method of operating an infant monitoring device according to one embodiment; 4 is a flow chart illustrating a method of operating an infant monitoring device according to one embodiment; 1 illustrates a mobile application for implementing an infant monitoring system according to one embodiment; FIG. FIG. 2 is a diagram for explaining a bioindex measuring device arranged in a library according to one embodiment; 4 is a flow chart for explaining a method of operating a biometric measuring device according to an embodiment; 4 is a flow chart for explaining a method of operating a biometric measuring device according to an embodiment; 1 is a diagram for explaining a biometric index measuring device used for cognitive rehabilitation treatment according to an embodiment; FIG. 4 is a flow chart for explaining a method of operating a biometric measuring device according to an embodiment; 1 is a diagram for explaining a biometric index measuring device used for immigration inspection according to one embodiment; FIG. 4 is a flow chart for explaining a method of operating a biometric measuring device according to an embodiment; 1 is a diagram for explaining a biometric measuring device used in a security device according to one embodiment; FIG. 4 is a flowchart illustrating a method of operating a security device according to one embodiment; 1 is a diagram for explaining a biometric measuring device used in a kiosk according to one embodiment; FIG. 4 is a flow chart for explaining a method of operating a biometric measuring device according to an embodiment;

The embodiments described herein are for the purpose of clearly explaining the idea of the present invention to those of ordinary skill in the art to which the present invention pertains. Rather than being limited to the illustrated embodiments, the scope of the invention should be construed to include any modifications or variations that do not depart from the spirit of the invention.

The terms used in this specification are selected as general terms that are used as widely as possible in consideration of the function in the present invention, but this is based on common knowledge in the technical field to which the present invention belongs. It may change due to the owner's intention, judicial precedent, or the emergence of new technology. However, when a specific term different from this is defined and used with an arbitrary meaning, the meaning of the term will be described separately. Therefore, terms used herein should be construed based on the actual meaning of the term rather than the simple terminology and the context of the specification as a whole.

The drawings in this specification are intended to facilitate the explanation of the present invention, and the shapes shown in the drawings are expanded and displayed as necessary for understanding the present invention. are not limited by the drawings.

If it is determined that a detailed description of the configuration or function disclosed in the present specification may obscure the gist of the present invention, detailed description thereof will be omitted as necessary.

According to one embodiment, a method of non-contact biometric measurement comprising:
acquiring a plurality of image frames of the subject;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
calculating a first difference value and a second difference value based on the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; and wherein the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is a difference value between the first color channel value and the second color channel value for the same image frame. indicating a difference value of the three color channel values;
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; obtaining a first characteristic value based on the average value of the difference values;
obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group; determining a biometric index for the subject based on the first characteristic value and the second characteristic value;
the first color channel value is the average pixel value of the first color channel for one image frame;
the second color channel value is the average pixel value of the second color channel for one image frame;
A biometric method can be provided, wherein the third color channel value indicates the average pixel value of the third color channel for one image frame.

Here, the biometric index may include at least one of heart rate and blood pressure.

Here, the first, second and third color channels are color channels according to the RGB color space.

Here, considering the absorbance of hemoglobin and oxyhemoglobin, the first color channel is set to Green channel, the second color channel is set to Red channel, and the third color channel is set to reduce noise. can be set to the Blue channel.

wherein the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group;
the second characteristic value is obtained based on a second deviation value of the second difference value for at least one image frame included in the first image frame group;
the first deviation value is calculated based on a first difference value for the at least one image frame and an average value of the first difference values for the first group of image frames;
The second deviation value may be calculated based on a second difference value for the at least one image frame and an average value of the second difference values for the first group of image frames.

Here, the first characteristic value and the second characteristic value are normalized values.

Here, the first characteristic value is a value normalized by a first standard deviation value, the second characteristic value is a value normalized by a second standard deviation value, and the first standard deviation value is The standard deviation value of the first difference values for the first group of image frames, and the second standard deviation value is the standard deviation value of the second difference values for the first group of image frames.

Here, the biometric index for the subject can be determined based on a third characteristic value obtained by summing the first characteristic value and the second characteristic value.

Here, the step of outputting a biometric index is further included, wherein the determined biometric index includes a first biometric index and a second biometric index, and the output biometric index is the first biometric index and the second biometric index. 2 can be determined based on biometric indices.

Here, the first biometric index is determined based on a second image frame group, the second biometric index is determined based on a third image frame group, and the number of image frames included in the first image frame group. may be less than the number of image frames included in the second and third image frame groups, and the first image frame group may be included in the second image frame group.

Here, the number of image frames included in the second image frame group may be the same as the number of image frames included in the third image frame group.

The step of outputting biometrics based on the determined biometrics, wherein the determined biometrics include at least four preliminary biometrics, and the output biometrics are the four biometrics. can be determined based on two preliminary biometric indices.

The step of outputting a biometric index based on the determined biometric index, wherein the output biometric index includes a first biometric index and a second biometric index, wherein the second biometric index is is the same kind of bioindex as the first bioindex, the second bioindex is output after the first bioindex is output, and the difference between the second bioindex and the first bioindex is If it exceeds the reference value, the second biometric index may be corrected and output.

According to another embodiment, a method for non-contact biometric measurement using an infrared camera, comprising:
obtaining a plurality of image frames of the subject using an infrared camera;
obtaining a first region value, a second region value and a third region value for at least one image frame included in the plurality of image frames;
calculating a first difference value and a second difference value based on the first area value, the second area value and the third area value for at least one image frame included in the plurality of image frames; ,
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; obtaining a first characteristic value based on the average value of the difference values;
obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group; determining a biometric index for the subject based on the first characteristic value and the second characteristic value;
the first region value is the average pixel value for the first region of interest in one image frame;
the second region value is an average pixel value for a second region of interest in one image frame;
the third region value is an average pixel value for a third region of interest in one image frame;
the first difference value is a difference value between a first area value and a second area value for the same image frame;
A biometric index measurement method may be provided, wherein the second difference value is a difference value between a first area value and a third area value for the same image frame.

Here, the biometric index may include at least one of heart rate and blood pressure.

Here, the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group, and the second characteristic value is obtained from the first image frame group. a second deviation value of the second difference value for at least one image frame included in a frame group, wherein the first deviation value is the first difference value for the at least one image frame and the first image; The second deviation value is calculated based on the average value of the first difference values for the group of frames, and the second deviation value is the average of the second difference values for the at least one image frame and the second difference value for the first group of image frames. values can be calculated.

According to another embodiment, a bioindex measurement device for non-contact bioindex measurement, comprising:
an image acquisition unit for acquiring an image frame of the person to be measured; and a control unit for acquiring a biometric index using the image frame,
the control unit acquires a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of acquired image frames;
calculating a first difference value and a second difference value based on the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; death,
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; Obtain the first characteristic value based on the average value of the difference values,
A second characteristic value is obtained based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group, and the first characteristic value is obtained. determining a biometric index for the subject based on the characteristic value and the second characteristic value;
the first color channel value is the average pixel value for the first color channel in one image frame;
the second color channel value is the average pixel value for the second color channel in one image frame;
the third color channel value is the average pixel value for the third color channel in one image frame;
the first difference value is a difference value between a first color channel value and a second color channel value for the same image frame;
A biometric measuring device can be provided, wherein the second difference value is a difference value between a first color channel value and a third color channel value for the same image frame.

Here, the first, second and third color channels are color channels according to the RGB color space, and the first color channel is the Green channel in order to reduce noise considering the absorbance of hemoglobin and oxyhemoglobin. , the second color channel can be set to the Red channel and the third color channel can be set to the Blue channel.

Here, the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group, and the second characteristic value is obtained from the first image frame group. a second deviation value of the second difference value for at least one image frame included in a frame group, wherein the first deviation value is the first difference value for the at least one image frame and the first image; The second deviation value is calculated based on the average value of the first difference values for the group of frames, and the second deviation value is the average of the second difference values for the at least one image frame and the second difference value for the first group of image frames. values can be calculated.

Here, the control unit acquires biometric information based on the determined biometric index, and the biometric information may include at least one of emotion information, drowsiness information, stress information, and excitement level information.

According to yet another embodiment, a method of non-contact biometric measurement comprising:
acquiring a plurality of image frames of the subject;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
Obtaining a first characteristic value based on the first, second and third color channel values for at least one image frame included in a first group of image frames acquired during a first preset time. stages and
Obtaining a second characteristic value based on the first, second and third color channel values for at least one image frame included in a second group of image frames acquired during a second preset time. and determining a biometric index based on the first characteristic value and the second characteristic value;
the first group of image frames and the second group of image frames are at least partially overlapped;
A first characteristic value for a first image frame included in the first image frame group but not included in the second image frame group is used to determine the biometric index. A third image frame included in the second image frame group but not included in the first image frame group by using the first characteristic and the second characteristic value for the second image frame included in the second image frame group A biometric method can be provided in which a second characteristic value for is utilized.

A first difference based on at least a portion of the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames. including calculating a value;
the first characteristic value is obtained based on the first difference value for at least one image frame included in the first image frame group and an average value of the first difference values for the first image frame group;
the second characteristic value is obtained based on an average value of the first difference value for at least one image frame included in the second image frame group and the second difference value for the second image frame group;
the first difference value is a difference value between a first color channel value and a second color channel value for the same image frame;
The second difference value is the difference value between the first color channel value and the third color channel value for the same image frame.

wherein the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group;
the second characteristic value is obtained based on a second deviation value of the first difference value for at least one image frame included in the second image frame group;
the first deviation value is calculated based on a first difference value for at least one image frame included in the first image frame group and an average value of the first difference values for the first image frame group;
The second deviation value may be calculated based on a first difference value for at least one image frame included in the second image frame group and an average value of the first difference values for the second image frame group. can.

According to another embodiment, a biometric output method for obtaining and outputting biometric data in a non-contact manner, comprising:
obtaining a plurality of image frames including a first group of image frames and a second group of image frames at least partially overlapping the first group of image frames;
obtaining a biometric index based on the first group of image frames;
outputting a biometric index at a first time point;
obtaining a first biometric index based on the second group of image frames;
and outputting the biometric index at a second time later than the first time,
the biometric index output at the first time point is the biometric index obtained based on the first image frame group;
The bioindex output at the second time point is the first bioindex if the difference between the first bioindex and the bioindex output at the first time point is equal to or less than a reference value; and the bioindex output at the first time point exceeds a reference value, the bioindex is corrected from the bioindex output at the first time point. .

wherein the corrected biometric index is
when the first bioindex is greater than the bioindex output at the first time point, the bioindex is corrected by adding a preset value to the bioindex output at the first time point;
When the first biometric index is smaller than the biometric index output at the first time point, the biometric index is corrected by subtracting a preset value from the biometric index output at the first time point.

According to one embodiment, a method for simultaneously measuring various Physiological parameters including heart rate, oxygen saturation and blood pressure, comprising:
acquiring a plurality of image frames of the subject;
setting a first area and a second area for at least one image frame included in the plurality of image frames;
oxygen saturation of the subject according to a first feature obtained based on at least two color channel values among a first color channel value, a second color channel value and a third color channel value for the first region; and
determining the heart rate of the subject based on a second characteristic obtained based on a first difference value that is the difference between the first color channel value and the second color channel value for the first region; ,
A second difference value, which is a difference value between the first color channel value and the second color channel value for the second region, and a third feature obtained based on the first difference value determine the blood pressure of the subject. the step of deciding and
and outputting the oxygen saturation, heart rate and blood pressure;
the first feature is obtained based on a first set of image frames obtained during a first time period;
the second feature is obtained based on a second set of image frames obtained during a second time period;
the third feature is obtained based on a third set of image frames obtained during a third time period;
The first group of image frames, the second group of image frames, and the third group of image frames commonly include a plurality of image frames so that the oxygen saturation, heart rate, and blood pressure are acquired in a mutually related manner; A biometric method can be provided.

wherein said first color channel value is a Green channel value, said second color channel value is a Red channel value and said third color channel value is a Blue channel value.

wherein the first feature is the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the value for a third color channel, in which the absorbance of oxyhemoglobin is higher than that of hemoglobin. It can be obtained based on the third color channel value.

Here, the second feature may be obtained based on a third difference value and the first difference value, which are difference values between the first color channel value and the second color channel value for the first region. can.

wherein said first color channel value is a Green channel value, said second color channel value is a Red channel value and said third color channel value is a Blue channel value so as to reduce noise due to extraneous light. .

Here, the second feature may include frequency component values of time-series data obtained based on the first difference value.

Here, the third feature may include PTT (Pulse Transit Time) obtained based on the second difference value and the first difference value.

wherein the fourth feature is a fourth difference value, the first difference value, the second difference value and the second difference value, which are difference values between the first color channel value and the third color channel value for the second region; 3 can be obtained based on the difference value.

Here, the first area and the second area may include the face area of the subject, and vertical positions of the center of the second area and the center of the first area may be different from each other.

According to another embodiment, a method for simultaneously measuring various Physiological parameters including heart rate, oxygen saturation and blood pressure, comprising:
acquiring a plurality of image frames of the subject;
setting a first area, a second area, and a third area for at least one image frame included in the plurality of image frames;
Oxygen saturation of the subject according to a first feature obtained based on at least two color channel values among a first color channel value, a second color channel value and a third color channel value for the first region and
determining the heart rate of the subject based on a second characteristic obtained based on a first difference value that is the difference between the first color channel value and the second color channel value for the first region; ,
a second difference value that is a difference value between the first color channel value and the second color channel value for the second region, and a difference value between the first color channel value and the second color channel value for the third region; determining the blood pressure of the subject according to a third characteristic obtained based on the third difference value, and outputting the oxygen saturation, heart rate and blood pressure;
the first feature is obtained based on a first set of image frames obtained during a first time period;
the second feature is obtained based on a second set of image frames obtained during a second time period;
the third feature is obtained based on a third set of image frames obtained during a third time period;
The first group of image frames, the second group of image frames, and the third group of image frames commonly include a plurality of image frames so that the oxygen saturation, heart rate, and blood pressure are acquired in a mutually related manner; A biometric method can be provided.

wherein the first feature is the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the value for a third color channel, in which the absorbance of oxyhemoglobin is higher than that of hemoglobin. It can be obtained based on the third color channel value.

Here, the second feature may be obtained based on a third difference value and the first difference value, which are difference values between the first color channel value and the second color channel value for the first region. can.

Here, the third feature may include PTT (Pulse Transit Time) obtained based on the second difference value and the first difference value.

According to another embodiment, a method for simultaneously measuring various Physiological parameters including heart rate, oxygen saturation and blood pressure, comprising:
acquiring a plurality of image frames of the subject;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
determining the oxygen saturation of the subject according to a first feature obtained based on at least two of the first, second and third color channel values; ,
Obtained based on a first difference value that is the difference between the first color channel value and the second color channel value and a second difference value that is the difference between the first color channel value and the third color channel value determining the heart rate of the subject according to a second characteristic;
determining the blood pressure of the subject by a third characteristic obtained based on the first difference value and the second difference value; and outputting the oxygen saturation, heart rate and blood pressure. ,
the first feature is obtained based on a first set of image frames obtained during a first time period;
the second feature is obtained based on a second set of image frames obtained during a second time period;
the third feature is obtained based on a third set of image frames obtained during a third time period;
The first group of image frames, the second group of image frames, and the third group of image frames commonly include a plurality of image frames so that the oxygen saturation, heart rate, and blood pressure are acquired in a mutually related manner; A biometric method can be provided.

wherein the first feature is the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the value for a third color channel, in which the absorbance of oxyhemoglobin is higher than that of hemoglobin. It can be obtained based on the third color channel value.

Here, the second feature may include frequency component values of time-series data obtained based on the first difference value and the second difference value.

Here, the third feature is the slope component value, maximum value, minimum value, local maximum value, local minimum value, local maximum average value of the time-series data acquired based on the first difference value and the second difference value. , a minimum mean value, a difference value for the maximum and minimum mean values, and/or an average value.

According to another embodiment, a method for simultaneously measuring various Physiological parameters including heart rate, oxygen saturation, blood pressure and body temperature, comprising:
acquiring a plurality of image frames of the subject;
setting at least two or more regions for at least one image frame included in the plurality of image frames;
obtaining a first color channel value, a second color channel value, a third color channel value, a fourth color channel value and a fifth color channel value for at least one image frame included in the plurality of image frames; ,
determining the oxygen saturation of the subject according to a first feature obtained based on at least two of the first, second and third color channel values; ,
determining the heart rate of the subject according to a second characteristic obtained based on at least two or more of the first, second and third color channel values;
determining the blood pressure of the subject according to a third characteristic obtained based on at least two or more of the first color channel value, the second color channel value and the third color channel value;
determining the body temperature of the subject according to a fourth characteristic obtained based on the fourth color channel values; and outputting the oxygen saturation, heart rate, body temperature and blood pressure;
the first color channel value is a Green channel value;
the second color channel value is a Red channel value;
the third color channel value is a Blue channel value;
the fourth color channel value is a Saturation channel value;
A biometric method can be provided, wherein the fifth color channel value is a Hue channel value.

wherein the first feature is the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the value for a third color channel, in which the absorbance of oxyhemoglobin is higher than that of hemoglobin. It can be obtained based on the third color channel value.

wherein the second feature is a first difference value that is a difference value between the first color channel value and the second color channel value, and a difference value between the first color channel value and the third color channel value; It can be obtained based on some second difference value.

Here, the second feature can include frequency component values of time-series data obtained based on the first difference value and the second difference value.

Here, the third feature is a first difference value that is a difference value between the first color channel value and the second color channel value, and a difference value between the first color channel value and the third color channel value. It can be obtained based on some second difference value.

Here, the third feature is the slope component value, maximum value, minimum value, local maximum value, local minimum value, local maximum average value of the time-series data acquired based on the first difference value and the second difference value. , a minimum mean value, a difference value for the maximum and minimum mean values, and/or an average value.

Here, the fourth feature can be obtained based on the fourth color channel value and the fifth color channel value.

Here, the fourth characteristic may include the skin temperature of the subject.

According to another embodiment, a method for simultaneously measuring various Physiological parameters including heart rate, oxygen saturation and blood pressure, comprising:
acquiring a plurality of image frames of the subject;
obtaining N preliminary heart rates based on at least one image frame included in the plurality of image frames;
obtaining M preliminary oxygen saturation levels based on at least one image frame included in the plurality of image frames;
obtaining K preliminary blood pressures based on at least one image frame included in the plurality of image frames;
obtaining a heart rate based on the N preliminary heart rates;
obtaining an oxygen saturation level based on the M preliminary oxygen saturation levels;
obtaining blood pressure based on the K preliminary blood pressures; and outputting the heart rate, oxygen saturation and blood pressure,
When the image frame acquired when the subject is in the first state is the first image frame, the first image frame includes the N preliminary heart rates, the M preliminary oxygen saturations and the A biometric method can be provided that is commonly included in the image frame for determining the K blood pressures.

According to another embodiment, a method of obtaining biometric information using various physiological parameters, comprising:
acquiring a plurality of image frames of the subject;
obtaining a first biometric index based on a first group of image frames including at least one image frame included in the plurality of image frames;
obtaining a second biometric index based on a second group of image frames including at least one image frame included in the plurality of image frames;
acquiring biometric information based on at least one of the first biometric index and the second biometric index; and outputting the first biometric index, the second biometric index and the biometric information. ,
A biometric information acquisition method is provided, wherein the first image frame group and the second image frame group are at least partially overlapped in order to acquire biometric information reflecting the specific state of the subject. can

Here, the step of obtaining personal and statistical data about the subject is further included, wherein the biometric information is based on the first biometric index, the second biometric index and the obtained personal and statistical data. can be obtained at

Here, the first and second biometric indices include at least one of heart rate, oxygen saturation, blood pressure, and body temperature, and the biometric information includes at least sleepiness information, stress information, excitement level information, and emotional information. can contain one.

A method for sensing drowsiness based on heart rate according to one embodiment includes:
obtaining the subject's heart rate, which may be performed by at least one processor;
comparing the heart rate and a reference heart rate to obtain a comparison result;
comparing the heart rate and a reference heart rate to obtain a comparison result;
obtaining a duration of the duration in which the heart rate is equal to or lower than the reference heart rate based on the comparison result; and determining whether the duration reaches the reference duration. a drowsiness detection stage for determining a drowsiness state;
the reference duration includes a first reference duration, a second reference duration and a third reference duration;
The second reference duration is longer than the first reference duration, the third reference duration is longer than the second reference duration, and the drowsiness state of the subject is a steady state, a first drowsiness state, and a second drowsiness state. state and a third drowsy state, wherein the first drowsy state indicates a state in which the subject is more likely to enter a sleep state than the steady state, and the second drowsy state indicates that the subject is more likely to enter the sleep state than the first drowsy state. The third drowsiness state may indicate a state in which the body is more likely to enter a sleep state than the second drowsiness state.

wherein said drowsiness sensing step comprises:
determining that the drowsiness state of the subject is the first drowsiness state if the duration is longer than the first reference duration and shorter than the second reference duration;
determining that the drowsiness state of the subject is the second drowsiness state if the duration is longer than a second reference duration and shorter than a third reference duration;
determining that the drowsiness state is the second drowsiness state if the duration is greater than a third reference duration.

Here, the drowsiness sensing step detects a drowsiness-related state of the subject if the duration of the subject's heart rate being equal to or less than the reference heart rate is shorter than the first reference duration. The step of determining at the steady state may be further included.

Here, the first drowsiness state may indicate a state in which the subject is not aware of drowsiness but may physically enter a sleep state.

Here, the heart rate of the subject may represent an average value of a plurality of heart rates in a predetermined time interval including the time point when the heart rate was acquired.

Here, when the drowsiness state of the subject is any one of the first drowsiness state to the third drowsiness state, a recovery duration in which the heart rate is equal to or higher than the reference heart rate obtaining;
A recovery sensing step of determining the drowsiness state of the subject based on whether the recovery duration reaches a recovery reference duration may be further included.

wherein the recovery reference duration includes a first recovery reference duration, a second recovery reference duration and a third recovery reference duration;
The recovery sensing step includes, if the drowsiness state of the subject is a third drowsiness state and the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the subject determining said sleepy state of the body as said second sleepy state;
determining the drowsiness state of the subject as the first drowsiness state if the recovery duration is longer than the second recovery reference duration and shorter than the third recovery reference duration;
determining the drowsy state of the subject as the steady state if the recovery duration is longer than the third recovery reference duration;
if the drowsiness state of the subject is a second drowsiness state, and if the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the drowsiness state of the subject is determining the first drowsiness state;
determining the drowsy state of the subject as the steady state if the recovery duration is longer than the second recovery reference duration and shorter than the third recovery reference duration;
if the drowsiness state of the subject is a first drowsiness state, and if the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the drowsiness state of the subject is The steady state can be determined.

Here, one of the first reference duration, the second reference duration and the third reference duration is the first recovery reference duration, the second recovery reference duration and the third recovery reference duration. It may be the same as at least one during the time.
Here, the first reference duration, the second reference duration and the third reference duration are all different from the first recovery reference duration, the second recovery reference duration and the third recovery reference duration. be able to.

A method for sensing drowsiness based on heart rate and LF/HF performed by at least one processor according to one embodiment comprises:
obtaining a heart rate for the subject;
comparing the heart rate and a reference heart rate to obtain a comparison result;
obtaining a duration of the heart rate change based on the comparison result;
obtaining a first drowsiness parameter (drowsiness stage determined based on heart rate) based on the duration;
obtaining an LF/HF (Low Frequency/High Frequency) value of the subject, which indicates a ratio of sympathetic nerve activity and sub-sympathetic nerve activity of the subject;
obtaining a second drowsiness parameter based on the LF/HF values of the subject; and determining a drowsiness state of the subject using at least one of the first drowsiness parameter and the second drowsiness parameter. and a drowsiness sensing phase.

Here, the step of obtaining the duration may obtain the duration based on a length of a time interval during which the heart rate remains equal to or lower than the reference heart rate based on the comparison result. can.

wherein the drowsiness state includes a first drowsiness state and a second drowsiness state;
The second drowsiness state is a state in which the subject is more likely to enter a sleep state than the first drowsiness state, and the drowsiness state compares the duration with a first reference duration and a second reference duration. determined by a first drowsiness parameter obtained based on the results, and a second drowsiness parameter obtained based on the results of comparing the subject's LF/HF with the first reference value and the second reference value,
the drowsiness state is determined to be the first drowsiness state when the duration is greater than the first reference duration and the LF/HF of the subject is less than the first reference value;
The drowsiness state may be determined to be the second drowsiness state when the duration is greater than the second reference interval or the subject's LF/HF is less than the second reference value.

Here, the drowsiness state further includes a third drowsiness state, and the first drowsiness state is a state in which the subject is not aware of drowsiness but is physically likely to enter the sleep state. The second drowsy state and the third drowsy state are states in which the person is aware of drowsiness and may physically enter the sleep state, and the third drowsy state enters the sleep state more than the second drowsy state. It is highly probable.

Here, the drowsy state includes a steady state, a first stage drowsy state, a second stage drowsy state and a third stage drowsy state,
The first-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the steady state,
The second-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the first-stage drowsiness state,
the third-stage drowsy state indicates a state in which the subject is more likely to enter a sleep state than the second-stage drowsy state;
The greater the value of the first drowsiness parameter and the value of the second drowsiness parameter, the higher the possibility that the subject will enter a sleep state,
If the value of the first drowsiness parameter and the value of the second drowsiness parameter match, the matching value is obtained, and based on the matching value, the drowsiness state is changed to the steady state, the first stage drowsiness state, and the second stage drowsiness. It can be determined by one of the state and the third stage drowsiness state.

wherein the drowsy state includes a steady state, a first stage drowsy state, a second stage drowsy state and a third stage drowsy state,
The first-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the steady state,
The second-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the first-stage drowsiness state,
the third-stage drowsy state indicates a state in which the subject is more likely to enter a sleep state than the second-stage drowsy state;
The greater the value of the first drowsiness parameter and the value of the second drowsiness parameter, the higher the possibility that the subject will enter a sleep state,
if the value of the first drowsiness parameter and the value of the second drowsiness parameter match or do not match, obtaining the greater value of the value of the first drowsiness parameter and the value of the second drowsiness parameter; Based on this, the drowsiness state can be determined as one of a steady state, a first stage drowsiness state, a second stage drowsiness state and a third stage drowsiness state.

wherein the drowsy state includes a steady state, a first stage drowsy state, a second stage drowsy state and a third stage drowsy state,
The first-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the steady state,
The second-stage drowsy state indicates a state in which the subject is more likely to enter a sleep state than the first-stage drowsy state,
the third-stage drowsy state indicates a state in which the subject is more likely to enter a sleep state than the second-stage drowsy state;
The greater the value of the first drowsiness parameter and the value of the second drowsiness parameter, the higher the possibility that the subject will enter a sleep state,
if the value of the first drowsiness parameter and the value of the second drowsiness parameter match or do not match, obtaining the smaller value of the first drowsiness parameter value and the second drowsiness parameter value, and Based on this, the sleepiness state can be determined as one of a steady state, a first stage sleepiness state, a second stage sleepiness state and a third stage sleepiness state.

wherein the drowsiness state includes a steady state, a first stage drowsiness state, a second stage drowsiness state and a third stage drowsiness state;
The first-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the steady state,
The second-stage drowsiness state indicates a state in which the subject is more likely to enter a sleep state than the first-stage drowsiness state,
the third-stage drowsy state indicates a state in which the subject is more likely to enter a sleep state than the second-stage drowsy state;
The greater the value of the first drowsiness parameter and the value of the second drowsiness parameter, the higher the possibility that the subject will enter a sleep state,
if the first drowsiness parameter value and the second drowsiness parameter value match or do not match, obtain an average value of the first drowsiness parameter value and the second drowsiness parameter value; The drowsiness state can be defined as one of a steady state, a first-stage drowsiness state, a second-stage drowsiness state, and a third-stage drowsiness state.

Here, the method can be executed through a recording medium on which a program for executing the method is recorded.

According to another embodiment, a smart mirror device for simultaneously displaying at least two bioindexes of heart rate, oxygen saturation, blood pressure and body temperature, comprising:
an image acquisition unit for acquiring a plurality of image frames of the reflective mirror surface and the subject;
a display unit positioned behind the reflective mirror surface to display visual information passing through the reflective mirror surface;
a control unit for controlling operations of the image acquisition unit and the display unit to acquire a biometric index in a non-contact manner;
The control unit
controlling the display unit to display a first biometric index obtained based on a first image frame group included in the plurality of image frames at a first time point;
controlling the display unit to display a second biometric index obtained based on a second image frame group included in the plurality of image frames at a second time point;
The first image frame group and the second image frame group include at least one image frame in common so that the first biometric index and the second biometric index are associated with each other;
The at least one image frame commonly included in the first image frame group and the second image frame group is the subject observed by the subject through the reflecting mirror surface before the first and second time points. A smart mirror device can be provided that includes image frames captured in a first state of a person.

Here, the first and second bio-indexes may include at least one bio-index among heart rate, oxygen saturation, blood pressure, and body temperature.

Here, the first image frame group and the second image frame group may be identical to each other.

Here, the first image frame group and the second image frame group may be different from each other.

Here, the first time point and the second time point are the same time point.

Here, the first time point is earlier than the second time point, and the at least one image frame commonly included in the first image frame group and the second image frame group is before the first time point. An image frame obtained in a second state of the subject as observed by the subject through a reflective mirror surface may be included.

wherein the control unit controls the display unit to display a third biometric index obtained based on a third image frame group included in the plurality of image frames at a third time point;
the first, second and third image frame groups commonly include at least one image frame so as to associate the first, second and third biometric indices;
At least one image frame commonly included in the first, second and third image frame groups is the subject observed by the subject through the reflecting mirror surface before the first, second and third time points. An image frame acquired in a first state of the subject may be included.

Here, the control unit obtains the first and second biometric indices based on at least some of the plurality of image frames for a first time period, and obtains the plurality of biometric indices for a second time period. recognizing the subject based on at least one of the image frames, wherein the second time is shorter than the first time but the second time is included in the first time; Information about the subject recognized before the second biometric index is displayed can be displayed.

Here, the information about the subject is the previously measured and stored first and second biometrics of the subject.

Here, the control unit controls the display unit to display first information at a fourth time point, controls the display unit to display second information at a fifth time point, and controls the display unit to display second information at a fifth time point. information and time information, wherein the second information includes information for the subject, the fourth point in time is earlier than the fifth point in time, and the fifth point in time is earlier than the first and second points in time It is time.

Here, the second information may include at least one of schedule information, medication information, recognition information, messenger information, and interest information of the subject.

Here, the control unit controls the display unit so as to display the first information at the fourth time point, and controls the display unit so as to display the first information and the second information at the fifth time point. The display unit can be controlled to display the first information, the second information, the first biometric index, and the second biometric index at the second time.

wherein a first image included in the first image frame group is operable to obtain the first biometric index and the second biometric index based on at least three color channel values for the plurality of image frames; The first biometric index and the second biometric index may all be displayed within 10 seconds after the frame is acquired.

Here, when the image frame acquired through the image acquisition unit includes images of a plurality of persons, the control unit performs the first image processing based on the image of one person selected from the plurality of persons according to priority. and a second biometric index, the acquired first and second biometric indices are displayed, and the display unit can be controlled to display information for one person selected from the plurality of people. .

Here, after the image frame including the first subject is obtained, the control unit obtains the image frame including the first subject and the second subject. The first and second biometric indices are obtained based on the image of the subject, and the display unit can be controlled to display information about the first subject.

Here, the controller can determine the priority of the plurality of people based on the acquired image frames.

Here, if the image frames acquired through the image acquiring unit include the images of the first subject and the second subject, the controller controls a fourth image frame included in the plurality of image frames. obtaining the first biometric index for the first subject based on the group, and obtaining the second biometric index for the first subject based on a fifth image frame group included in the plurality of image frames; obtaining the first biometric index for the second subject based on a sixth image frame group included in the plurality of image frames; and obtaining a seventh image frame group included in the plurality of image frames. wherein the fourth and fifth image frame groups commonly include at least one image frame, and the sixth and seventh image A group of frames can have at least one image frame in common.

Here, the control unit acquires the biometric information based on the first biometric index and controls the display unit so that the biometric information can be displayed at a third time point, but the first biometric index and the biometric information are displayed in real time. The biometric information displayed at the third time point to display the biometric information is obtained based on the first biometric index displayed at the first time point, and the first time point is earlier than the third time point. is.

Here, the control unit acquires the biometric information based on the first and second biometric indices, and the biometric information displayed at the third time point in order to improve the accuracy of the biometric information is the obtained based on the first biometric index displayed at a first time point and the second biometric index displayed at a second time point, wherein the third time point is later than the first and second time points .

Here, the biometric information may include at least one of condition information, concentration information, drowsiness information, and emotion information.

Here, the control unit acquires the biometric information based on the first biometric index and controls the display unit so that the biometric information can be displayed at a third time point. The biological information displayed at the third time point is obtained based on the first biological index displayed up to the third time point, and the third time point is later than the first time point.

Here, the biological information displayed at the third time can be obtained based on the average value of the first biological indices displayed up to the third time.

Here, the control unit acquires the biometric information based on the first and second biometric indices, and the biometric information displayed at the third time point is the first biometric information displayed up to the third time point. and the second biometric index displayed up to the third time point, the third time point being later than the first and second time points.

Here, a cycle of updating the first biometric index and a cycle of updating the second biometric index may be different from each other.
Here, the control unit determines the information provision situation, and controls the display unit so that the first information can be displayed in the first situation, and the second information can be displayed in the second situation. In some cases, the first information includes first and second biometric indices, and the second information does not include first and second biometric indices, although the display unit is otherwise controlled.

Here, the control unit can determine the information provision status based on the moving direction of the person.
Here, the smart mirror device may further include a motion sensing sensor for sensing a moving direction of the subject.

Here, the controller may determine the information provision status based on the obtained plurality of image frames.

Here, the first information includes at least one of external weather information, the subject's schedule information, and time information, and the second information includes internal temperature information, internal humidity information, internal air information, and security information. , activity time information.

Here, the first information may include at least two biometric indices, and the second information may include at least one of weather information, time information, news information, schedule information, and medication information.

The controller acquires at least three color channel values for at least one image frame included in the plurality of image frames, and calculates a first difference value and a second difference value based on the at least three color channel values. A difference value may be obtained, and the first biometric index and the second biometric index may be obtained based on the first difference value and the second difference value.

According to another embodiment, there is provided a method of operating a smart mirror device for simultaneously displaying at least two bioindexes of heart rate, oxygen saturation, blood pressure and body temperature, comprising:
obtaining an on-trigger;
acquiring a plurality of image frames of the subject;
obtaining a first biometric index based on a first group of image frames included in the plurality of image frames;
obtaining a second biometric index based on a second group of image frames included in the plurality of image frames;
displaying the first and second biometric indices;
acquiring an off-trigger; and ceasing acquisition of multiple image frames for the subject;
The first image frame group and the second image frame group commonly include at least one image frame so that the first biometric index and the second biometric index are associated with each other, and the on-trigger (on- a trigger is obtained from at least one sensor, and the off-trigger is obtained from an image sensor for obtaining the plurality of image frames. .

Here, the on-trigger may be obtained from at least one sensor of a motion sensor, a touch sensor, a mouse, and a keyboard.

According to another embodiment, a smart mirror device for displaying at least one biometric index of heart rate, oxygen saturation, blood pressure and body temperature, comprising:
a reflective mirror surface, an image acquisition unit for acquiring a plurality of image frames;
a display unit positioned behind the reflective mirror surface to display visual information passing through the reflective mirror surface;
an opening/closing unit disposed in front of the image acquisition unit for opening and closing the field of view of the image acquisition unit; and a control for acquiring a biometric index by controlling operations of the image acquisition unit and the display unit. including the part and
A surface of the opening/closing part is formed of a reflective mirror, and when the opening/closing part is in an open state, the controller controls the display unit so that the biometric index and at least one or more visual information can be displayed through the display unit. and when the opening/closing part is in a closed state, the control part controls the display part so that at least one or more visual information can be displayed through the display part. can be done.

Here, when the opening/closing unit changes from the closed state to the open state, the control unit controls the image obtaining unit to obtain an image frame from the image obtaining unit, and the opening/closing unit changes from the open state to the open state. The image acquisition unit may be controlled not to acquire image frames from the image acquisition unit when the closed state is reached.

0. DEFINITIONS OF TERMS As used herein, the term "measurement" is understood to include all of the following: determining by measuring, determining by inferring, and measuring the size of another quantity relative to a given quantity. be able to.

As used herein, "Heart rate" is understood in terms of the number of heart beats, which can mean the heart rate measured near the heart as a result of heart beats. and the concept of "pulse", which can mean that vibrations produced as a result of a heart beat propagate to peripheral blood vessels.

As used herein, "blood pressure" is understood as the pressure generated in blood vessels when the heart pumps blood, which is a value commonly understood as "blood pressure" regardless of the measurement site (e.g. , measured from the upper arm artery).

As used herein, "oxygen saturation" is understood as the degree of saturation of oxygen in the blood, and more specifically can be understood as the ratio of oxyhemoglobin to total hemoglobin in the blood. .

As used herein, "Core Temperature" is understood as the body temperature possessed by a person or animal and is understood differently from "Skin Temperature" which can be measured at the skin. can be

As used herein, "Skin Temperature" can be understood as the measured surface temperature of the skin.

The term 'image' used herein may be understood as a concept including all of a plurality of images included in one image or video.

As used herein, "Color Channel" is understood as the respective axes that make up the Color space, e.g., the Red, Green and Blue channels make up the RGB color space. Red axis, Green axis, and Blue axis are meant, and such color channels may be configured two-dimensionally, three-dimensionally, or four-dimensionally.

As used herein, the term "image of the person being measured" is understood to mean an image that includes the measurement target position of the person being measured. can be understood with an image containing

As used herein, "personal and statistical data" means personal and statistical data such as age, gender, height, and weight that can be collected from the subject. It can also mean observable personal and statistical data such as facial expressions, wrinkles, complexion, etc. Statistical data calculated for a group that includes or is related to the subject (e.g., 20 It can also mean personal and statistical data that can be quantified along with average blood pressure, average skin color of yellow people, average height of men in their 30s, average weight of Korean men, etc.).

As used herein, "time-series data" may mean, but is not limited to, data enumerated by a time axis, enumerated by an image frame axis that can correspond to time. It can also be commonly understood as 'time-series data', as it means data and data that can be aligned by a time axis or an image frame axis.

1. Biometric parameters and biometric information management system “Physiological parameters” shall mean the results of the physiological activity of the human body that are measured or estimated, such as heart rate, oxygen saturation, blood pressure, body temperature, and blood flow. etc., but is not limited thereto, and may mean other results of physiological activities of the human body that are measured or estimated.

'Physiological information' means information that can be calculated by taking into account at least some of the results of physiological activities of the human body such as biometric indices and personal and statistical data such as facial expressions, postures and ages. For example, sleepiness information, stress information, excitement level information, emotion information, etc., but not limited thereto.

The biometric index and biometric information management system may mean a management system that can store, share, or analyze the biometric index and calculable biometric information that can be measured and can be used to comprehensively manage personal health.

1 and 2 are diagrams related to a biometric index and biometric information management system according to one embodiment, respectively.

Referring to FIG. 1, a biometric and biometric information management system (100) according to one embodiment may include a biometric acquisition device (10) and a server (20).

At this time, the bio-index acquisition device 10 can measure the subject's bio-index (physiological parameters). More specifically, the bioindex acquisition device (10) may measure the bioindex of the subject invasively, non-invasively and by contact, or non-contact. You may measure by the method.

For example, the biometric index acquisition device 10 may analyze a video or image of the subject to measure biometrics such as heart rate, oxygen saturation, and blood pressure of the subject. is not limited to

In addition, the bioindex acquisition device (10) can calculate physiological information based on the measured bioindex. More specifically, the biometric index acquisition device (10) may calculate biometric information based on the measured biometric index, and may include personal and statistical Biometric information may be calculated by comprehensively considering data.

For example, the biometric index acquisition device (10) can calculate biometric information such as emotion information and drowsiness information of the person to be measured based on measured biometric indices such as heart rate, oxygen saturation, and blood pressure. Not limited.

In addition, the biometric index acquisition device (10) can store the measured biometric index and the calculated biometric information. For example, the biometric acquisition device 10 may store the measured biometrics and calculated biometric information of the subject in an internal memory, but is not limited thereto.

In addition, the biometric index acquisition device (10) can display the measured biometric index and the calculated biometric information. For example, the biometric acquisition device (10) may further include a display and use the display to display the measured biometric and calculated biometric information of the subject. Without limitation, the relevant information can also be transmitted so that it can be displayed on an external display.

Also, the biometric index may be displayed once through a display, or the biometric index that changes in real time may be continuously displayed.

In addition, the biometric acquisition device (10) can transmit the biometric and biometric information of the subject to the server (20).

At this time, the biometric index and biometric information transferred to the server 20 can be stored in the server 20 as personal data. For example, the bioindex measured and calculated biometric information from subject A are stored in the server (20) as data for subject A, and the bioindex measured and calculated from subject B are stored in the server (20). The obtained biological information can be stored in the server (20) as data for the person B to be measured.

In addition, the server 20 can communicate with an external terminal as needed to transmit the biometric index and biometric information of the subject. For example, when subject A, whose biometric index and biometric information data are stored in the server (20), visits a hospital for medical treatment, the doctor who treats subject A The biometric index and biometric information for Person A can be used. At this time, the server (20) communicates with the external terminal and communicates with the subject A when the external terminal installed in the hospital requests the transmission of the biometric index and biometric information data to the subject A. can transmit biometric and biometric data to the

In this way, the biometric index and biometric information management system (100) can serve as a basis for providing a continuous and comprehensive management service for personal health. It is a self-evident fact that the biometric index and biometric information management system (100) is the basis for providing various comprehensive management services using biometric indices and biometric information that are continuously measured, stored and managed. is.

Also, referring to FIG. 2, the biometric index and biometric information management system (100) according to one embodiment may include an image acquisition device (30) and a server (20).
At this time, the image acquisition device 30 can acquire a video or image of the subject.

Also, the server (20) can acquire a video or image of the subject from the image acquisition device (30).

In addition, the server (20) can acquire personal and statistical data about the person to be measured from an input device, an external terminal, or the like.

Also, the server (20) can measure the biometric index of the subject based on the acquired video or image. For example, the server 20 may analyze the acquired video or image to measure the subject's biometrics such as heart rate, oxygen saturation, blood pressure, etc., but is not limited thereto.

Also, the server (20) can calculate biometric information based on the measured biometric index. More specifically, the server 20 can also calculate biometric information based on the measured biometric index, and use the measured biometric index and personal and statistical data such as facial expression, posture, age, etc. Biometric information can be calculated by comprehensive consideration.

For example, the server 20 can calculate the subject's emotional information, drowsiness information, and other biometric information based on measured biometric indices such as heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

Also, the server 20 can store the measured biometric index and the calculated biometric information.

Also, it is clear that the biometric and biometric information management system (100) including the image acquisition device (30) and the server (20) can perform the functions of the biometric and biometric information management system described with reference to FIG. , the repeated explanation will be omitted.

2. Various Embodiments of Bioindex Acquisition Apparatus FIG. 3 is a diagram illustrating a bioindex acquisition apparatus according to one embodiment.

Referring to FIG. 3, a biometric index acquisition device (1000) according to one embodiment may include an image acquisition unit (1010), a control unit (1020), a storage unit (1030) and a communication unit (1040). However, the biometric index acquisition device (1000) may include at least part of the image acquisition unit (1010), the control unit (1020), the storage unit (1030), and the communication unit (1040). can. For example, the biometric index acquisition device 1000 includes only the image acquisition unit 1010 and the control unit 1020, but is not limited thereto and can be implemented in various ways.

Also, the image acquisition unit (1010) can acquire a video or an image of the subject. More specifically, the image acquiring unit (1010) includes a photographing device, which can acquire a video or image of the subject using the photographing device, or an external image of the biometric acquisition device (1000). A video or image of the person to be measured can be acquired from a photographing device arranged, but is not limited thereto.

In addition, when the image acquisition unit (1010) acquires a video or an image of the person to be measured from a photographing device, the photographing device includes a visible light camera for obtaining a visible light image and an infrared image. However, it is not limited thereto, and can be provided by a hybrid type camera for acquiring visible light images and infrared images.

Also, when the photographing device acquires a visible light image, the acquired visible light image may be acquired from at least one or more color channel values. For example, the obtained visible light image is obtained from color channel values in an RGB color space denoted by Red, Green, and Blue, and has Hue, Saturation, and Brightness. It can be obtained from the color channel value of the HSV color space indicated by (Value), but is not limited thereto, and can be obtained from the color channel value of various color spaces such as YCrCb and YiQ.

Also, when the photographing device obtains the infrared image, the photographing device may obtain the infrared image through an infrared illuminator disposed inside or outside the photographing device. At this time, the infrared illuminator irradiates infrared rays in the near infrared region, which is a wavelength band of 750 nm to 3000 nm, but is not limited thereto, and includes middle infrared and far infrared regions. infrared) and extreme infrared.

In addition, the control unit (1020) can obtain a biometric index using the image of the subject obtained from the image obtaining unit (1010).

For example, the control unit (1020) analyzes the image of the subject acquired from the image acquisition unit (1010), and obtains biometrics such as heart rate, oxygen saturation, blood pressure, and body temperature of the subject. A wide variety of biometric indices can be obtained, including but not limited to.

Also, the control unit (1020) can calculate biometric information based on the acquired biometric index.

For example, the control unit (1020) can calculate biological information such as emotional information and drowsiness information of the subject based on acquired biological indices such as heart rate, oxygen saturation, blood pressure and body temperature. Various biological information can be calculated without limitation.

Also, the control unit (1020) can control at least part of the operation of the image acquisition unit (1010), the storage unit (1030), and the communication unit (1040).

In addition, the storage unit (1030) can store the biometric index and biometric information obtained from the control unit (1020). More specifically, the storage unit 1030 can store biometrics and biometric information for one subject, and can store biometrics and biometric information for any number of subjects. .

Also, the communication unit (1040) can transmit the biometric index and biometric information acquired from the control unit (1020). More specifically, the communication unit (1040) can transmit the biometric index and biometric information acquired from the control unit (1020) to the management server or to the user's terminal. can.

3. Various embodiments of biometric index and biometric information acquisition method
3.1 Bioindex Acquisition Method FIG. 4 is a flow chart showing a bioindex acquisition method according to one embodiment.

Referring to FIG. 4, the biometric acquisition method (1100) according to one embodiment may include acquiring an image of the subject (S1110).

At this time, in order to acquire an image of the person to be measured, it is possible to acquire images using various cameras such as a visible light camera and an infrared camera, or to acquire images from various cameras, as described above. Detailed description is omitted.

Also, the biometric index acquisition method (1100) according to one embodiment may include detecting a skin area (S1120).

At this time, the skin area may mean an area that can be assumed to be the skin area of the subject in the image of the subject.

Also, the skin area can better reflect changes in blood vessels due to heartbeat, and detecting the skin area in this way can improve the accuracy in biometric index acquisition.

Also, according to one embodiment, in order to detect the skin area, areas other than the skin, such as the subject's eyes and hair, are removed, excluding the skin area that can reflect color changes due to dilation of blood vessels. For example, the color values of areas other than the skin, such as the eyes and hair of the person to be measured, are replaced with meaningless values such as black, but the present invention is not limited thereto.

Also, according to one embodiment, a specific color space can be used to detect the skin area. For example, in the step of detecting the skin area (S1120), the obtained image of the subject is replaced with the value of the YCrCb color space, and the skin area is detected based on the image expressed in the YCrCb color space. It is not limited to this.

It is also obvious that the skin area can be detected using various techniques known in the art for detecting the skin area.

Also, the biometric acquisition method (1110) according to one embodiment may include setting a region of interest (S1130).

At this time, the region of interest may mean a region of interest for data processing in an image of the subject, and may be a region used for data processing to obtain a biometric index. It may mean, but is not limited to.

Also, according to one embodiment, a face region of the person to be measured can be set to set the region of interest. For example, the subject's face region may be set to set a region of interest contained within the subject's face.

More specifically, an area having a constant ratio based on the set center of the face area of the person to be measured can be set as the area of interest.

For example, based on the center of the face region of the subject, it is possible to crop vertically by 80% and horizontally by 60%, but the present invention is not limited to this.

Also, according to one embodiment, feature points can be used to set the region of interest. For example, the nose region of the subject is extracted from the feature points of the acquired image of the subject, and the region of interest is set based on the extracted feature points, but is not limited thereto.

More specifically, a region having a predetermined size centering on the feature point extracted from the set face region is set as the region of interest, but the invention is not limited thereto.

Also, according to one embodiment, a plurality of specific points can be used to set the region of interest. For example, the eye and nose regions of the subject may be extracted from the feature points of the acquired image of the subject, and the region of interest may be set based on the extracted feature points.

Also, when a region of interest is set for each of a plurality of images that are successively acquired, the region of interest for each of the plurality of images can be independently set and associated with each other. Can also be set.

For example, in order to set the region of interest for each of the plurality of images so as to be relevant, the face region of the subject is set for each of the plurality of images, and the face region set in the first image frame and the center of the face region set in the second image frame acquired next to the first image frame does not exceed a critical value, the face region of the second image frame is the same as that of the first image It is set to be the same as the face area set in the frame, but is not limited thereto.

Also, if the difference between the center of the face region set in the first image frame and the center of the face region set in the second image frame exceeds a critical value, the face region of the second image frame is It is set differently from the face area set in one image frame, but is not limited thereto.

In addition, the region of interest according to an embodiment may be set to include a part of the body part or a part of the face according to the biometrics to be acquired, and at least one or more regions of interest may be set. .

For example, the region of interest can be set to include at least a portion of the cheek region in order to obtain the heart rate among biometric indices. More specifically, the region of interest is set to include, but is not limited to, the cheek region, which can reflect the extent of blood vessel dilation due to blood flow and can easily obtain the heart rate.

Also, for example, in order to obtain blood pressure among bioindexes, at least two or more regions of interest are set. A region of interest including a region and a region of interest including a lower region of the face are set, but are not limited thereto.

Also, for example, at least two regions of interest are set in order to obtain blood pressure among the bioindexes, and more specifically, the regions of interest are set to two or more regions of interest that are at different distances from the heart. can For example, the region of interest may be a region of interest including a hand region and a region of interest including a face region, but is not limited thereto.

Also, the biometric acquisition method (1110) according to one embodiment may include processing data for the region of interest (S1140).

Also, to process data for the region of interest, color channel values for the region of interest can be extracted. At this time, the color channel value may be an average value of pixel values of color channels of pixels included in the region of interest, and is also referred to as an average pixel value.

For example, when a color channel value is extracted in an RGB color space, a Red channel pixel value, a Green channel pixel value, and a Blue channel pixel value of each pixel included in the region of interest are extracted. A Red channel value that is the average of the Red channel pixel values, a Blue channel value that is the average of the Blue channel pixel values, and a Green channel value that is the average of the Green channel pixel values are extracted. , color channel values in various color spaces such as HSV, YCrCb color space can be extracted.

Also, the color channel values extracted by a particular color space can be converted to another color space. For example, color channel values extracted in RGB color space can be converted into color channel values in various color spaces such as HSV, YCrCb color space.

Also, the extracted color channel values are color channel values obtained by applying a weighting value to at least some of the color channel values extracted from various color spaces to process the data for the region of interest. may

Also, the color channel values are extracted for each of a plurality of image frames that are successively acquired, and may be extracted for at least some of the image frames.

Also, the color channel values extracted from one image frame can be processed through operations or the like. More specifically, a plurality of channel values obtained from one image frame may be processed through operations such as addition and subtraction, for example, a green channel value and a red channel value obtained from one image frame. Values are processed through differential operations, but the present invention is not limited to this, and various channel values can be processed through various operations.

Also, color channel values or processed values extracted from each of a plurality of image frames may be processed through calculations or the like. More specifically, the color channel values extracted from each of a plurality of image frames or the processed values of the color channel values are processed through operations such as averaging or deviation in a certain interval. However, without limitation, the difference between the maximum value and the minimum value in a certain interval may be obtained and processed through various operations.

Also, the color channel values extracted from each of the plurality of image frames and the processed values of the color channel values may be processed to obtain at least one piece of time-series data.

Also, a feature value for obtaining a biometric index can be extracted based on at least part of the color channel value, the processed value, and the time-series data. For example, the frequency component for obtaining the heart rate is extracted based on the frequency component of the time-series data, but it is not limited to this.

Also, the biometric acquisition method (1110) according to one embodiment may include acquiring a biometric (S1150).

Also, feature values extracted from the region of interest can be used to obtain the biometric index. For example, the heart rate can be obtained based on the frequency components of the time-series data extracted from the region of interest, but the present invention is not limited thereto.

In addition, since the step of processing data for the region of interest (S1140) and the step of obtaining the biometric index (S1150) are different for each biometric index, more details will be described in the corresponding sections.

FIG. 5 is a diagram showing a biometric index acquisition method according to one embodiment.

Referring to FIG. 5, the image for the subject (1161) may include a face region, a skin region can be detected (1162), and a region of interest can be detected (1163). A detailed description of this has been given above, so a repeated description will be omitted.

Also, referring to FIG. 5, a color channel value for the region of interest is extracted (1164), the extracted color channel value can be processed (1165), and a biometric index is obtained based thereon. (1166).

However, this will be addressed in detail in the relevant section below.

3.2 Biometric Information Acquisition Method FIG. 6 is a flow chart showing a biometric information acquisition method according to an embodiment.

Referring to FIG. 6, the biometric information acquisition method according to one embodiment may include acquiring a biometric index (S1210).
At this time, the bioindex is obtained by the bioindex obtaining method described above, but is not limited thereto, and may be obtained by an external sensor such as an ECG sensor.

Also, a repeated description of the biometric index will be omitted.

Also, the biometric information acquisition method according to one embodiment may include acquiring personal and statistical data (S1220).

At this time, the personal and statistical data may mean collectible personal and statistical data of the subject such as age and gender, observable personal and statistical data such as facial expressions and wrinkles of the subject. It may mean statistical data, but is not limited thereto, and can be various personal and statistical data other than biometrics for obtaining biometric information.

Also, the biometric information acquisition method according to one embodiment may include acquiring biometric information (S1230).

At this time, the biometric information may be drowsiness information, emotion information, etc. of the subject, but is not limited thereto.

Also, the biometric index can be used to obtain the biometric information. For example, heart rate among the biometric indices may be used to obtain the drowsiness information. More specifically, when the heart rate of the subject falls below the reference heart rate, it can be determined that the subject is in a drowsy state, and the degree of drowsiness of the subject according to the time when the heart rate falls below the reference heart rate. can be obtained.

Also, for example, heart rate and blood pressure among the biometric indices may be used to acquire the emotional information. More specifically, when the subject's heart rate and blood pressure are greater than or equal to the reference heart rate and blood pressure, it can be determined that the subject is in an excited state.

Also, the biometric index and the personal and statistical data can be used to obtain the biometric information. For example, to obtain the drowsiness information, personal and statistical data such as heart rate, age, and gender of the subject can be used.

Also, for example, personal and statistical data such as heart rate, blood pressure, facial expression, age, and gender of the subject can be used to obtain the emotional information.
In addition, the biometric index and the personal and statistical data may be weighted to obtain the biometric information. For example, different weights may be given to the biometric index and the personal and statistical data to obtain the biometric information, and different weights may be given to different subjects.

3.3 Bioindex Acquisition Method Using Bioindex Acquisition Model FIGS. 7 and 8 are diagrams for explaining the bioindex acquisition method using the bioindex acquisition model.

FIG. 7(a) is a diagram illustrating a biometric acquisition method (1300) using a biometric acquisition model (1302) according to one embodiment.
At this time, the biometric acquisition model 1302 according to one embodiment can be implemented by a machine learning method. For example, the biometric acquisition model 1302 may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, etc. be.

Also, the biometric acquisition model (1302) according to one embodiment can be implemented with an artificial neural network (ANN). For example, the biometric acquisition model (1302) is embodied by a feedforward neural network, a RBF network (radial basis function network), a kohonen self-organizing network, or the like. , but not limited to.

Also, the biometric acquisition model (1302) according to one embodiment can be implemented with a deep neural network (DNN). For example, the biometric acquisition model (1302) may be a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM) or GRUs (Gated Recurrent Units). etc., but is not limited thereto.

The image (1301) input to the biometric acquisition model (1302) is the acquired image data itself.

Also, the image (1302) input to the biometric acquisition model (1302) is preprocessed image data. For example, the image (1302) may be subject to various pre-processing such as, but not limited to, averaging acquired RGB values through Eulerian video magnification.

Also, the acquired biometric index (1303) can be heart rate, oxygen saturation, blood pressure, body temperature, and the like.

In addition, the acquired biometric index (1303) may be one, and multiple biometric indices can be acquired at the same time. For example, heart rate is obtained from the results of the biometric acquisition model (1302), but without limitation, heart rate and blood pressure are simultaneously obtained from the results of the biometric acquisition model (1302). can also

Also, FIG. 7(b) is a diagram showing a biometric acquisition method (1350) using a biometric acquisition model (1354) according to another embodiment.

At this time, the biometric acquisition model (1354) can acquire features (1352) extracted from the image (1351) and personal and statistical data (1353) as input values. For example, features of time-series data for color channel values are extracted from the image (1351), and the biometric acquisition model (1354) acquires time-series data for color channel values and personal and statistical data according to input values. It can be obtained and calculated from the Biometric Index (1355) results.

In addition, the personal and statistical data may mean personal and statistical data that can be collected from the person to be measured, such as age, gender, height, weight, etc., and the expression, wrinkles, complexion, etc. of the person to be measured can be observed. It may mean possible personal and statistical data, and may mean personal and statistical data that can be quantified, such as average blood pressure, average color, average height, and average weight.

In addition, since the biometric model (1354) may apply the content of the biometric model (1302) described above, the repeated description will be omitted.

Also, FIG. 8 is a diagram illustrating a biometric acquisition method (1400) using a biometric acquisition model (1405) according to another embodiment.

At this time, the biometric index acquisition method 1400 may include a feature extraction model 1402, and the feature extraction model 1402 according to one embodiment may be implemented by a machine learning method. For example, the feature extraction model (1402) may be a model embodied through supervised learning, but is not limited thereto, and may be a model embodied through unsupervised learning, semi-supervised learning, reinforcement learning, and the like.

Also, the feature extraction model (1402) according to one embodiment can be implemented with an artificial neural network (ANN). For example, the biometric acquisition model (1302) is embodied by a feedforward neural network, a RBF network (radial basis function network), a kohonen self-organizing network, or the like. , but not limited to.

Also, the feature extraction model (1402) according to one embodiment can be implemented in a deep neural network (DNN). For example, the biometric acquisition model (1302) may be a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM) or GRUs (Gated Recurrent Units). etc., but is not limited thereto.

In addition, the biometric acquisition model (1405) can acquire the features (1403) extracted from the feature extraction model (1402) and personal and statistical data (1404) from the input values. 1406) can be calculated from the results.

In addition, since the biometric model (1405) may apply the content of the biometric model (1302) described above, the repeated description will be omitted.

Machine running, artificial neural network or deep neural network models can be utilized to obtain the biometrics along with the examples described with reference to FIGS.

4. Various Embodiments of Bioindex Acquisition
4.1 Various Embodiments of Heart Rate Measurement Methods In the body of a living organism, when the heart beats, blood can be transported throughout the body by the beating of the heart. At this time, the blood flows through the blood vessel, and the volume of the blood vessel changes with time, and the amount of blood contained in the blood vessel can change.

Thus, when measuring changes in vessel volume or changes in blood volume, heart rate can be obtained. For example, when the amount of blood contained in a blood vessel changes, the amount of hemoglobin and oxyhemoglobin contained in the blood changes, and accordingly the amount of light reflected by the blood can change. Thus, when measuring changes in the amount of light reflected by blood in this way, heart rate can be obtained.

Also, it is obvious that various principles for measuring heart rate by light can be applied in addition to the exemplary principles described above.

FIG. 9 is a flowchart for explaining a heart rate measurement method according to one embodiment.

Referring to FIG. 9, a heart rate measuring method (1500) according to an embodiment includes acquiring an image (S1510) for at least one image frame among a plurality of acquired image frames; (S1520), detecting a region of interest (S1530), and processing data for the region of interest (S1540).

Also, the step of acquiring the image (S1510), the step of detecting the skin area (S1520), and the step of detecting the region of interest (S1530) will be omitted as described above.

Also, the step of processing data for the region of interest (S1540) may be performed for at least one image frame among the plurality of acquired image frames.

Also, color channel values for the region of interest may be extracted for at least one image frame among a plurality of image frames acquired to process data for the region of interest. At this time, the color channel value may be an average value of color channel values of pixels included in the region of interest, and may be referred to as an average pixel value.

Also, the detailed contents of the step of processing data for the region of interest (S1540) will be omitted as described above.

In addition, the heart rate measurement method (1500) according to an embodiment includes extracting time-series data for at least some image frame groups among the plurality of acquired image frames (S1550); including, but not limited to, at least a portion of the step (S1560).

Also, the step of extracting the time-series data (S1550) may be performed on at least a partial image frame group among the plurality of acquired image frames.

At this time, the image frame group may refer to a plurality of continuous or discontinuous image frame groups. For example, the image frame group may mean a group of consecutive image frames from the 1st image frame to the 180th image frame, but is not limited thereto, and the image frames of the 1st image frame to the 180th image frame. It can also mean a group of at least some of the image frames.

Also, the step of acquiring the heart rate (S1560) may be performed for at least a partial image frame group among the plurality of acquired image frames.

At this time, frequency components of the obtained time-series data can be extracted to obtain the heart rate. For example, to obtain the heart rate, the time-series data is transformed by a Fourier transform (FT) to extract frequency components, but is not limited thereto, and the time-series data is a fast Fourier transform (Fast fourier transform, FFT), discrete fourier transform (DFT), STFT (short time fourier transfom), etc., but not limited to this, there are various processes for extracting frequency components. can be done.

Also, the heart rate may be obtained based on one heart rate, or obtained based on at least two heart rates. For example, one heart rate can be obtained based on one time-series data, and another heart rate can be obtained based on another time-series data, and at least two obtained A final heart rate may be obtained based on the above heart rates, but is not limited thereto.

4.2 Various Examples of Oxygen Saturation Measurement Method Oxygen saturation (SPO2) means the amount of oxygen bound to hemoglobin, and can be expressed as the ratio of oxyhemoglobin to total hemoglobin in blood.

Also, hemoglobin and oxyhemoglobin can have the same or different absorptances for light having one wavelength. For example, hemoglobin and oxyhemoglobin may have different absorbances for light in the 700 nm band, different absorbances for light in the 1000 nm band, and similar absorbances for light in the 800 nm band.

Also, when the amount of blood contained in blood vessels changes, the amounts of hemoglobin and oxyhemoglobin contained in the blood may change.

Therefore, the absorption coefficient of hemoglobin for light in the first wavelength band, the absorption coefficient of oxyhemoglobin, the absorption coefficient of hemoglobin for light in the second wavelength band, the absorption coefficient of oxyhemoglobin, and the absorption coefficient of light in the first wavelength band due to changes in blood volume Oxygen saturation can be obtained by using the degree of change and the degree of change of light in the second wavelength band due to the change in blood volume.

酸素飽和度はS*100(%)と示されるが、これに対し限定されない。 For example, ε 1 is the absorbance of oxyhemoglobin for light in the first wavelength band, ε ₂ is the absorbance of hemoglobin for light in the first wavelength band, ε ₃ is the absorbance of oxyhemoglobin for light in the second wavelength band, and ε ₃ is the absorbance of oxyhemoglobin for light in the second wavelength band. When ε ₄ is the absorbance of hemoglobin with respect to the light of , and S is the ratio of oxygen hemoglobin, the following mathematical formula 1 holds,
<Mathematical formula 1>

Oxygen saturation is shown as S*100(%), but is not limited thereto.

It is also obvious that various principles for measuring oxygen saturation with light can be applied in addition to the exemplary principles described above.

FIG. 10 is a flow chart for explaining an oxygen saturation measuring method according to one embodiment.

Referring to FIG. 10, the oxygen saturation measurement method (1600) according to an embodiment includes acquiring an image (S1610) for at least one image frame among a plurality of acquired image frames; detecting a region (S1620); detecting a region of interest (S1630); and processing at least two color channel values for the region of interest (S1640). is not limited to

Also, the step of acquiring the image (S1610), the step of detecting the skin area (S1620), and the step of detecting the region of interest (S1630) will be omitted as described above.

Also, the step of processing at least two color channel values for the region of interest (S1640) can perform the operation of the above-described step of processing data for the region of interest, so a repeated description will be omitted.

Also, the two color channels can be selected in consideration of the absorbance of hemoglobin and the absorbance of oxyhemoglobin. For example, when using the RGB color space, a Red channel in which the absorbance of hemoglobin is higher than that of oxyhemoglobin and a Blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin are selected, but not limited thereto.

In addition, the oxygen saturation measurement method (1600) according to an embodiment includes extracting time-series data for at least two color channel values for at least a partial image frame group among the acquired plurality of image frames ( S1650), and at least part during the step of obtaining oxygen saturation (S1660), but is not limited thereto.

Also, the step of extracting time-series data for the at least two color channel values (S1650) may be performed on at least some of the acquired plurality of image frames.

At this time, the image frame group may refer to a plurality of continuous or discontinuous image frame groups. For example, the image frame group may mean a group of consecutive image frames from the 1st image frame to the 180th image frame, but is not limited thereto. It can also mean a group of at least some of the image frames.

Also, the step of obtaining the oxygen saturation (S1660) may be performed for at least a partial image frame group among the plurality of obtained image frames.

At this time, AC and DC components of the obtained at least two time-series data can be used to obtain the oxygen saturation. At this time, the AC component may mean the difference between the maximum value and the minimum value of the time-series data, or the difference between the average maximum value and the average minimum value, but is not limited to this. can be understood as the AC component. Also, the DC component is understood as an average value of time-series data, but is not limited to this and can be understood as a normal DC component.

Also, a mathematical formula can be utilized to obtain the oxygen saturation.

For example, ε ₁ is the absorbance of oxyhemoglobin to the Red channel, ε ₂ is the absorbance of hemoglobin to the Red channel, ε ₃ is the absorbance of oxyhemoglobin to the Blue channel, ε ₄ is the absorbance of hemoglobin to the Blue channel, and S is the ratio of oxyhemoglobin. When , the following formula 2 holds.

この時、酸素飽和度はS*100(%)としめされるが、これに対し限定されない。 <Mathematical formula 2>

At this time, the oxygen saturation is shown as S*100(%), but is not limited thereto.

Also, it should be appreciated that various mathematical formulas for determining oxygen saturation using at least two color channel values may be utilized in addition to the exemplary mathematical formulas described above.

Also, the oxygen saturation can be obtained based on one oxygen saturation, or obtained based on at least two oxygen saturations. For example, one oxygen saturation can be obtained based on at least two time-series data, and another oxygen saturation can be obtained based on at least two other time-series data. The final oxygen saturation may be obtained based on at least two oxygen saturations, but is not limited thereto.

Also, the final obtained oxygen saturation and an oxymeter can be utilized to obtain a more accurate oxygen saturation.

FIG. 11 is a flow chart for explaining an oxygen saturation measuring method according to another embodiment.

Referring to FIG. 11, a method for measuring oxygen saturation (1700) according to an embodiment includes acquiring an image for at least one image frame among a plurality of acquired image frames (S1710); detecting a region (S1720); detecting a region of interest (S1730); and processing at least one color channel value for the region of interest (S1740). is not limited to

In addition, the oxygen saturation measurement method (1700) according to one embodiment can acquire a plurality of infrared (IR) image frames, and for at least one IR image frame among the acquired plurality of IR image frames, an IR At least of acquiring an image (S1711), detecting a skin region (S1721), detecting a region of interest (S1731), and processing IR data for the region of interest (S1741) Including but not limited to.

Further, the steps of acquiring the image (S1710, S1711), the steps of detecting the skin area (S1720, S1721), and the steps of detecting the region of interest (S1730, S1731) will not be described repeatedly. I decide to

Also, processing at least one color channel value for the region of interest (S1740) and processing IR data for the region of interest (S1741) are operations of processing data for the region of interest as described above. can be executed, so a repeated description will be omitted.

Also, the at least one color channel and the IR wavelength band may be selected in consideration of the absorbance of hemoglobin and the absorbance of oxyhemoglobin. For example, when using the RGB color space, the Red channel, in which the absorbance of hemoglobin is higher than that of oxyhemoglobin, can be selected, and the IR wavelength band of 880 nm, in which the absorbance of oxyhemoglobin is higher than that of hemoglobin, is selected. but not limited thereto.

Also, the oxygen saturation measuring method (1700) according to an embodiment includes extracting time-series data for color channel values for at least some image frame groups among the plurality of acquired image frames (S1750). , extracting time-series data for IR data (S1751), and acquiring oxygen saturation (S1760), but not limited thereto.

In addition, since the step of extracting time-series data for the color channel value (S1750) and the step of extracting time-series data for IR data (S1751) are performed by the operation of the step of extracting time-series data described above, A repeated explanation will be omitted.

In addition, the step of acquiring oxygen saturation (S1760) includes at least a partial image frame group among the plurality of acquired image frames and at least a portion of the acquired plurality of IR image frames. It can be performed on IR image frames.

At this time, at least a partial image frame group among the plurality of acquired image frames and at least a partial IR image frame group among the plurality of acquired IR image frames may be identical to each other. Well, it can be different.

For example, when the image frames and the IR image frames are obtained from different sequences, the image frames and the IR image frames are different from each other, and the image frames and the IR image frames are obtained from the same sequence, The image frame group and the IR image frame group may be identical to each other, but are not limited thereto.

Also, a mathematical formula can be utilized to obtain the oxygen saturation.

For example, the absorbance of oxyhemoglobin to the Red channel is ε ₁ , the absorbance of hemoglobin to the Red channel is ε ₂ , the absorbance of oxyhemoglobin at 880 nm is ε ₃ , the absorbance of hemoglobin at 880 nm is ε ₄ , and the ratio of oxyhemoglobin is S. Then, the following mathematical formula 3 can be established.

<Mathematical formula 3>

At this time, the oxygen saturation is indicated by S*100(%), but is not limited thereto.

Also, it is obvious that various mathematical formulas for determining oxygen saturation using at least one color null value and IR data can be used in addition to the exemplary mathematical formulas described above.

Also, the oxygen saturation can be obtained based on one oxygen saturation, or obtained based on at least two oxygen saturations. For example, one oxygen saturation can be obtained based on at least two time-series data, and another oxygen saturation can be obtained based on at least two other time-series data. The final oxygen saturation may be obtained based on at least two oxygen saturations, but is not limited thereto.

Also, the final obtained oxygen saturation and an oxymeter can be utilized to obtain a more accurate oxygen saturation.

4.3 Various Embodiments of Blood Pressure Measurement Methods Blood pressure may refer to the pressure that blood flowing about a vessel exerts on the walls of the vessel.

Therefore, blood pressure can be affected by blood flow velocity, vessel wall thickness, waste products accumulated in the vessel, and the like.

In addition, when blood flows through a blood vessel, the volume of the blood vessel changes with time, and the amount of blood contained in the blood vessel can change.

Therefore, blood pressure can be acquired when measuring the rate of change in blood vessel volume and the rate of change in blood volume.

For example, if vascular changes are measured at two points at different distances from the heart, blood pressure can be obtained based on the difference in vascular changes at the two points, indicating changes in blood vessels over time. The blood pressure can be obtained by extracting the features that can be obtained, but is not limited to this.

Also, it is obvious that various principles for measuring blood pressure by light can be applied in addition to the exemplary principles described above.

FIG. 12 is a flow chart for explaining a blood pressure measurement method according to one embodiment.

Referring to FIG. 12, a blood pressure measurement method (1800) according to an embodiment includes acquiring an image (S1810) for at least one image frame among a plurality of acquired image frames, and measuring a skin region. detecting (S1820), detecting a first region of interest (S1830), detecting a second region of interest (S1831), processing data for the first region of interest (S1840), and and processing data for the second region of interest (S1841), but is not limited thereto.

In addition, the step of acquiring the image (S1810) and the step of detecting the skin area (S1820) will be omitted as described above.

Also, the detailed contents of the step of detecting the first ROI and the second ROI (S1830, S1831) will be omitted as described above.

At this time, the first region of interest and the second region of interest can be set as two regions having different distances from the heart of the subject. For example, the first region of interest is set to the upper region of the subject's face, and the second region of interest is set to the lower region of the subject's face, but without limitation, the first region of interest is The second region of interest may be set to the subject's face region and the subject's back hand region.

Also, the step of processing data for the first region of interest (S1840) and the step of processing data for the second region of interest (S1841) may perform the operation of the above-described step of processing data for the region of interest. Therefore, we will omit repeated explanations.

In addition, the blood pressure measurement method (1800) according to an embodiment includes extracting time-series data for the first and second regions of interest for at least some of the acquired image frames (S1850). ), calculating PTT (Pulse transit Time) based on the acquired time-series data (S1860), and acquiring blood pressure (S1870). not.

Also, the step of extracting time-series data for the first and second regions of interest (S1850) may perform the operation of the step of extracting time-series data described above, so a repeated description will be omitted. do.

Also, the step of calculating the PTT based on the acquired time-series data (S1860) may be performed for at least some of the acquired plurality of image frames.

At this time, the PTT can be calculated based on extreme values of the time-series data for the first ROI and the time-series data for the second ROI. For example, the PTT is calculated based on the time difference between the maximum value of the time-series data for the first region of interest and the maximum value of the time-series data for the second region of interest, but is not limited thereto. The PTT can also be calculated based on the time difference between the minimum value of the time-series data for the region and the minimum value of the time-series data for the second region of interest.

Also, the PTT may be calculated based on inflection points of the time-series data for the first region of interest and the time-series data for the second region of interest. For example, the PTT is calculated based on the time difference between the inflection point of the time-series data for the first region of interest and the inflection point of the time-series data for the second region of interest, but is not limited thereto.

In addition, the PTT can be calculated based on various points of the time-series data for each region of interest in addition to the extremum and inflection point.

In addition, it can be calculated based on frames acquired at points such as extreme values, inflection points, etc., which are time differences between the time-series data for the first region of interest and the time-series data for the second region of interest. For example, when the time-series data for the first region of interest obtains a maximum value in the 10th frame, and the time-series data for the second region of interest obtains a maximum value in the 12th frame, the first region of interest and the second region of interest It may be the time for two frames to be acquired, which is the time difference of the time series data for the region of interest, on which the PTT can be calculated.

Also, the step of obtaining blood pressure (S1870) may be performed for at least a partial image frame group among the plurality of obtained image frames.

Also, PTT can be utilized to obtain the blood pressure. For example, a function for PTT can be used to obtain the blood pressure. More specifically, a function such as Equation 4 is used, and the function uses various functions such as a linear function, a quadratic function, a log function, and an exponential function with PTT as a variable, but is not limited thereto. .

<Mathematical formula 4>

Also, PTT and personal, statistical data can be used to obtain the blood pressure. For example, a function for PTT and a function of personal and statistical data such as age, weight and height can be used to obtain the blood pressure. More specifically, the following formula 5 is used, and various functions such as PTT, weight, height, and age are used as variables, such as linear function, quadratic function, log function, and exponential function. , but not limited to.

<Mathematical formula 5>

In addition, a regression analysis method using the functions described above is used to obtain the blood pressure, but the present invention is not limited thereto.

In addition, a machine learning method using the above functions is used to obtain the blood pressure, but the present invention is not limited thereto.

Also, it is obvious that various mathematical formulas for obtaining blood pressure using PTT can be used in addition to the exemplary mathematical formulas described above.

Also, the blood pressure may be obtained based on one blood pressure, or obtained based on at least two blood pressures. For example, one blood pressure is obtained based on the first PTT calculated based on the time-series data for the first and second regions of interest, and another blood pressure is calculated based on the time-series data for the first and second regions of interest. Another blood pressure may be obtained based on the second PTT, and a final blood pressure may be obtained based on at least two or more obtained blood pressures, but is not limited thereto.

FIG. 13 is a flow chart for explaining a blood pressure measurement method according to another embodiment.

Referring to FIG. 13, a blood pressure measurement method (1900) according to an embodiment includes acquiring an image (S1910) for at least one image frame among a plurality of acquired image frames; At least part of detecting (S1920), detecting a region of interest (S1930), and processing data for the region of interest (S1940), but is not limited thereto.

At this time, details of acquiring the image (S1910), detecting the skin area (S1920), detecting the region of interest (S1930), and processing data for the region of interest (S1940). As described above, repeated explanation of the contents will be omitted.

In addition, a blood pressure measurement method (1900) according to an embodiment includes steps of extracting time-series data for at least a part of a group of image frames among a plurality of acquired image frames (S1950); At least some of the step of extracting features based on data (S1960) and the step of obtaining blood pressure (S1970) are included, but are not limited thereto.

In addition, the step of extracting time-series data (S1950) performs the operation of the above-described step of extracting time-series data, so a repeated description will be omitted.

Also, the step of extracting features based on the acquired time-series data (S1960) may be performed on at least some of the acquired plurality of image frames.

At this time, the features may mean mathematical and physical features of the acquired time-series data. For example, the features are local maximum, average local maximum, local minimum, average local minimum, difference between maximum and minimum, average, point of inflection, primary differential data, secondary differential It may mean mathematical features such as data, slope at a specific time, etc. It may mean physical features such as blood change amount, blood change rate, blood vessel change amount, blood vessel change rate, etc. is not limited to

Also, it is self-evident that the features can be various features for acquiring blood pressure in addition to the exemplary features described above.

Also, the step of obtaining the blood pressure (S1970) may be performed for at least a partial image frame group among the plurality of obtained image frames.

Also, the feature can be used to obtain the blood pressure. For example, a function on the features can be used to obtain the blood pressure. More specifically, a function such as Equation 6 is used, and the function uses various functions such as a linear function, a quadratic function, a log function, an exponential function, etc., but is not limited thereto.

<Mathematical formula 6>

Also, the characteristics and personal, statistical data can be used to obtain the blood pressure. For example, to obtain the blood pressure, it is possible to use a function on the features and a function of personal, statistical data such as age, weight, and height. More specifically, the following mathematical formula 7 is used, and the functions use various functions such as linear functions, quadratic functions, log functions, exponential functions with characteristics, weight, height, and age as variables. , but not limited to.

<Mathematical formula 7>

In addition, a regression analysis method using the functions described above is used to obtain the blood pressure, but the present invention is not limited thereto.

In addition, a machine learning method using the above functions is used to obtain the blood pressure, but the present invention is not limited thereto.

Also, it is self-evident that mathematical formulas for determining blood pressure using features can be used in addition to the exemplary mathematical formulas described above.

Also, the blood pressure may be obtained based on one blood pressure, or obtained based on at least two blood pressures. For example, one blood pressure is obtained based on a first feature calculated based on time-series data, and another blood pressure is obtained based on a second feature calculated based on other time-series data. A final blood pressure is obtained based on the obtained at least two or more blood pressures, but is not limited thereto.

4.4 Various Embodiments of Body Temperature Measurement Method FIG. 14 is a flowchart illustrating a body temperature measurement method according to one embodiment.

Referring to FIG. 14, the method for measuring body temperature (2000) according to one embodiment includes at least part of acquiring skin temperature (S2010) and acquiring body temperature (S2020). is not limited to

Also, the step of obtaining the skin temperature (S2010) may be performed in a non-contact manner.

For example, an image sensor such as a camera can be used to acquire the skin temperature. More specifically, at least one or more color channel values of an image acquired from an image sensor such as a camera can be used to acquire the skin temperature. Exemplarily, when the HSV color space is used, the skin temperature is obtained using a saturation (S) value, but is not limited thereto.

Also, for example, a sensor such as a thermal image camera is used to acquire the skin temperature, and an image sensor such as an infrared camera is used, but the invention is not limited thereto.

Also, the step of obtaining the body temperature (S2020) may be performed in a non-contact manner.

For example, skin temperature can be used to obtain the body temperature. More specifically, the body temperature can be obtained based on the skin temperature using the relationship between the site where the skin temperature is measured and the body temperature.
At this time, the skin temperature is obtained from the step of obtaining the skin temperature (S2010), and may be obtained by another external sensor.

Also, for example, an image sensor such as a camera can be used to acquire the body temperature. More specifically, an image obtained from an image sensor such as a camera can be used to obtain the skin temperature. For example, the body temperature may be obtained by using the image obtained from the image sensor such as the camera as a body temperature measurement machine running model, but the embodiment is not limited thereto.

Hereinafter, a method for obtaining skin temperature and a method for obtaining body temperature will be described in detail.

First, a method of obtaining skin temperature using an image sensor such as a camera will be described.

According to one embodiment, the brightness value of the captured image can be used to obtain the skin temperature. For example, a function on brightness values can be used to obtain the skin temperature. More specifically, the following formula 8 is used, and various functions such as linear functions, quadratic functions, log functions, and exponential functions with brightness values as variables are used. not.

<Formula 8>

Also, according to one embodiment, the brightness value of the acquired image and personal, statistical data can be used to acquire the skin temperature. For example, to obtain the skin temperature, it is possible to use a function of brightness value and a function of personal and statistical data such as age, race, and gender. More specifically, the following mathematical formula 9 is used, and various functions such as linear function, quadratic function, log function, exponential function with brightness value, age, race, and gender as variables are used. but not limited thereto.

<Mathematical formula 9>

Also, according to one embodiment, the brightness and color values of the captured image can be used to obtain the skin temperature. For example, a function for lightness values and a function for color values can be used to obtain the skin temperature. More specifically, the following mathematical formula 10 is used, and various functions such as linear functions, quadratic functions, log functions, exponential functions, etc. are used with lightness values and color values as variables. is not limited to

<Mathematical formula 10>

Also, according to one embodiment, lightness values, color values, and personal and statistical data of the captured image can be used to obtain skin temperature. For example, in order to obtain the skin temperature, it is possible to use functions for brightness values, functions for color values, and functions for personal and statistical data such as age, race, and gender. More specifically, the following mathematical formula 11 is used, and various functions such as linear functions, quadratic functions, log functions, exponential functions, etc. are used with brightness value, color value, age, race, and gender as variables. Uses, but is not limited to, functions.

<Mathematical formula 11>

Also, according to one embodiment, the brightness value, the color value, and the change value of the brightness value of the captured image can be used to obtain the skin temperature. For example, a function for brightness value, a function for color value, and a function for change value of brightness value can be used to obtain the skin temperature. More specifically, the following formula 12 is used, and the function is a linear function, a quadratic function, a log function, an exponential function, etc. with the variable of the brightness value, the color value, and the change value of the brightness value. is used, but is not limited thereto. Also, when using the change value in this way, it is possible to reduce the measurement noise caused by the external environment.

<Formula 12>

Also, according to one embodiment, the brightness value, color value, change value of brightness value, and personal and statistical data of the acquired image can be used to acquire the skin temperature. For example, in order to obtain the skin temperature, it is possible to use a function of brightness value, a function of color value, a function of change value of brightness value, and a function of personal and statistical data such as age, race, and sex. More specifically, the following formula 13 is used, and the function is a linear function, a quadratic function, and a log function with variables of brightness value, color value, change value of brightness value, age, race, and gender. , an exponential function, but not limited thereto.

<Mathematical formula 13>

Also, according to one embodiment, the brightness value, color value, change value of brightness value, and change value of color value of the captured image may be used to obtain the skin temperature. For example, to obtain the skin temperature, it is possible to use a function for lightness value, a function for color value, a function for change value of lightness value, and a function for change value of color value. More specifically, the following mathematical formula 14 is used, and the functions are a linear function, a quadratic function, a log function, and Various functions such as an exponential function are used, but not limited thereto. Also, when using the change value in this way, it is possible to reduce the measurement noise caused by the external environment.

<Formula 14>

Also, according to one embodiment, the brightness value, color value, change value of brightness value, change value of color value, and personal and statistical data of the acquired image are used to obtain the skin temperature. can be done. For example, in order to obtain the skin temperature, a function for the lightness value, a function for the color value, a function for the change value of the lightness value, a function for the change value of the color value, and personal and statistical data such as age, race, and gender. functions can be used. More specifically, the following mathematical formula 15 is used, and the function is a linear function with variables of lightness value, color value, change value of lightness value, change value of color value, age, race, and sex, Various functions such as quadratic function, log function, and exponential function are used, but not limited thereto.

<Formula 15>

Also, a regression analysis method using the above-described function is used to obtain the skin temperature, but the method is not limited thereto.

In addition, a machine learning method using the above functions or a deep learning method may be used to obtain the skin temperature, but the invention is not limited thereto.

Also, it is obvious that various mathematical formulas can be used in addition to the exemplary mathematical formulas described above. For example, a mathematical formula using at least a part of the above-described lightness value, color value, lightness value change value, color value change value, age, race, gender, and other personal and statistical data is used. Mathematical formulas using data other than are also available.

Next, let us consider a method of acquiring body temperature using an image sensor such as a camera.

According to one embodiment, the obtained skin temperature can be used to obtain body temperature. For example, a function for skin temperature can be used to obtain the body temperature. More specifically, the following formula 16 is used, and the function can be a linear function, a quadratic function, a log function, an exponential function, etc. with the skin temperature as a variable. Not limited.

<Formula 16>

In addition, according to one embodiment, when acquiring body temperature indoors, the acquired skin temperature and room temperature can be used to acquire the body temperature. For example, a function for skin temperature and a function for room temperature can be used to obtain the body temperature. More specifically, the following formula 17 is used, and the function can be various functions such as a linear function, a quadratic function, a log function, and an exponential function with skin temperature and room temperature as variables. , but not limited to.

<Formula 17>

In addition, according to one embodiment, when acquiring body temperature indoors, the acquired skin temperature, room temperature, and heart rate may be used to acquire body temperature. For example, a function for skin temperature, a function for room temperature, and a function for heart rate can be used to obtain the body temperature. More specifically, the following formula 18 is used, and the function can be a variety of functions such as a linear function, a quadratic function, a log function, and an exponential function with skin temperature, room temperature, and heart rate as variables. can be, but is not limited to.

<Formula 18>

Also, according to one embodiment, various skin temperatures can be used simultaneously. For example, when acquiring the body temperature indoors, the first skin temperature for the first body part, the second skin temperature for the second body part, the room temperature, and the heart rate that are acquired to acquire the body temperature may be used. can. For example, a function for a first skin temperature, a function for a second skin temperature, a function for room temperature, and a function for heart rate can be used to obtain the body temperature. More specifically, the following formula 19 is used, and the function is a linear function, a quadratic function, a log function, and an exponential function with the first skin temperature, the second skin temperature, the room temperature, and the heart rate as variables. It can be a variety of functions such as, but is not limited to.

<Formula 19>

In addition, a regression analysis method using the functions described above is used to obtain the body temperature, but the method is not limited thereto.

In addition, a machine learning method using the above functions or a deep learning method may be used to obtain the body temperature, but the present invention is not limited thereto.

Also, it is obvious that various mathematical formulas can be used in addition to the exemplary mathematical formulas described above. For example, a mathematical formula using at least part of the above-mentioned first skin temperature, second skin temperature, room temperature, and heart rate is used. A mathematical formula using statistical data can also be used.

5. Various Embodiments of Heart Rate Measurement Method FIG. 15 is a flow chart illustrating a heart rate acquisition method according to one embodiment.

Referring to FIG. 15, a heart rate acquisition method (2100) according to one embodiment includes acquiring an image (S2110), detecting a skin area (S2120), and detecting a region of interest (S2130). and processing data for the region of interest (S2140), acquiring characteristic values (S2150), and acquiring heart rate (S2160), but not limited thereto. not.

At this time, the step of acquiring the image (S2110), the step of detecting the skin area (S2120), and the step of detecting the region of interest (S2130) will be omitted as described above.

Also, the step of processing data for the region of interest (S2140) may be performed for at least one image frame among the plurality of acquired image frames.

Also, the step of processing data for the region of interest (S2140) may be performed to reduce motion artifacts, noise caused by external light, and the like.

Also, the step of acquiring the characteristic value (S2150) may be performed for at least some of the acquired plurality of image frames.

Also, the step of obtaining the characteristic value (S2150) may be performed to reduce noise caused by motion, noise caused by external light, and the like.

Also, the step of obtaining the heart rate (S2160) may be performed on at least a partial image frame group among the plurality of obtained image frames.

At this time, the image frame group for obtaining the heart rate and the image frame group for obtaining the characteristic value may be the same or different. For example, the group of image frames for obtaining the characteristic value may include 18 image frames, and the group of image frames for obtaining the heart rate may include 180 image frames, but are limited thereto. not.

Hereinafter, the steps of processing data for the region of interest (S2140), obtaining characteristic values (S2150), and obtaining heart rate (S2160) will be described in more detail.

FIG. 16 is a graph for color channel values according to one embodiment.

According to one embodiment, color channel values for a region of interest can be extracted for processing data for the region of interest. At this time, the color channel value may be an average value of color channel values of pixels included in the region of interest, and may also be referred to as an average pixel value.

Referring to FIG. 16, color channel values according to the RGB color space for the region of interest can be extracted to process the data for the region of interest. More specifically, a Red channel value that is the average value of the pixel values in the Red channel, a Blue channel value that is the average value of the pixel values in the Blue channel, and a Green channel value that is the average value of the pixel values in the Green channel are extracted. can

For example, when a color channel value is extracted in an RGB color space, a Red channel pixel value, a Green channel pixel value, and a Blue channel pixel value of each pixel included in the region of interest are extracted. A Red channel value that is the average of the Red channel pixel values, a Blue channel value that is the average of the Blue channel pixel values, and a Green channel value that is the average of the Green channel pixel values are extracted. However, color channel values in various color spaces such as HSV, YCrCb color spaces can be extracted.

Also, the color channel values extracted by a particular color space can be converted to another color space. For example, color channel values extracted in RGB color space can be converted into color channel values in various color spaces such as HSV, YCrCb color space.

Also, the color channel values extracted for processing the data for the region of interest are color channel values obtained by applying a weighting value to at least some of the color channel values extracted from various color spaces and combining them. .

Also, the color channel values are extracted for each of a plurality of image frames that are successively acquired, and may be extracted for at least some of the image frames.

Although the following description is based on the color channel values extracted from the RGB color space, it is obvious that the color channel values are not limited to this and various color channel values can be applied.

In FIG. 16, (a) is a graph showing Red channel values extracted from the RGB color space, (b) is a graph showing Green channel values extracted from the RGB color space, and (c) is a graph showing the RGB color space. 3 is a graph showing Blue channel values extracted by .

As shown in FIG. 16, each color channel value may experience variations in value due to heart beats.

However, each color channel value may fluctuate due to movement of the person to be measured and changes in the intensity of external light, as well as fluctuations due to the beating of the heart.

Therefore, in order to obtain the heart rate using the color channel values extracted in this way, the value fluctuation due to the movement of the subject and the change in the intensity of the external light is reduced, and the value fluctuation due to the heart beat is reduced. Sometimes it is necessary to act to maximize

5.1 Various Embodiments of Method for Processing Data on Region of Interest It is self-evident that the above description applies to a method for processing data on a region of interest, and redundant descriptions will be omitted.

FIG. 17 is a graph for explaining a noise reduction method according to one embodiment.

FIG. 17(a) is a graph showing Red channel values extracted from the RGB color space, and (b) is a graph showing Green channel values extracted from the RGB color space.

Referring to (a) and (b) of FIG. 17, it can be seen that the extracted color channel values fluctuate over time.

At this time, the extracted color channel values may fluctuate due to the heartbeat, movement of the subject, and changes in the intensity of external light.

More specifically, large and slow fluctuations in color channel values are affected more by the movement of the person being measured and changes in the intensity of external light, and small and rapid fluctuations occur. In other words, fluctuations can occur due to the influence of the heartbeat of the person being measured.

Therefore, the relative difference between the values of at least two color channels is calculated to reduce variations in values due to subject movement and changes in external light intensity, which are greater than variations due to heart beat. can be used.

Illustratively, the difference between the Green and Red channel values can be used to reduce noise. More specifically, the Green channel value and the Red channel value obtained from the same image frame can reflect the same motion and the same external light intensity, and the difference value between the Green channel value and the Red channel value of the same frame is , to reduce noise due to, but not limited to, subject movement and changes in external light intensity, etc. The relative difference between at least two color channel values can be used to reduce noise. .

(c) of FIG. 17 is a graph showing the difference value between the Green channel value and the Red channel value.

As shown in (c) of FIG. 17, the difference value between the Green channel value and the Red channel value can reduce noise due to movement of the subject and changes in intensity of external light.

Also, the method of reducing noise described above is performed on at least one image frame out of a plurality of acquired image frames, and may be performed on each of a plurality of successive image frames.

Also, although not shown in (c) of FIG. 17, noise can be reduced using the difference value between the Green channel value and the Blue channel value, and the difference between the Red channel value and the Blue channel value can be reduced. Difference values can also be used to reduce noise.

Also, at least two color channel values can be selected to determine the difference value to reduce noise by utilizing the relative difference between the at least two color channel values as described above.

At this time, the at least two color channel values can be selected considering the absorbance of blood.

FIG. 18 is a diagram showing the absorbance of hemoglobin and oxyhemoglobin in the visible light band.

According to one embodiment, the Red channel may include at least a portion of the 620 nm to 750 nm wavelength band, and the Green channel may include at least a portion of the 495 nm to 570 nm wavelength band. , the Blue channel may be, but is not limited to, a channel that includes at least a portion in the 450 nm to 495 nm wavelength band, and the Red channel, Green channel, and Blue channel are each commonly understood color channels. .

Referring to FIG. 18, the absorbance of hemoglobin and oxyhemoglobin according to the wavelength band of light can be seen. For example, as shown in FIG. 18, the absorbance of hemoglobin and oxyhemoglobin to light in the 550 nm wavelength band contained in the Green channel is higher than the absorbance of hemoglobin and oxyhemoglobin to light in the 650 nm wavelength band contained in the Red channel. There is also

Also, for example, as shown in FIG. 18, the absorbance of hemoglobin and oxyhemoglobin to light in the 550 nm wavelength band contained in the Green channel is higher than the absorbance of hemoglobin and oxyhemoglobin to light in the 470 nm wavelength band contained in the Blue channel. Sometimes it becomes

In addition, when blood is transported throughout the body by heartbeat, the blood flow can change the volume of the blood vessels and the amount of blood contained in the blood vessels.

Therefore, the color channel value including the wavelength band of light that is relatively more absorbed by hemoglobin contained in blood and oxygenated hemoglobin can fluctuate relatively more depending on changes in blood volume due to heartbeat.

On the contrary, the color channel values, which include wavelength bands of light that are relatively less absorbed by hemoglobin in blood and oxyhemoglobin, can fluctuate less due to changes in blood volume due to heart beats.

Therefore, according to one embodiment, at least two color channels can be selected for noise reduction considering the absorbance of hemoglobin and oxyhemoglobin.

For example, according to one embodiment, a difference value between a Green channel value with relatively high absorption due to hemoglobin and oxyhemoglobin to reduce noise and a Red channel value with relatively low absorption due to hemoglobin and oxyhemoglobin. can be used.

Also, for example, according to one embodiment, in order to reduce noise, a Green channel value in which absorption is relatively increased by hemoglobin and oxyhemoglobin, and a Blue channel value in which absorption by hemoglobin and oxyhemoglobin is relatively decreased. A difference value can be used.

Also, for example, according to one embodiment, in order to reduce noise, a Blue channel value in which absorption is relatively increased by hemoglobin and oxyhemoglobin, and a Red channel value in which absorption is relatively decreased by hemoglobin and oxyhemoglobin. A difference value can be used.

Also for example, according to one embodiment, the difference value between the Green channel value and the Red channel value and the difference value between the Green channel value and the Blue channel value can be used simultaneously.

In addition, although the above examples are explained based on the difference value, in order to reduce noise caused by the movement of the subject and noise caused by external light, a weighted value is used for each channel value in addition to the difference value. A processed value processed by

For example, a processed value processed using a weighted value for each channel value can be used as in Equation 20 below.

<Formula 20>
Processing value = a Red channel value + b Blue channel value + c Green channel value

At this time, each a, b, c value is determined so that a+b+c=0 in order to efficiently remove noise due to the movement of the subject and noise due to external light. Since each channel value contains similar motion noise and external light noise in one image frame, it is advantageous to effectively reduce noise.

5.2 Various Embodiments of Characteristic Value Acquisition Characteristic values can be acquired in order to reduce noise caused by the movement of the subject and noise caused by the intensity of external light.

At this time, the characteristic value may be acquired for at least a partial image frame group among the plurality of acquired image frames.

Also, the characteristic value may be a value representing a characteristic of the acquired color channel value or processing value. For example, the characteristic value may mean, but is not limited to, an average value, a deviation value, a standard deviation value, etc. of color channel values or processed values included in an image frame group.

FIG. 19 is a diagram for explaining a characteristic value acquisition method according to an embodiment.

FIG. 19(a) is a graph showing color channel values obtained according to one embodiment, and more specifically, a graph showing difference values between Green channel values and Red channel values. However, for convenience of explanation, this is only a specific difference value between the green channel value and the red channel value, and it is not limited to this, and various color channel values, difference values, processing values, etc. may be used. .

Referring to (a) of FIG. 19, it can be seen that the difference value between the Green channel value and the Red channel value (hereafter, the "G-R value" will be explained) does not have a constant magnitude of change over time.

At this time, the G-R value may not be constant due to the movement of the subject. For example, when the subject's movement is small, the change in the G-R value is small, and when the subject's movement is large, the change in the G-R value is large, but the present invention is not limited to this.

Also, the G-R value may not be constant depending on the intensity of the external light. For example, when the intensity of external light is weak, the change in G-R value may be small, and when the intensity of external light is strong, the change in G-R value may be large, but the present invention is not limited thereto.

Therefore, characteristic values can be extracted to reduce noise due to subject movement, external light intensity, and the like.

Also, a window for the characteristic value can be set to extract the characteristic value.

At this time, the window for the characteristic value may mean a preset time interval or a preset number of frames, but is not limited thereto, and acquires the characteristic value. It may mean a window for setting at least a part of a group of frames among a plurality of frames.

FIG. 19(b) is a schematic diagram for explaining the window for characteristic values, and more specifically, the window for characteristic values set in 18 image frames obtained by dividing 180 image frames into 10 equal parts. It is a schematic diagram for explaining. However, for convenience of explanation, this only shows a window for characteristic values set in 18 image frames obtained by dividing 180 image frames into 10 equal parts, and is not limited to this. A window can be set for the values.

Referring to FIG. 19(b), a plurality of acquired image frames may be grouped by windows for characteristic values. For example, as shown in FIG. 19(b), 180 image frames can be set into groups containing 18 image frames each with a window for the property values. More specifically, even if the 1st to 18th image frames are included in the first image frame group (2210) and the 19th to 36th image frames are included in the second image frame group (2220), Good, but not limited to this.

At this time, the property values can be obtained for image frames set by a window for property values. For example, the characteristic values can be obtained by color channel values for the first group of image frames (2210) and obtained by color channel values for the second group of image frames (2220).

Also, for example, if the characteristic value is an average value, the average value of the color channel values for the image frames can be obtained. More specifically, the average G-R values for the 1st to 18th image frames included in the first image frame group (2210) are obtained, and the 19th to 36th images included in the second image frame group (2220) are obtained. An average G-R value for the frame is obtained, but is not so limited.

Also, for example, if the characteristic value is a standard deviation value, the standard deviation value of the color channel values for the image frames can be obtained. More specifically, the standard deviation values of the G-R values for the 1st to 18th image frames included in the first image frame group (2210) are obtained, and the 19th to 36th values included in the second image frame group (2220) are obtained. A standard deviation value of the G-R values for the image frame is obtained, but is not limited thereto.

However, various characteristic values can be obtained for a group of image frames without being limited to the above examples.

Also, the characteristic values can be obtained for at least some image frames included in a group of image frames separated by a window for characteristic values. For example, the characteristic values are obtained by color channel values for at least some of the 18 image frames included in the first image frame group (2210) and included in the second image frame group (2220). The color channel values for at least some of the 18 image frames can be obtained.

Also, for example, if the characteristic value is a deviation value, the deviation value of the color channel values for at least some of the image frames included in the image frame group can be obtained. More specifically, the deviation value of the G-R value of the first image frame included in the first image frame group with respect to the average G-R value of the first image frame group (2210) is obtained, and the G-R value of the second image frame group (2220) is obtained. A deviation value of the G-R values of the 19th image frame included in the second image frame group with respect to the value average is obtained, but is not limited thereto.

Also, for example, if the characteristic value is a deviation value, the deviation value of the color channel values for at least some of the image frames included in the image frame group can be obtained. More specifically, the deviation value of the G-R value of the first image frame included in the first image frame group with respect to the average G-R value of the first image frame group (2210) is obtained, and is included in the first image frame group (2210). A deviation value of the G-R values of the second image frame obtained is obtained, but is not limited thereto.

Also, the obtained characteristic values can be normalized.

For example, if the characteristic value is a deviation value, the deviation value can be normalized by a standard deviation value. More specifically, when the deviation value of the G-R value of the first image frame included in the first image frame group (2210) with respect to the G-R value average of the first image frame group (2210) is obtained, the first image frame is normalized by the standard deviation value of the first image frame group 2210, but is not limited thereto, and can be normalized by various methods.

In addition, when normalized in this way, the magnitude of the variation is normalized to better reflect the variation of the value due to the heartbeat, and the noise due to the movement of the subject and the external light. Noise due to changes in strength can be effectively reduced.

FIG. 19(c) is a graph showing characteristic values obtained according to an example, and more specifically, a graph showing deviation values obtained based on GR values. However, for convenience of explanation, this is only a specific deviation value obtained based on the G-R value, and is not limited thereto, and is obtained based on various color channel values, difference values and processing values. Various characteristic values may be used.

Referring to FIG. 19(c), compared with FIG. 19(a), it can be seen that the magnitude of change in value is more constant.

Therefore, obtaining the characteristic values as described above reduces noise caused by the movement of the person being measured and noise caused by changes in the intensity of external light, etc., so that changes in values due to heartbeats can be better reflected. to

FIG. 20 is a diagram for explaining a characteristic value acquisition method according to another embodiment.

FIG. 20(a) is a graph showing color channel values obtained according to one embodiment, and more specifically, a graph showing difference values between Green channel values and Red channel values. However, for convenience of explanation, this is only a specific difference value between the Green channel value and the Red channel value, and is not limited to this, and may be various color channel values, difference values, processing values, etc. can.

Referring to (a) of FIG. 20, it can be seen that the difference value between the Green channel value and the Red channel value (hereinafter referred to as the "G-R value") varies in magnitude over time.

At this time, the G-R value may not be constant due to the movement of the subject. For example, in the time interval (2301) when the subject is in the first state, the overall G-R value is small, and in the time interval (2302) when the subject is in the second state different from the first state, The overall G-R value may be large, but is not limited thereto.

Also, the G-R value may not be constant depending on the intensity of the external light. For example, the intensity of external light in the time interval (2301) in which the subject is in the first state and the intensity of external light in the time interval (2302) in which the subject is in the second state are They are different from each other, which may cause an overall G-R value difference, but is not limited thereto.

Thus, characteristic values can be extracted to reduce noise due to subject movement, external light intensity, and the like.

Also, a window for the characteristic value can be set to extract the characteristic value.

At this time, the window for the characteristic value may mean a preset time interval or a preset number of frames, but is not limited thereto, and acquires the characteristic value. It may mean a window for setting at least a part of a group of frames among a plurality of frames.
Also, at least some frames set by the window may be at least partially overlapped.

FIG. 20(b) is a schematic diagram for explaining a window for characteristic values, and more specifically, for explaining a window for characteristic values set to a size obtained by dividing 180 image frames into eight equal parts. 1 is a schematic diagram for FIG. However, for convenience of explanation, this only shows the window for characteristic values set to the size of 180 image frames equally divided into eight, and is not limited to this. can be set.

Also, referring to (b) of FIG. 20, a plurality of acquired image frames can be grouped by a window for characteristic values. For example, as shown in FIG. 20(b), 180 image frames can be set into groups containing 22 or 23 image frames depending on the window for the property value. More specifically, the first image frame to the 22nd image frame may be included in the first image frame group (2310).

Also, referring to (b) of FIG. 20, the image frames set by the window for the characteristic value can be at least partially overlapped. For example, as shown in FIG. 20(b), the first image frame to the 22nd image frame may be included in the first image frame group (2310), and the sixth image frame to the 28th image frame may be included in the first image frame group (2310). The 12th to 33rd image frames may be included in the 2nd image frame group (2320), the 12th to 33rd image frames may be included in the 3rd image frame group (2330), and the 17th to 39th image frame groups ( 2340), but is not limited thereto.

Also, referring to FIG. 20(b), the image frames set by the windows for the property values may not overlap. For example, as shown in FIG. 20(b), the 1st image frame to the 22nd image frame may be included in the first image frame group (2310), and the 23rd image frame to the 45th image frame may be included in the first image frame group (2310). It may be included in the group of 5 image frames (2350), but is not limited thereto.

At this time, the characteristic value is obtained for the image frame group set by the window for the characteristic value, and is also obtained for at least some image frames included in the image frame group. Since the above-mentioned contents are applied to , the repeated explanation will be omitted.

However, the processing of characteristic values obtained for image frames included in image frames that are at least partially overlapped will be described in detail below.

When obtaining characteristic values, a plurality of characteristic values can be obtained for image frames included in an area where at least two image frame groups overlap.

For example, at least two characteristic values can be obtained for the 6th to 22nd image frames in which the first group of image frames (2310) and the second group of image frames (2320) are overlapped.

More specifically, for the sixth image frame, a first deviation value, which is the deviation value of the G-R value of the sixth image frame with respect to the average G-R value of the first image frame group (2310), is obtained, and the second image frame group is obtained. A second deviation value is obtained, which is the deviation value of the G-R values of the sixth image frame from the G-R value average of (2320), but is not limited thereto.

In addition, a plurality of obtained characteristic values can be obtained as one characteristic value through calculation. For example, the deviation value of the sixth image frame can be obtained by combining the first deviation value and the second deviation value, but the embodiment is not limited thereto.

Also, the operations described above can be applied to obtain characteristic values for image frames contained in regions where multiple groups of image frames, such as three, four, etc. overlap.

Also, in addition to the above-described operations, a sliding window scheme that can be commonly understood may be applied.

FIG. 20(c) is a graph showing characteristic values obtained according to an embodiment, and more specifically, a graph showing deviation values obtained based on GR values. However, for convenience of explanation, this is only to specify the deviation value obtained based on the G-R value, and is not limited thereto. Various characteristic values may be used.

Referring to FIG. 20(c), it can be seen that the magnitude of the overall value is constant compared to FIG. 20(a).

More specifically, the overall characteristic value in the time interval (2301) in which the subject is in the first state and the overall characteristic in the time interval (2302) in which the subject is in the second state Values may be similar to each other.

Therefore, obtaining the characteristic values as described above reduces the noise due to the movement of the person being measured and the noise due to changes in the intensity of external light, etc., and better reflects the changes in the values due to the heartbeat. to be able to

5.3 Various Embodiments of Methods Using Multiple Property Values

The characteristic values obtained by the methods described above can be affected by the underlying color channel values, the difference values and the processing values. Therefore, obtaining a plurality of characteristic values based on various color channel values, difference values, and processing values, and using the plurality of characteristic values can enable acquisition of a more accurate biometric index. .

FIG. 21 is a diagram for explaining a method of using a plurality of characteristic values.

FIG. 21(a) is a graph showing two characteristic values obtained according to one embodiment. More specifically, the first characteristic value obtained based on the G-R value and the G-B value obtained 3 is a graph showing second characteristic values. However, this is only specified for convenience of explanation, and is not limited thereto, and can be characteristic values obtained based on various color channel values, difference values, and processing values.

At this time, the first characteristic value obtained based on the G-R value can be affected by the G-R value. For example, if the external light is light close to the Blue channel, the G-R value may not be able to reflect well the changes in blood due to heartbeat.

Alternatively, for example, changes in blood due to heartbeat can be reflected by being influenced by the difference between the absorbance of the Green channel and the absorbance of the Red channel.

Also, the second characteristic value obtained based on the G-B value can be affected by the G-B value. For example, if the external light is light close to the Red channel, the G-B value may not well reflect changes in blood due to heartbeat.

Alternatively, for example, changes in blood due to heartbeat can be reflected by being influenced by the difference in absorbance between the Green channel and the Blue channel.

Also, referring to (a) of FIG. 21, the first characteristic value and the second characteristic value can have a complementary relationship. For example, the second characteristic value can well reflect the change due to the heartbeat in the section where the first characteristic value cannot well reflect the change due to the heartbeat, and vice versa.

Therefore, the first characteristic value and the second characteristic value can be used to reduce noise due to changes in the wavelength of external light, and to better reflect changes in blood due to heartbeat. .

(b) of FIG. 21 is a graph showing a third characteristic value obtained using the first characteristic value and the second characteristic value. 9 is a graph showing third characteristic values obtained by combining values; However, this is only specifically shown for convenience of explanation, and is not limited thereto.

Also, the third characteristic value can be obtained based on the calculation of the first characteristic value and the second characteristic value. For example, the third characteristic value is obtained based on the sum calculation of the first characteristic value and the second characteristic value, but is not limited thereto, and is obtained based on various calculations such as difference calculation and product calculation. can be obtained.

Also, the third characteristic value may be obtained by giving various weights to the first characteristic value and the second characteristic value. For example, it is obtained based on the following Equation 21, but is not limited thereto.

<Formula 21>
3rd characteristic value = a · 1st characteristic value + b · 2nd characteristic value

Also, referring to (a) and (b) of FIG. 21, the third characteristic value can well reflect changes in blood due to heartbeat from the first characteristic value and the second characteristic value. , noise due to changes in the wavelength of external light can be reduced.

5.4 Various Embodiments of Heart Rate Acquisition Methods In order to acquire heart rate from the data acquired by the methods described above, it is necessary to sense periodic changes due to heart beats. For example, in order to obtain the heart rate from the obtained characteristic values, it is necessary to obtain the wavelength or frequency component most contained in the characteristic values.

FIG. 22 is a graph obtained by extracting frequency components from the characteristic value graph shown in FIG. 21(b). More specifically, FIG. 22 is a graph transformed into the frequency domain by fast Fourier transforming the graph for characteristic values. However, although this shows the fast Fourier transform specifically for convenience of explanation, it is not limited to this, and graphs for characteristic values are fast fourier transform (FFT), discrete Fourier transform (FFT), and fourier transform, DFT), STFT (short time fourier transfom), etc.

Graphs for characteristic values can also be transformed to the frequency domain, as shown in FIG.

At this time, the frequency index with the highest intensity is obtained, and the heart rate can be obtained by Equation 22 below.

<Formula 22>
heart rate = frequency index 60

For example, as shown in FIG. 22, if the highest intensity frequency index is 1.2 Hz, the heart rate can be 72 bpm.

Also, the graph for characteristic values is not shown in FIG. 22, but can be transformed into the frequency*measurement time domain.

At this time, the index with the highest intensity (wavelength index) is obtained, and the heart rate can be obtained by Equation 23 below.

<Formula 23>
Heart rate = (index/measurement time) 60

For example, not shown in FIG. 22, if the highest intensity index is 8 and the measurement time is 6.6 seconds, the heart rate is 8/6.6*60, which can be 72 bpm.

In addition, various mathematical formulas for obtaining the heart rate can be used in addition to the examples described above.

Also, a preliminary heart rate can be obtained to obtain the heart rate. At this time, the preliminary heart rate may be the calculated heart rate on which to obtain the heart rate.

FIG. 23 is a diagram for explaining a heart rate acquisition method according to one embodiment.

Before explaining, the "preliminary heart rate" described below may mean the heart rate obtained by the heart rate obtaining method, and is based on which one heart rate is obtained. It may mean heart rate. For example, the at least two heart rates obtained by the heart rate obtaining method described above may be the first preliminary heart rate and the second preliminary heart rate that are the basis for obtaining one final heart rate, It is not limited to this.

Also, the preliminary heart rate itself may be the final heart rate, or the final heart rate may be obtained based on a plurality of preliminary heart rates.

FIG. 23(a) is a graph showing values obtained from time-series data. For example, (a) in FIG. 23 may mean color channel values obtained from time-series data, but is not limited thereto, and may mean difference values and processed values obtained from time-series data. can also mean characteristic values obtained from time-series data.

As shown in (a) of FIG. 23, the above-described contents are applied to the method of converting the graph showing the values obtained from the time-series data into the wavelength domain or the frequency domain, so repeated explanations are omitted. to

FIG. 23(b) is a schematic diagram for explaining the window for the heart rate reserve, and more specifically, a schematic diagram for explaining the window for the heart rate reserve set to a size of 6 seconds. be. However, this is only specified for convenience of explanation and is not limited to this, and windows with various sizes can be set.

At this time, the window for the preliminary heart rate may mean a preset time interval or a preset number of frames, but is not limited thereto. It may mean a window for setting at least a part of a group of frames among a plurality of frames.

Also, referring to FIG. 23(b), a plurality of acquired image frames can be grouped by a window for the preliminary heart rate. For example, as shown in FIG. 23(b), the image frames acquired from 0 seconds to 6 seconds are included in the first image frame group (2410), but are not limited thereto.

Also, referring to FIG. 23(b), the image frames set by the window for the preliminary heart rate can be at least partially overlapped. For example, as shown in Figure 23(b), the image frames captured between 0 and 6 seconds are included in the first image frame group (2410) and captured between 0.5 and 6.5 seconds. The captured image frames are included in the second image frame group (2420), and the image frames acquired between 1 second and 7 seconds are included in the third image frame group (2430), and are included in the 1.5 second to 7.5 second image frame group. The image frames acquired during are included in the fourth group of image frames (2440), but are not limited thereto.
At this time, the heart rate reserve can be acquired for a group of image frames set by a window for the heart rate reserve. For example, the first preliminary heart rate can be obtained based on characteristic values obtained from image frames included in the first image frame group (2410).

Also, in each image frame group to obtain the preliminary heart rate, the values obtained by the time-series data can be transformed into the wavelength domain or the frequency domain. For example, in the first image frame group, the values obtained from the time series data are converted to the first frequency data (2460), and in the second image frame group, the values obtained from the time series data are converted to the second frequency data. data (2470), and in the third image frame group, the values acquired by the time series data are converted into third frequency data (2480), and in the fourth image frame group, the time series data is converted into fourth frequency data (2490), but is not limited thereto.

In addition, since the above-described heart rate acquisition method is applied to acquire the first to fourth preliminary heart rates from the first to fourth frequency data (2460, 2470, 2480, 2490), repeated description will be omitted. to decide.

A heart rate can also be obtained based on a plurality of preliminary heart rates. For example, a heart rate is acquired by performing calculations on a plurality of preliminary heart rates, and more specifically, calculations for extracting the average, maximum value, minimum value, etc. of the first to fourth preliminary heart rates are performed. Runs to obtain heart rate, but is not so limited.

A heart rate can also be obtained based on a plurality of preliminary heart rates. For example, the heart rate can be obtained by performing calculations on the remaining heart rates, except for ten different preliminary heart rates among the obtained plurality of heart rates.

More specifically, if the first preliminary heart rate is 72 bpm, the second preliminary heart rate is 80 bpm, the third preliminary heart rate is 75 bpm, and the fourth preliminary heart rate is 73 bpm, the second heart rate at 10 different positions , the heart rate can be obtained by performing calculations on the first, third, and fourth preliminary heart rates. At this time, the operation is an operation for extracting an average, maximum value, minimum value, and the like.

A heart rate can also be obtained based on a plurality of preliminary heart rates. For example, if the 10 positions of the 4 acquired heart rate reserves are pairwise different, then the 10 positions among the 4 acquired heart rate reserves are different from the previously acquired heart rate reserve heart rate pairs. The heart rate can be obtained by performing calculations on the rest of the heart rates except for .

More specifically, the first heart rate reserve is 72 bpm, the second heart rate reserve is 80 bpm, the third heart rate reserve is 85 bpm, the fourth heart rate reserve is 73 bpm, and the previously acquired heart rate is 75 bpm. If so, the heart rate can be obtained by performing calculations on the first and fourth preliminary heart rates, excluding the second and third heart rates.

Also, as described above, when the heart rate is obtained using a plurality of preliminary heart rates, the heart rate can be more robust against noise and accurate.

5.5 Various Embodiments of Heart Rate Output The heart rate obtained by the above method can be output through a display or the like, or can be transferred to a terminal or server using a communication unit. In the following, for convenience of explanation, such an operation will be explained by outputting the heart rate.

When performing continuous real-time measurements on heart rate, corrections can be made to the output heart rate to impart stability and reliability to the measured heart rate.

In addition, when correction is performed on the output heart rate, even if a bad image is acquired in some of the acquired image frames, the acquired and output heart rate is stable. and credibility.

FIG. 24 is a flowchart for explaining a method for correcting the output heart rate according to one embodiment.

Referring to FIG. 24, a method for correcting an output heart rate (2500) according to an embodiment includes obtaining a first heart rate (S2510), calculating a difference between the first heart rate and the first time point heart rate as a reference value. and comparing (S2520) with, but not limited to.

Since the heart rate acquisition method described above is applied to the step of acquiring the first heart rate (S2510), repeated description will be omitted.

In the step of comparing the difference between the first heart rate and the first time point heart rate with a reference value (S2520), the first time point heart rate is obtained or output before the first heart rate is obtained. It may mean heart rate. For example, if the first heart rate is acquired at 6.5 seconds, the heart rate at the first time point may be the heart rate acquired at 6 seconds or the heart rate output at 6 seconds, It is not limited to this.

Also, the reference value may be determined as a predetermined numerical value or as a constant ratio. For example, the reference value may be set to 10, and at this time, it can be determined that the difference between the first heart rate and the first heart rate exceeds 10, but is not limited thereto.

Further, if the difference between the first heart rate and the first heart rate is equal to or less than the reference value, the step of outputting the first heart rate as the second heart rate (S2531) is performed. The second time point is a time point later than the first time point.

For example, if the heart rate at the first time point is 72 bpm, the first heart rate is 75 bpm, and the reference value is 10, the heart rate output at the second time point is 75 bpm. Not limited.

Further, if the difference between the first heart rate and the first heart rate exceeds a reference value, the step of outputting the heart rate corrected from the first heart rate as a second heart rate (S2532) is performed. , at this time, the second time point is a time point later than the first time point.

Also, an operation can be performed on the first time point heart rate to obtain a corrected heart rate from the first time point heart rate. For example, an operation such as adding or subtracting a predetermined value from the heart rate at the first time point is executed, but the present invention is not limited to this.

For example, if the heart rate at the first time point is 72 bpm, the first heart rate is 85 bpm, and the reference value is 10, the heart rate output at the second time point is equal to the heart rate at the first time point. It may be +3bpm plus 75bpm, but is not limited to this.

Further, for example, when the heart rate at the first time point is 72 bpm, the first heart rate is 61 bpm, and the reference value is 10, the heart rate output at the second time point is the heart rate at the first time point. It may be 69bpm plus -3bpm, but is not limited to this.

5.6 Various Embodiments of Heartbeat Signal Extraction A heartbeat signal may mean a signal that can vary with heart beats, or a signal that can be estimated to vary with heart beats.

Also, a heartbeat signal can be extracted based on a plurality of acquired image frames.

FIG. 25 is a diagram for explaining a heartbeat signal extraction method according to an embodiment.

First, FIG. 25(a) is a graph showing values obtained from time-series data. For example, (a) in FIG. 25 may mean, but is not limited to, color channel values obtained from time-series data, and may mean difference values and processed values obtained from time-series data. It may mean a characteristic value obtained from time-series data.

At this time, the values obtained from the time-series data can be extracted from the heartbeat signal through a band pass filter. More specifically, the value obtained from the time-series data can be extracted from the heartbeat signal through a bandpass filter of a frequency band or wavelength band corresponding to the heartbeat rate.

In addition, the frequency band or wavelength band corresponding to the heart rate may be a commonly understood frequency band or wavelength band, but is not limited thereto, and is based on the heart rate obtained by the above method. is a frequency band or wavelength band determined by

For example, a normal heart rate may be 60-100 bpm and the relevant frequency band may be 1 Hz-1.67 Hz, for which a relevant bandpass filter is utilized, but is not so limited. .

Also, for example, if the acquired heart rate is 72 bpm, the corresponding frequency is 1.2 Hz, and the frequency band can be set based on this. More specifically, when setting the frequency band in the range of 0.5 Hz, the frequency band is set in the range of 0.95 Hz to 1.45 Hz, for which a corresponding bandpass filter is used, but not limited thereto.

FIG. 25(b) is a heartbeat signal graph obtained by extracting the values obtained from the time-series data shown in FIG. 25(a) into heartbeat signals using a band-pass filter.

Referring to (b) of FIG. 25, it can be seen that the values obtained from the time-series data can be extracted into the heartbeat signal by the band-pass filter.

5.7 Various Embodiments of Heart Rate Measurement Method Using Infrared Rays Basically, the heart rate measurement method described above can also be used for the heart rate measurement method using infrared rays.

FIG. 26 is a diagram for explaining a heart rate acquisition method using infrared rays according to one embodiment.

At this time, the infrared rays in the near infrared region, which is a wavelength band of 750 nm to 3000 nm, are used, but are not limited thereto. infrared), and extreme infrared may also be used.

Referring to FIG. 26, a heart rate measuring method using infrared rays (2600) according to an embodiment acquires an image for at least one image frame among a plurality of acquired image frames (S2610). , detecting a skin region (S2620), detecting a region of interest (S2630), and processing data for the region of interest (S2640). .

In addition, the steps of obtaining the image (S2610) and the step of detecting the skin area (S2620) are the same as described above, so repeated descriptions will be omitted.

In addition, the detailed operation of the step of detecting the region of interest (S2630) is applied to the method of detecting the region of interest described above, so repeated description will be omitted.

Also, detecting the region of interest (S2630) may include detecting first, second and third regions of interest, and is performed on at least one image frame among the plurality of acquired image frames. can be
Also, the first, second and third regions of interest may at least partially overlap each other. For example, the first region of interest is set to be included in the second and third regions of interest, and the second region of interest is set to be included in the third region of interest. , the first to third regions of interest can be set so as to at least partially overlap each other.

Also, the first, second and third regions of interest may be set so as not to overlap each other. For example, the first region of interest and the second region of interest may be positioned vertically on the left cheek of the subject, and the third region of interest may be positioned on the right cheek of the subject. , but not limited thereto, the first to third regions of interest may be set so as not to overlap each other.

In addition, the detailed operation of the step of processing data for the first, second, and third regions of interest (S2640) is the same as the method for processing data for the regions of interest described above, so repeated descriptions will be omitted. to

Also, the step of processing data for the region of interest (S2640) may be performed for at least one image frame among the plurality of acquired image frames.

Also, the step of processing data for the region of interest (S2640) may be performed for the first, second and third regions of interest described above.

IR intensity values for the first to third ROIs are extracted for at least one image frame among a plurality of image frames acquired to process data for the first to third ROIs. be able to. At this time, the IR intensity value may be an average value of IR intensity values of pixels included in the first to third regions of interest, and may be referred to as an average pixel value.

Also, when applying the method of processing data for regions of interest described above, the IR intensity values for each region of interest can correspond to the color channel values described above. For example, the IR intensity values of the first region of interest can correspond to the Red channel values, the IR intensity values of the second region of interest can correspond to the Green channel values, and the IR intensity values of the third region of interest can correspond to the Blue channel values. It can accommodate, but is not limited to.

Also, for example, the G-R values described above can correspond to the difference values between the IR intensity values of the second region of interest and the IR intensity values of the first region of interest, and the G-B values described above can correspond to the IR intensity values of the second region of interest and the IR intensity values of the first region of interest. It can correspond to the difference value of the IR intensity values of the three regions of interest.

Therefore, the data may be processed based on the IR intensity values for each region of interest, and the detailed operations can follow the data processing method for the regions of interest described above.

In addition, a heart rate measurement method using infrared rays (2600) according to an embodiment extracts time-series data for at least some image frame groups among the plurality of acquired image frames (S2650); At least part of the heart rate acquisition step (S2660) is included, but not limited thereto.

At this time, the step of extracting the time-series data (S2650) and the step of obtaining the heart rate (S2660) may be applied to the above-described content, so repeated description will be omitted.

FIG. 27 is a diagram for explaining a heart rate acquisition method using infrared rays according to one embodiment.

Referring to FIG. 27, at least two regions of interest can be set in the face region of the subject. More specifically, a first region of interest (2710), a second region of interest (2720) and a third region of interest (2730) are set in the face region of the subject, but the present invention is not limited thereto.

Also, IR intensity values for the first to third regions of interest (2710, 2720, 2730) can be extracted.

For example, an IR intensity value ((a) of FIG. 27) is extracted for the first region of interest (2710), and an IR intensity value ((b) of FIG. 27) is extracted for the second region of interest (2720). Alternatively, an IR intensity value ((c) in FIG. 27) for the third region of interest (2730) may be extracted, but is not limited thereto.

Data for the first to third regions of interest (2710, 2720, 2730) are processed based on the IR intensity values extracted for the first to third regions of interest (2710, 2720, 2730). be able to. However, the detailed operation for this will be omitted as described above.

Also, characteristic values can be obtained based on data processed for the first to third regions of interest (2710, 2720, 2730). However, the detailed operation for this will be omitted as described above.

Also, the heart rate can be obtained using characteristic values obtained based on the IR intensity values extracted from the first to third regions of interest (2710, 2720, 2730). However, the detailed operation for this will be omitted as described above.

6. Biometric index acquisition method according to one embodiment

FIG. 28 is a flowchart for explaining a biometric index acquisition method according to one embodiment.

Referring to FIG. 28, a method for obtaining a biometric index according to an embodiment includes steps of obtaining a plurality of image frames of a subject (S2810) and obtaining first, second and third color channel values. (S2820); calculating a first difference value and a second difference value (S2830); obtaining a first characteristic value and a second characteristic value (S2840); At least a part of step (S2850) of determining the biometric index of the subject based on the measurement may be included.

At this time, the plurality of images can be acquired by a camera. For example, the plurality of images can be acquired by a camera such as a visible light camera, an IR camera, or the like.

Also, the plurality of images may be acquired from an external camera. For example, the plurality of images are images obtained from a camera such as a visible light camera, an IR camera, etc., which are installed outside.

Also, the step of obtaining the first, second and third color channel values (S2820) may be performed for at least one image frame among the plurality of obtained image frames.

At this time, the first color channel value may mean the average pixel value for the first color channel of the image frame in which the step is performed, and the second color channel value may mean the second color channel of the image frame. and the third color channel value may mean the average pixel value for the third color channel of the image frame.

For example, the first color channel value may be the Green channel value, the second color channel value may be the Red channel value, and the third color channel value may be the Blue channel value. , but not limited to.

Also, the step of calculating the first difference value and the second difference value (S2830) may be performed for at least one image frame among the plurality of acquired image frames.

At this time, the first difference value may mean a difference value between a first color channel value and a second color channel value, for example, the first difference value is the same image frame as the first color channel value. It may mean, but is not limited to, the difference value of the second color channel value.

More specifically, if said first color channel value is a Green channel value and said second color channel value is a Red channel value, said first difference value may be a G-R value, but is limited thereto. not.

Also, the second difference value may mean a difference value between a first color channel value and a third color channel value, for example, the second difference value is a first color channel value and a third color channel value for the same image frame. It may mean, but is not limited to, the difference value of the three color channel values.

More specifically, if said first color channel value is a Green channel value and said second color channel value is a Blue channel value, said second difference value may be a G-B value, but is limited thereto. not.

In addition, the step of obtaining the first characteristic value and the second characteristic value (S2840) may be performed on at least some of the obtained plurality of image frames.

For example, the first characteristic value may be acquired for a first group of image frames, and the first group of image frames may mean a group of image frames acquired during a preset time.

Also, for example, the second characteristic value may be acquired for a second group of image frames, and the second group of image frames may mean a group of image frames acquired during a preset time.

Also, the first characteristic value may be obtained based on an average value of the first difference values for the first image frame group and a first difference value for image frames included in the first image frame group. .

For example, the first characteristic value may be a deviation value obtained based on the average value of the G-R values for the first image frame group and the G-R values for the image frames included in the first image frame group. , but not limited to.

Also, the second characteristic value may be obtained based on an average value of the first difference values for the second image frame group and a second difference value for image frames included in the second image frame group. .

For example, the second characteristic value may be a deviation value obtained based on the average value of the G-B values for the second image frame group and the G-B values for the image frames included in the second image frame group. , but not limited to.

Also, the first image frame group and the second image frame group may be the same, but are not limited thereto, and may be different.

Also, the first characteristic value may be an average value, a standard deviation value, etc. for the first image frame group, but is not limited thereto.

Also, the first characteristic value may be a deviation value for at least some image frames included in the first image frame group, but is not limited thereto.

Also, the second characteristic value may be an average value, a standard deviation value, etc. for the second image frame group, but is not limited thereto.

Also, the second characteristic value may be a deviation value for at least some image frames included in the second image frame group, but is not limited thereto.

Also, the first and second characteristic values may be normalized. For example, the first characteristic value is normalized using the standard deviation value for the first image frame group, but is not limited thereto.

Also, for example, the second characteristic value is normalized using a standard deviation value for the second image frame group, but is not limited thereto.

Also, the step of determining the biometric index of the person to be measured based on the first and second characteristic values (S2850) may be performed on at least a partial image frame group among the plurality of acquired image frames. can.

At this time, the biometric index may be heart rate, blood pressure, oxygen saturation, body temperature, etc., but is not limited thereto.

Also, the biometric index can be obtained based on the first characteristic value and the second characteristic value.

For example, the biometric index is obtained based on a third characteristic value that is a combination of the first characteristic value and the second characteristic value, but is not limited thereto.

7. Various Embodiments of Methods for Acquiring Multiple Biometrics and Biometric Information FIG. 29 is a diagram illustrating a method for acquiring multiple biometrics and biometric information according to one embodiment.

Referring to FIG. 29, a method for acquiring multiple biometrics and biometric information according to one embodiment can acquire multiple biometrics based on a captured image (3000). More specifically, as shown in FIG. 29, obtaining a first biometric index (3010), a second biometric index (3020) and a third biometric index (3030) based on the acquired image (3000). Can, but is not limited to, at least two or more bioindexes can be obtained, for example, four bioindexes can be obtained.

At this time, the acquired image (3000) may be a visible light image acquired using a visible light camera or a visible light image acquired from an external visible light camera.

Also, the acquired image (3000) may be an infrared image acquired using an infrared camera or an infrared camera installed outside.

Also, the first to third biometric indices (3010, 3020, 3030) include at least one of heart rate, oxygen saturation, blood pressure, and body temperature, but are not limited thereto.

In addition, the first to third biometric indices (3010, 3020, 3030) can be obtained so as to be related to each other. For example, the first to third biometric indices (3010, 3020, 3030) may be biometric indices obtained from the same or similar conditions of the subject, but are not limited thereto.

Also, the first to third biometric indices (3010, 3020, 3030) can be obtained independently of each other. For example, the first to third biometric indices (3010, 3020, 3030) may be biometric indices obtained from different states of the subject, but are not limited thereto.

Also, the first to third biometric indices (3010, 3020, 3030) can be output at the same time. For example, when the first to third biometric indices (3010, 3020, 3030) are measured in real time, they may be output at the same time, but the embodiment is not limited thereto.

Also, the first to third biometric indices (3010, 3020, 3030) can be output at different times. For example, after the first biometric is output, the second biometric is output, and after the second biometric is output, the third biometric is output.

Also, referring to FIG. 29, the method for obtaining multiple biometric indices and biometric information according to one embodiment can acquire multiple biometric information based on the multiple biometric indices obtained.

More specifically, as shown in FIG. 29, first biological information (3040), second biological information (3050) and The third biometric information (3060) can be obtained, but is not limited thereto.

At this time, the first to third biometric information (3040, 3050, 3060) includes at least one of drowsiness information, stress information, excitement level information, and emotion information, but is not limited thereto.

Also, the first to third biometric information (3040, 3050, 3060) are obtained based on at least one biometric index among the first to third biometric indices (3010, 3020, 3030). is not limited to

Also, the first to third biometric information (3040, 3050, 3060) are acquired in consideration of personal and statistical data in addition to the first to third biometric indices (3010, 3020, 3030). . At this time, the personal and statistical data may mean personal and statistical data that can be collected from the person to be measured, such as age, gender, height, weight, facial expression, wrinkles, complexion, etc. of the person to be measured. It may mean observable personal, statistical data, including the subject or statistical data calculated for a group related to the subject (e.g., 20 average blood pressure, yellow person It may mean personal and statistical data that can be quantified along with average skin color, average height of men in their 30s, average weight of Korean men, etc.).

Also, the first to third biometric information (3040, 3050, 3060) can be obtained based on independent biometric indices. For example, the first to third biometric information (3040, 3050, 3060) are obtained based on the first to third biometric indices (3010, 3020, 3030) independently obtained. can be done.

Also, the first to third biometric information (3040, 3050, 3060) can be obtained based on related biometric indices. For example, the first to third biometric information (3040, 3050, 3060) are acquired based on the first to third biometric indices (3010, 3020, 3030) acquired to have relevance. can be done.

In addition, when the first to third biometric information (3040, 3050, 3060) are acquired based on related biometric indices, the accuracy of the biometric information can be improved. For example, in order to obtain hypertension information included in the first to third biometric information (3040, 3050, 3060), heart rate and blood pressure included in the first to third biometric indices (3010, 3020, 3030) , the hypertension information may be more accurate when the heart rate and blood pressure are related to each other.

More specifically, when blood pressure is measured while the subject is exercising or excited, heart rate is measured when the subject is stable, and hypertension information is obtained based on this, the subject's blood pressure is measured. may be perceived as hypertension even if it is not hypertension. However, on the contrary, when the subject is exercising or excited, the blood pressure and heart rate are measured, and the hypertension information is obtained based on the measurement, the hypertension information can be obtained more accurately.

In the following, the plurality of biometric indices and biometric information acquisition method will be described in more detail.

FIG. 30 is a diagram for explaining a multiple biometric index acquisition method according to an embodiment.

Referring to FIG. 30, an image frame (3110) can be acquired to acquire a plurality of biometric indices according to one embodiment.

At this time, the image frame 3110 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, the image frame 3110 includes a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 30, at least one pixel value (3120) can be obtained to obtain a plurality of biometric indices according to one embodiment.

At this time, the pixel value 3120 may mean the intensity value of at least one pixel constituting the image frame, more specifically, the intensity value of at least one pixel for at least one color channel. may mean. For example, if the image frame is a 640*480 image, the image frame may contain 640*480 pixels, and the Red channel pixel value, Green channel pixel value and Blue channel pixel value for each pixel are Pixel values for color channels in various color spaces can be obtained, but are not limited to this.

Also, the pixel value 3120 may be obtained for at least some pixels among the plurality of pixels forming the image frame.

Also, referring to FIG. 30, at least one or more color channel values (3130) can be obtained to obtain multiple biometric indices according to one embodiment.

At this time, the color channel value 3130 may mean an average value of pixel values of the color channel. For example, the Red channel value may mean the mean value of the Red channel pixel values, the Green channel value may mean the mean value of the Green channel pixel values, and the Blue channel value may mean the mean value of the Blue channel pixel values. It may mean an average value, but is not limited thereto.

Although the color channel value and the processed value are generally described separately in the specification, the processed value is included in the color channel value below in order to describe a method of obtaining a plurality of biometric indices. can be explained in terms of concepts

Also, the color channel values (3130) can be described including processing values. For example, the color channel values are described as including G-R values that are the difference between Red channel values and Green channel values, but are not limited thereto.

Also, the color channel values 3130 can be obtained for at least one image frame among the plurality of image frames obtained.

Also, the color channel value 3130 may be obtained based on pixel values of one color channel, or may be obtained based on pixel values of a plurality of color channels. For example, the Red channel value is obtained based on the Red channel pixel value, and the G-R value is obtained based on the Green channel pixel value and the Red channel pixel value, but is not limited thereto.

Also, referring to FIG. 30, at least one or more time-series data can be obtained to obtain a plurality of biometric indices according to one embodiment.

At this time, the time-series data (3140) is obtained based on at least one or more obtained color channel values (3130), and is applied to at least a part of the obtained plurality of image frames. can be obtained for
More specifically, when the time-series data (3140) is obtained based on at least one or more color channel values (3130) obtained for a first image frame, the time-series data is obtained from the first Image frames may be acquired for a group of image frames.

Also, the time-series data (3140) may be time-series data of the color channel values (3130). For example, the time-series data (3140) is time-series data of Red channel values, time-series data of Green channel values, time-series data of Blue channel values, time-series data of Hue channel values, time-series data of G-R values, or Including, but not limited to, time-series data of G-B values.

It may also be time-series data (3140) of characteristic values obtained based on the color channel values (3130). For example, the time-series data includes, but is not limited to, deviation, standard deviation, or average of color channel values for at least some image frames included in the image frame group.

For example, the time-series data (3140) may be red channel value time-series data or G-R value time-series data, but is not limited thereto.

Also, for example, the time-series data (3140) may be time-series data of deviation values of color channel values for at least some image frames included in the image frame group, but is not limited thereto.

The time-series data (3140) is time-series data acquired based on a plurality of pieces of time-series data. For example, the time-series data (3140) is based on first time-series data for a first image frame group including the first image frame and second time-series data for a second image frame group including the second image frame. It may be time-series data obtained, but is not limited thereto.

In a more specific example, when the time-series data (3140) is time-series data for an image frame group including 180 image frames, the time-series data (3140) includes the 1st to 18th image frames. First time-series data acquired for a first image frame group including 19th to 36th image frames, second time-series data acquired for a second image frame group including 19th to 36th image frames, and 37th to 54th image frames The third time-series data acquired for the third image frame group including the 55th to 72nd image frames, the fourth time-series data acquired for the fourth image frame group including the 55th to 72nd image frames, and the 73rd to 90th image frames 5th time-series data acquired for the 5th image frame group including the 5th time-series data, 6th time-series data acquired for the 6th image frame group including the 91st to 108th image frames, and the 109th to 126th image frames 7th time-series data acquired for the 7th image frame group including the 7th time-series data acquired for the 8th image frame group including the 127th to 144th image frames, and the 145th to 162nd image frames 9th time-series data acquired for the 9th image frame group including the 10th time-series data acquired for the 10th image frame group including the 163rd to 180th image frames. can.

Also, referring to FIG. 30, features (3150) for obtaining multiple biometrics can be obtained according to one embodiment.

At this time, the features (3150) may mean mathematical and physical features of the acquired time-series data. For example, the features are the frequency component of the acquired time-series data, the maximum value, the average maximum value, the minimum value, the average minimum value, the difference between the maximum and minimum values, the difference between the average maximum value and the minimum average value, AC value, DC value, average value, inflection point, first derivative data, second derivative data, slope at a specific time, etc., blood change amount, blood change speed, blood vessel It may refer to physical features such as the amount of change and the rate of change of blood vessels, but is not limited thereto.

Also, the feature (3150) may mean a mathematical or physical feature between a plurality of pieces of time-series data. For example, the feature may mean a mathematical feature such as a time difference between a plurality of acquired time series data, a time difference between maxima, a time difference between local minima, a time difference between inflection points, etc. Physical features such as PTT (Pulse Transit Time), blood change speed difference, blood vessel change time difference, etc., are not limited thereto.

Also, the features (3150) can be acquired for at least some image frames among the acquired plurality of image frames.

Also, the features (3150) can be obtained based on at least one piece of time-series data (3140). For example, the feature (3150) may be obtained based on one time-series data, or may be obtained based on at least two or more time-series data, but is not limited thereto.

Also, referring to FIG. 30, a biometric index (3160) can be obtained to obtain a plurality of biometric indices according to one embodiment.

At this time, the biometric index 3160 includes, but is not limited to, heart rate, oxygen saturation, blood pressure, body temperature, and the like.

Also, the biometric index (3160) can be obtained based on different characteristics. For example, heart rate can be obtained based on the frequency component of the time series data, blood pressure can be obtained based on the PTT value between the two time series data, and oxygen saturation can be obtained based on each of the two time series data. It is obtained based on the AC value and the DC value, but is not limited thereto.

In addition, a plurality of biometric indices (3160) can be obtained. For example, but not limited to, heart rate, oxygen saturation, and blood pressure are obtained.

In addition, the biometric index 3160 may be obtained for at least some image frames among the plurality of obtained image frames.

Also, a group of image frames based on which the plurality of biometric indices (3160) are obtained may be identical to each other. For example, heart rate, oxygen saturation, and blood pressure are acquired based on a group of image frames including, but not limited to, 1st to 180th image frames.

Also, the group of image frames based on which the plurality of biometric indices (3160) are obtained can be different from each other. For example, the heart rate is obtained based on a first group of image frames including 1st to 90th image frames, and the oxygen saturation is obtained based on a second group of image frames including 91st to 180th image frames. and the blood pressure is obtained based on the third image frame group including the 181st to 270th image frames, but is not limited thereto.

Also, a group of underlying image frames can be at least partially overlapped with each other to obtain a plurality of the biometric indices (3160). For example, the heart rate is obtained based on a first group of image frames including the 1st to 180th image frames, and the oxygen saturation is obtained based on a second group of image frames including the 30th to 100th image frames. and the blood pressure is obtained based on a third image frame group including the 10th to 180th image frames, but is not limited thereto.

Also, the number of image frames included in the group of image frames that are the basis for obtaining the plurality of biometric indices (3160) may be the same or different.

7.1 Multiple Biometrics Acquisition Method According to an Embodiment FIG. 31 is a diagram for explaining a multiple biometrics acquisition method according to an embodiment.

Referring to FIG. 31, an image frame (3210) can be acquired to acquire a plurality of biometric indices according to one embodiment.

At this time, the image frame 3210 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, the image frame 3210 includes a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 31, at least one region of interest 3220 can be set to acquire a plurality of biometric indices according to one embodiment. More specifically, a first ROI, a second ROI, a third ROI and a fourth ROI can be set.

At this time, the first region of interest may be a region of interest for obtaining oxygen saturation, the second region of interest may be a region of interest for obtaining heart rate, and the third region of interest may be a region of interest for obtaining heart rate. The region may be a region of interest for obtaining blood pressure, and the fourth region of interest may be a region of interest for obtaining body temperature, but is not limited thereto.

Also, the first to fourth regions of interest may be the same or different.

Also, the first to fourth regions of interest may overlap each other at least partially.
Also, the sizes and areas of the first to fourth regions of interest can be set based on the biometrics to be obtained. For example, the second region of interest for obtaining heart rate may be set large so as to include the cheek region of the person to be measured in order to better sense changes in blood due to heartbeat, and obtain blood pressure. The third region of interest may be set to have a small length and a long width so as to include the cheek region of the person to be measured in order to sense fine blood flow velocities, but is not limited thereto. , the first to fourth regions of interest may be set in various sizes and regions.

Also, referring to FIG. 31, at least one pixel value (3230) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, the pixel values 3230 may be obtained for at least one image frame among the plurality of obtained image frames.

More specifically, at least some of a Red channel pixel value, a Green channel pixel value and a Blue channel pixel value in an RGB color space are obtained for the first region of interest, and an RGB color space for the second region of interest is obtained. at least a portion of a Red channel pixel value, a Green channel pixel value, and a Blue channel pixel value in a color space may be obtained, wherein the Red channel pixel value in an RGB color space for the third region of interest; At least a portion of a Green channel pixel value and a Blue channel pixel value is obtained, and a Hue channel pixel value, a Saturation channel pixel value, and a Value channel pixel value are obtained in the HSV color space for the fourth region of interest. but not limited thereto.

Also, referring to FIG. 31, at least one color channel value (3240) can be obtained to obtain a plurality of biometric indices according to one embodiment.

At this time, the color channel value (3240) may mean the average value of the pixel values of the color channel, or may mean the processed value. I will omit it.

Also, the color channel values (3240) can be obtained considering the biometrics obtained.

More specifically, at least two color channel values can be obtained for said first region of interest for obtaining oxygen saturation.

For example, for the first region of interest for obtaining oxygen saturation, red channel value, green channel value, blue channel value in RGB color space, Hue channel value, saturation channel value, value channel value in HSV color space, etc. At least two of the color channel values are obtained, but are not so limited.

Further, for example, for the first region of interest for obtaining oxygen saturation, the G-R value, which is the difference value between the Red channel value and the Green channel value, the G-B value, which is the difference value between the Green channel value and the Blue channel value, the Hue channel At least two of the color channel values are obtained, such as, but not limited to, H-V values that are the difference between the Value and the Value channel values.

Also, the at least two color channel values can be selected taking into account the absorbance of hemoglobin and oxyhemoglobin.

For example, a Blue channel in which the oxyhemoglobin absorbance is higher than the hemoglobin absorbance and a Red channel in which the oxyhemoglobin absorbance is lower than the hemoglobin absorbance may be selected, whereby the at least two color channel values are the Red channel values. and Blue channel values, but are not limited thereto.

Also, at least two color channel values can be obtained for the second region of interest for obtaining heart rate.

For example, for the second region of interest for obtaining the heart rate, colors such as Red channel value, Green channel value, Blue channel value in RGB color space, Hue channel value, Saturation channel value, Value channel value in HSV color space, etc. At least two color channel values of the channel values are obtained, but are not so limited.

Further, for example, for the second region of interest for obtaining the heart rate, the G-R value that is the difference value between the Red channel value and the Green channel value, the G-B value that is the difference value between the Green channel value and the Blue channel value, the Hue channel At least two of the color channel values are obtained, such as, but not limited to, H-V values that are the difference between the Value and the Value channel values.

Also, the at least two color channel values may be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference value between the Green channel value where the absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value where the absorption by hemoglobin and oxyhemoglobin relatively decreases, and the relative Generally, the G-B value, which is the difference value between the Green channel value at which absorption is increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto.

Also, for example, in order to reduce noise, the difference value between the Green channel value that relatively reflects changes due to heart beat, the Red channel value that reflects relatively less changes due to heart beat, and the Blue channel value G-R and G-B values are selected that are, but are not limited to.

Also, at least two color channel values can be obtained for the third region of interest for obtaining blood pressure.

For example, for the third region of interest for obtaining blood pressure, color channels such as Red channel value, Green channel value, Blue channel value in RGB color space, Hue channel value, Saturation channel value, Value channel value in HSV color space. At least two color channel values of the values are obtained, but are not so limited.

Further, for example, for the third region of interest for obtaining blood pressure, the G-R value that is the difference between the Red channel value and the Green channel value, the G-B value that is the difference between the Green channel value and the Blue channel value, the Hue channel value, and At least two of the color channel values are obtained, such as, but not limited to, H-V values that are the difference values of the Value channel values.

Also, the at least two color channel values may be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference value between the Green channel value where the absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value where the absorption by hemoglobin and oxyhemoglobin relatively decreases, and the relative Generally, the G-B value, which is the difference value between the Green channel value at which absorption is increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto.

Also, for example, in order to reduce noise, the difference value between the Green channel value that relatively reflects changes due to heart beat, the Red channel value that reflects relatively less changes due to heart beat, and the Blue channel value G-R and G-B values are selected that are, but are not limited to.

Also, at least two color channel values can be obtained for the fourth region of interest for obtaining body temperature.

For example, color channels such as Red channel value, Green channel value, Blue channel value in RGB color space, Hue channel value, Saturation channel value, Value channel value in HSV color space for the fourth region of interest for obtaining body temperature. At least two color channel values of the values are obtained, but are not so limited.

Further, for example, for the fourth region of interest for acquiring body temperature, the G-R value that is the difference value between the Red channel value and the Green channel value, the G-B value that is the difference value between the Green channel value and the Blue channel value, the Hue channel value and At least two of the color channel values are obtained, such as, but not limited to, H-V values that are the difference values of the Value channel values.

Also, the at least two color channel values can be selected taking into account body temperature characteristics and the subject's skin color.

For example, but not limited to, a Saturation (brightness) channel value associated with the subject's body temperature and a Hue (color) channel value associated with the subject's skin color are selected.

Also, referring to FIG. 31, at least one or more time series data (3250) can be obtained to obtain a plurality of biometric indices according to one embodiment.

Also, the time-series data (3250) can be obtained in consideration of the biometrics obtained.

More specifically, at least two time series data can be acquired for the first region of interest for acquiring oxygen saturation. For example, a first time-series data and a second time-series data can be acquired for the first region of interest for acquiring oxygen saturation.

At this time, the first time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest for obtaining oxygen saturation is the Red channel value, the first time-series data is obtained based on the Red channel value. is not limited to

Also, the first time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the first time-series data is obtained based on the Red channel values obtained for the first image frame, the first time-series data is obtained for the first image frame group including the first image frame. obtained, but not limited to.

Also, the second time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest for obtaining oxygen saturation is the Blue channel value, the second time-series data is obtained based on the Blue channel value. is not limited to

Also, the second time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the second time-series data is acquired based on the Blue channel value acquired for the second image frame, the second time-series data is acquired for the second image frame group including the second image frame. is obtained for, but not limited to.

Also, the first and second image frame groups may be identical to each other, may be different from each other, and may at least partially overlap each other.

Also, at least one time-series data can be acquired for the second region of interest for acquiring heart rate. For example, a third time series data can be acquired for the second region of interest for acquiring heart rate.
At this time, the third time-series data can be obtained based on the color channel values obtained for the second region of interest. For example, if the color channel values obtained for the second region of interest for obtaining heart rate are GR and GB values, the third time-series data are obtained based on GR and GB values. but not limited thereto.

Also, the third time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the third time-series data is obtained based on the G-R values and the G-B values obtained for the third image frame, the third time-series data is the third image frame group including the third image frame. is obtained for, but not limited to.

Also, the third time-series data can be obtained based on characteristic values obtained for the second region of interest. For example, the third time-series data is obtained based on the characteristic value obtained based on the G-R value and the characteristic value obtained based on the G-B value obtained for the second region of interest. is not limited to

Also, at least one time-series data can be acquired for the third region of interest for acquiring blood pressure. For example, a fourth time series data can be acquired for the third region of interest for acquiring blood pressure.

At this time, the fourth time-series data can be obtained based on the color channel values obtained for the third region of interest. For example, if the color channel values obtained for the third region of interest for obtaining blood pressure are G-R values and G-B values, the fourth time-series data is obtained based on the G-R values and G-B values, It is not limited to this.

Also, the fourth time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the fourth time-series data is obtained based on the G-R values and the G-B values obtained for the fourth image frame, the fourth time-series data is the fourth image frame group including the fourth image frame. is obtained for, but not limited to.

Also, the fourth time-series data can be obtained based on characteristic values obtained for the third region of interest. For example, the fourth time-series data is obtained based on the characteristic value obtained based on the G-R value and the characteristic value obtained based on the G-B value obtained for the third region of interest, It is not limited to this.

Also, the first, second, third and fourth image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may at least partially overlap each other. good.

Also, referring to FIG. 31, at least one biometric acquisition model 3251 can be used to acquire a plurality of biometrics according to one embodiment.

However, as described above, repeated description of the biometric acquisition model (3251) will be omitted.

Also, the biometric acquisition model (3251) can be used to acquire body temperature.

More specifically, the biometric acquisition model (3251) can take as input values the color channel values acquired for the fourth region of interest for acquiring body temperature. For example, the biometric acquisition model (3251) may use Hue channel values and Saturation channel values acquired for the fourth region of interest as input values, but is not limited thereto.

In addition, the input values of the biometric index acquisition model (3251) may be color channel values, characteristic values, average color channel values for a group of image frames, etc., but are not limited thereto.

Also, referring to FIG. 31, at least one feature (3260) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, at least one or more of the features (3260) can be obtained in consideration of the biometrics obtained.

More specifically, at least one or more features can be obtained for the first region of interest for obtaining oxygen saturation. For example, a first feature can be acquired for the first region of interest for obtaining oxygen saturation.

At this time, the first feature can be obtained based on color channel values or time-series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are a Red channel value and a Blue channel value, then the first feature is obtained based on the Red channel value and the Blue channel value. is not limited to

Also, for example, the first feature is acquired based on first time-series data and second time-series data acquired for the first region of interest, but is not limited thereto.

Also, the first feature is a feature for obtaining oxygen saturation. For example, the first feature may be an AC value and a DC value obtained based on the first time-series data, and an AC value and a DC value obtained based on the second time-series data, It is not limited to this.

Further, for example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second It includes, but is not limited to, at least one of the difference between the average maximum value and the average minimum value obtained based on the time-series data, and the average value obtained based on the second time-series data.

Also, the first feature may include a plurality of features. For example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second time-series data, It includes, but is not limited to, at least two or more of the difference between the average of the maximum values and the average of the minimum values obtained based on the data, and the average value obtained based on the second time-series data.

Also, at least one or more features can be acquired for the second region of interest for acquiring heart rate. For example, a second feature can be acquired for the second region of interest for acquiring heart rate.

At this time, the second feature can be obtained based on color channel values or time-series data obtained for the second region of interest. For example, if the color channel values obtained for the second region of interest are G-R values and G-B values, the second feature is obtained based on the G-R values and the G-B values, but is not limited thereto. .

Also, for example, the second feature is acquired based on the third time-series data acquired for the second region of interest, but is not limited thereto.

Also, the second feature is a feature for obtaining a heart rate. For example, the second feature may be, but is not limited to, the frequency value, the wavelength value, or the number of repetitions of the measurement time period obtained based on the third time-series data.

Also, at least one or more features can be obtained for the third region of interest for obtaining blood pressure. For example, a third feature can be obtained for the third region of interest for obtaining blood pressure.

At this time, the third feature can be obtained based on color channel values or time-series data obtained for the third region of interest. For example, if the color channel values obtained for the third region of interest are G-R values and G-B values, the third feature is obtained based on the G-R values and the G-B values, but is not limited thereto. .

Also, for example, the third feature is acquired based on the fourth time-series data acquired for the third region of interest, but is not limited to this.

Also, the third feature is a feature for obtaining blood pressure. For example, the third feature is the slope value, maximum value, minimum value, average maximum value, average minimum value, average maximum value, and average minimum value obtained based on the fourth time-series data. may be, but is not limited to.

Also, the third feature may include a plurality of features. For example, the third feature is at least the slope value, the maximum value, the minimum value, the average maximum value, the average minimum value, the average maximum value, and the difference value between the average minimum values obtained based on the fourth time-series data. Including but not limited to two or more.

Also, at least one or more features can be acquired for the fourth region of interest for acquiring body temperature. For example, a fourth feature can be acquired for the fourth region of interest for acquiring body temperature.

At this time, the fourth feature can be obtained based on the color channel values obtained for the fourth region of interest or the output values of a biometric obtaining model. For example, if the color channel value obtained for the fourth region of interest is a Hue channel value and a Saturation channel value, the fourth feature is obtained based on the Hue channel value and the Saturation value. is not limited to

Also, for example, the fourth feature is acquired based on the output value of the biometric acquisition model, but is not limited thereto.

Also, the fourth feature is a feature for acquiring body temperature. For example, the fourth characteristic may be skin site, skin temperature, etc., but is not limited thereto.

Also, referring to FIG. 31, at least one biometric index (3270) can be obtained to obtain a plurality of biometric indices according to one embodiment.

More specifically, each biometric index can be obtained based on the corresponding features. For example, oxygen saturation can be obtained based on the first feature, heart rate can be obtained based on the second feature, blood pressure can be obtained based on the third feature, and body temperature can be obtained based on the third feature. 4 It is obtained based on features, but is not limited thereto.

At this time, the above mathematical formula may be used to obtain the oxygen saturation based on the first characteristic, and repeated description will be omitted.

In addition, the above mathematical formula may be used to obtain the heart rate based on the second feature, and repeated description will be omitted.

Also, the above mathematical formula may be used to obtain the blood pressure based on the third feature, and the repeated description will be omitted.

Also, the above mathematical formula may be used to obtain the body temperature based on the fourth feature, and the repeated description will be omitted.

In addition, when the biometric index 3270 is plural, each biometric index is obtained at the same point in time, but is not limited thereto, and may be obtained at different times. For example, oxygen saturation and heart rate are obtained 6 seconds after measurement, and blood pressure is obtained 8 seconds after measurement, but are not limited thereto.

In addition, when the biometric index 3270 is plural, each biometric index can be obtained based on a group of image frames. For example, oxygen saturation can be obtained based on the fifth image frame group, heart rate can be obtained based on the sixth image frame group, and blood pressure can be obtained based on the seventh image frame group. can.

In addition, each image frame group for obtaining the biometric index (3270) may be identical to each other, but is not limited thereto, may be different from each other, and may at least partially overlap each other. good. For example, the fifth, sixth and seventh image frame groups may be identical to each other, different from each other, or at least partially overlapped with each other.

Also, at least one preliminary biometric index may be obtained to obtain the biometric index (3270). For example, at least four preliminary heart rates can be obtained to obtain the heart rate.

However, the method of acquiring the heart rate or the biometric index using such a preliminary heart rate or biometric index is the same as described above, so a repeated description will be omitted.

Also, the number of preliminary biometrics for obtaining the biometrics (3270) may be the same or different for each biometric. For example, the heart rate reserve for obtaining the heart rate may be at least 4, and the oxygen saturation reserve and blood pressure reserve for obtaining the oxygen saturation and the blood pressure may be at least 2. Good, but not limited to this.

7.2 Multiple Biometrics Acquisition Method According to an Embodiment FIG. 32 is a diagram for explaining a multiple biometrics acquisition method according to an embodiment.

Referring to FIG. 32, an image frame (3310) can be acquired to acquire multiple biometrics according to one embodiment.

At this time, the image frame 3310 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, the image frame 3310 includes a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 32, at least one region of interest 3320 can be set to acquire a plurality of biometric indices according to one embodiment. More specifically, a first ROI, a second ROI and a third ROI can be set.

At this time, the first region of interest may be a region of interest for obtaining oxygen saturation and heart rate, and the second region of interest and the third region of interest may be regions of interest for obtaining blood pressure. may

Also, the first to third regions of interest may be the same or different.

Also, the first to third regions of interest may overlap each other at least partially.

Also, the sizes and areas of the first to third regions of interest can be set based on the biometrics to be acquired. For example, to obtain heart rate and oxygen saturation, the first region of interest may be sized to include the subject's cheek region to better sense changes in blood due to heart beats. Often, the second and third regions of interest for obtaining blood pressure may be set with respect to the direction of blood flow in order to reflect the flow of blood well, but are not limited thereto, and the above The first to third ROIs can be set to have various sizes and various areas.

Also, referring to FIG. 32, at least one pixel value (3330) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, the pixel values 3330 may be obtained for at least one image frame among the plurality of obtained image frames.

More specifically, at least some of a Red channel pixel value, a Green channel pixel value and a Blue channel pixel value in an RGB color space are obtained for the first region of interest, and an RGB color space for the second region of interest is obtained. at least a portion of a Red channel pixel value, a Green channel pixel value, and a Blue channel pixel value in a color space may be obtained, wherein the Red channel pixel value in an RGB color space for the third region of interest; At least some of the Green channel pixel values and the Blue channel pixel values are obtained, but are not so limited.

Also, referring to FIG. 32, at least one color channel value (3340) can be obtained to obtain a plurality of biometrics according to one embodiment.

At this time, the color channel value (3340) may mean the average value of the pixel values of the color channel, or may mean the processed value. I will omit it.

Also, the color channel values (3340) can be obtained considering the biometrics obtained.

More specifically, at least two or more color channel values can be obtained for the first region of interest for obtaining oxygen saturation and heart rate.

For example, for the first region of interest, at least two of color channel values such as a Red channel value, a Green channel value, a Blue channel value in an RGB color space, a Hue channel value in an HSV color space, a Saturation channel value, and a Value channel value. The above color channel values are obtained, but are not limited thereto.

Also, for example, for the first region of interest, a G-R value that is the difference between the Red channel value and the Green channel value, a G-B value that is the difference between the Green channel value and the Blue channel value, and a difference value between the Hue channel value and the Value channel value At least two or more of the color channel values, such as H-V values, are obtained, but are not limited thereto.

Also, the at least two or more color channel values can be selected considering the absorbance of hemoglobin and oxyhemoglobin to obtain oxygen saturation.

For example, a Blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin and a Red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. Selected for the Blue channel value, but not limited thereto.

Also, the at least two or more color channel values for obtaining heart rate may be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference value between the Green channel value where the absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value where the absorption by hemoglobin and oxyhemoglobin relatively decreases, and the relative Generally, the G-B value, which is the difference value between the Green channel value at which absorption is increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto.

Also, for example, to reduce noise, the difference value between the Green channel value that relatively reflects changes due to heart beat, and the Red channel value that reflects relatively less change due to heart beat, and the Blue channel value. Certain G-R and G-B values are chosen, but are not limited thereto.

Also, at least two color channel values can be obtained for the second and third regions of interest for obtaining blood pressure.

For example, Red channel value, Green channel value, Blue channel value in RGB color space, Hue channel value, Saturation channel value, Value channel value in HSV color space, etc. for the second and third regions of interest for obtaining blood pressure. , at least two color channel values are obtained, but are not limited thereto.

Also, for example, for the second and third regions of interest for obtaining blood pressure, the G-R value that is the difference value between the Red channel value and the Green channel value, the G-B value that is the difference value between the Green channel value and the Blue channel value, At least two color channel values are obtained, such as, but not limited to, an H-V value that is the difference between the Hue channel value and the Value channel value.

Also, the at least two color channel values may be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference value between the Green channel value where the absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value where the absorption by hemoglobin and oxyhemoglobin relatively decreases, and the relative Generally, the G-B value, which is the difference value between the Green channel value at which absorption is increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto.

Also, for example, in order to reduce noise, the difference value between the Green channel value that relatively reflects changes due to heart beat, the Red channel value that reflects relatively less changes due to heart beat, and the Blue channel value G-R and G-B values are selected that are, but are not limited to.

Also, referring to FIG. 32, at least one or more time-series data (3350) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, the time-series data (3350) can be obtained in consideration of the obtained biometric index.

More specifically, at least two or more time series data can be acquired for the first region of interest for acquiring oxygen saturation and heart rate. For example, first time-series data and second time-series data may be acquired to acquire oxygen saturation for the first region of interest, and third time-series data may be acquired to acquire heart rate. can.

At this time, the first time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest is the Red channel value, the first time-series data is obtained based on the Red channel value, but is not limited thereto.

Also, the first time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the first time-series data is obtained based on the Red channel values obtained for the first image frame, the first time-series data is obtained for the first image frame group including the first image frame. obtained, but not limited to.

Also, the second time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest is the Blue channel value, the second time-series data is obtained based on the Blue channel value, but is not limited thereto.

Also, the second time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the second time-series data is obtained based on the Blue channel value obtained for the second image frame, the second time-series data is obtained for the second image frame group including the second image frame. obtained, but not limited to.

Also, the third time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R values and G-B values, the third time-series data are obtained based on the G-R values and G-B values, but are not limited thereto. .

Also, the third time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the third time-series data is obtained based on the G-R values and the G-B values obtained for the third image frame, the third time-series data is the third image frame group including the third image frame. is obtained for, but not limited to.

Also, the third time-series data can be obtained based on characteristic values obtained for the first region of interest. For example, the third time-series data is obtained based on characteristic values obtained based on G-R values and G-B values obtained for the first region of interest, It is not limited to this.

Also, the first, second and third image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may at least partially overlap each other.

Also, at least one time-series data can be acquired for each of the second region of interest and the third region of interest to obtain blood pressure. For example, fourth time-series data is acquired for the second region of interest, and fifth time-series data is acquired for the third region of interest, but the present invention is not limited thereto.

At this time, the fourth time-series data can be obtained based on the color channel values obtained for the second region of interest. For example, if the color channel values obtained for the second region of interest are G-R values and G-B values, the fourth time-series data are obtained based on the G-R values and G-B values, but are not limited thereto. .

Also, the fourth time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the fourth time-series data is obtained based on the G-R values and the G-B values obtained for the fourth image frame, the fourth time-series data is the fourth image frame group including the fourth image frame. is obtained for, but not limited to.

Also, the fourth time-series data can be obtained based on characteristic values obtained for the second region of interest. For example, the fourth time-series data is obtained based on the characteristic value obtained based on the G-R value and the characteristic value obtained based on the G-B value obtained for the second region of interest, It is not limited to this.

Also, the fifth time-series data can be obtained based on the color channel values obtained for the third region of interest. For example, if the color channel values obtained for the third region of interest are G-R values and G-B values, the fifth time-series data are obtained based on the G-R values and G-B values, but are not limited thereto. .

Also, the fifth time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the fifth time-series data is obtained based on the G-R values and the G-B values obtained for the fifth image frame, the fifth time-series data is the fifth image frame group including the fifth image frame. is obtained for, but not limited to.

Also, the fifth time-series data can be obtained based on characteristic values obtained for the third region of interest. For example, the fifth time-series data is obtained based on the characteristic value obtained based on the G-R value and the characteristic value obtained based on the G-B value obtained for the third region of interest, It is not limited to this.

In addition, the first, second, third, fourth and fifth image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may overlap each other at least partially. May be wrapped.

Also, referring to FIG. 32, at least one feature (3360) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, at least one or more of the features (3360) can be obtained in view of the biometrics obtained.

More specifically, at least one or more features can be acquired for the first region of interest for acquiring oxygen saturation and heart rate. For example, a first feature can be acquired for the first region of interest to obtain oxygen saturation, and a second feature can be acquired for the first region of interest to obtain heart rate.

At this time, the first feature can be obtained based on color channel values or time-series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are a Red channel value and a Blue channel value, then the first feature is obtained based on the Red channel value and the Blue channel value. is not limited to

Also, for example, the first feature is acquired based on first time-series data and second time-series data acquired for the first region of interest, but is not limited thereto.

Also, the first feature is a feature for obtaining oxygen saturation. For example, the first feature may be AC and DC values obtained based on the first time-series data, and AC and DC values obtained based on the second time-series data. is not limited to

Further, for example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second It includes, but is not limited to, at least one of the difference between the average maximum value and the average minimum value obtained based on the time-series data, and the average value obtained based on the second time-series data.

Also, the first feature may include a plurality of features. For example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second time-series data, It includes, but is not limited to, at least two or more of the difference between the average of the maximum values and the average of the minimum values obtained based on the data, and the average value obtained based on the second time-series data.

Also, the second feature can be obtained based on color channel values or time series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R and G-B values, the second feature is obtained based on the G-R and G-B values, but is not limited thereto. .

Also, for example, the second feature is acquired based on the third time-series data acquired for the first region of interest, but is not limited thereto.

Also, the second feature is a feature for obtaining a heart rate. For example, the second feature may be, but is not limited to, the frequency value, the wavelength value, or the number of repetitions of the measurement time period obtained based on the third time-series data.

Also, at least one or more features can be obtained for the second and third regions of interest for obtaining blood pressure. For example, a third feature can be acquired for the second and third regions of interest to obtain blood pressure.

At this time, the third feature can be obtained based on color channel values or time-series data obtained for the second and third regions of interest. For example, if the color channel values obtained for the second and third regions of interest are G-R values and G-B values, the third feature is obtained based on the G-R values and the G-B values. is not limited to

Also, for example, the third feature is acquired based on the fourth time-series data acquired for the second region of interest and the fifth time-series data acquired for the third region of interest. is not limited to

Also, the third feature is a feature for obtaining blood pressure. For example, the third feature is the slope value, the maximum value, the minimum value, the average maximum value, the average minimum value, the difference value between the average maximum value and the average minimum value obtained based on the fourth and fifth time-series data. etc., but is not limited thereto.

Further, for example, the third feature may be the time difference between the fourth and fifth time series data, the time difference between maximum values, the time difference between minimum values, the time difference between inflection points, etc. Good, but not limited to this.
Also, the third feature may include a plurality of features. For example, the third feature may include at least two or more of the features described above, but is not limited thereto.

Also, referring to FIG. 32, at least one biometric index (3370) can be obtained to obtain a plurality of biometric indices according to one embodiment.

More specifically, each biometric index can be obtained based on the corresponding features. For example, oxygen saturation can be obtained based on the first feature, heart rate can be obtained based on the second feature, and blood pressure can be obtained based on the third feature. is not limited to

At this time, the above mathematical formula may be used to obtain the oxygen saturation based on the first characteristic, and repeated description will be omitted.

In addition, the above mathematical formula may be used to obtain the heart rate based on the second feature, and repeated description will be omitted.

Also, the above mathematical formula may be used to obtain the blood pressure based on the third feature, and the repeated description will be omitted.

In addition, when there are a plurality of biometric indices 3370, each biometric index is obtained at the same point in time, but is not limited thereto, and may be obtained at different times. For example, oxygen saturation and heart rate are obtained 6 seconds after measurement, and blood pressure is obtained 8 seconds after measurement, but are not limited thereto.

In addition, when the biometric index 3370 is plural, each biometric index can be obtained based on a group of image frames. For example, oxygen saturation can be obtained based on the sixth image frame group, heart rate can be obtained based on the seventh image frame group, and blood pressure can be obtained based on the eighth image frame group.

In addition, each image frame group for obtaining the biometric index (3370) may be identical to each other, but is not limited thereto, may be different from each other, and may be at least partially overlapped with each other. good too. For example, the sixth, seventh and eighth image frame groups may be identical to each other, different from each other, or at least partially overlapped with each other.

Also, at least one preliminary biometric index may be obtained to obtain the biometric index (3370). For example, at least four preliminary heart rates can be obtained to obtain the heart rate.

However, the method of acquiring the heart rate or the biometric index using such a preliminary heart rate or biometric index is the same as described above, so a repeated description will be omitted.

Also, the number of preliminary biometrics for obtaining the biometrics (3370) may be the same or different for each biometric. For example, the heart rate reserve for obtaining the heart rate may be at least 4, and the oxygen saturation reserve and blood pressure reserve for obtaining the oxygen saturation and the blood pressure may be at least 2. Good, but not limited to this.

7.3 Multiple Biometrics Acquisition Method According to an Embodiment FIG. 33 is a diagram for explaining a multiple biometrics acquisition method according to an embodiment.

Referring to FIG. 33, an image frame (3410) can be acquired to acquire multiple biometrics according to one embodiment.

At this time, the image frame 3410 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, the image frame 3410 includes a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 33, at least one region of interest 3420 can be set to acquire a plurality of biometrics according to one embodiment. More specifically, a first ROI and a second ROI can be set.

At this time, the first region of interest may be a region of interest for obtaining oxygen saturation, heart rate and blood pressure, and the first and second regions of interest may be regions of interest for obtaining blood pressure. may

Also, the first and second regions of interest may be the same or different.

Also, the first and second regions of interest may at least partially overlap each other.

Also, the size and area of the first and second regions of interest can be set based on the biometrics to be acquired. For example, to obtain heart rate and oxygen saturation, the first region of interest may be sized to include the subject's cheek region to better sense changes in blood due to heart beats. In order to obtain blood pressure, the first and second regions of interest may be set along the direction of blood flow to better reflect the flow of blood, but are not limited thereto. , the first and second regions of interest may be set to various sizes and regions.

Also, referring to FIG. 33, at least one pixel value (3430) can be obtained to obtain a plurality of biometric indices according to one embodiment.

Also, the pixel values 3430 may be obtained for at least one image frame among the plurality of obtained image frames.

More specifically, at least some of a Red channel pixel value, a Green channel pixel value and a Blue channel pixel value in an RGB color space are obtained for the first region of interest, and an RGB color space for the second region of interest is obtained. At least some of the Red channel pixel value, the Green channel pixel value, and the Blue channel pixel value according to the color space are obtained, but the embodiment is not limited thereto.

Also, referring to FIG. 33, at least one or more color channel values (3440) can be obtained to obtain multiple biometrics according to one embodiment.

At this time, the color channel value (3440) may mean the average value of the pixel values of the color channel or may mean the processed value, and the detailed description thereof will be repeated as described above. will be omitted.

Also, the color channel values (3440) can be obtained considering the biometrics obtained.

More specifically, at least two or more color channel values can be acquired for the first region of interest for acquiring oxygen saturation, heart rate and blood pressure.

For example, for the first region of interest, at least two of color channel values such as a Red channel value, a Green channel value, a Blue channel value in an RGB color space, a Hue channel value in an HSV color space, a Saturation channel value, and a Value channel value. Color channel values on this are obtained, but are not limited thereto.

Also, for example, for the first region of interest, the G-R value that is the difference between the Red channel value and the Green channel value, the G-B value that is the difference between the Green channel value and the Blue channel value, and the difference between the Hue channel value and the Value channel value At least two or more color channel values are obtained, such as, but not limited to, H-V values.

Also, the at least two or more color channel values can be selected considering the absorbance of hemoglobin and oxyhemoglobin to obtain oxygen saturation.

For example, a Blue channel in which the absorbance of oxyhemoglobin is higher than the absorbance of hemoglobin and a Red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. Selected for the Blue channel value, but not limited thereto.

Also, the at least two or more color channel values for obtaining heart rate may be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference value between the Green channel value where the absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value where the absorption by hemoglobin and oxyhemoglobin relatively decreases, and the relative Generally, the G-B value, which is the difference value between the Green channel value at which absorption is increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto.

Also, for example, in order to reduce noise, the difference value between the Green channel value that relatively reflects changes due to heart beat, the Red channel value that reflects relatively less changes due to heart beat, and the Blue channel value G-R and G-B values are selected that are, but are not limited to.

Also, at least two color channel values can be obtained for the first and second regions of interest for obtaining blood pressure.

For example, Red channel value, Green channel value, Blue channel value in RGB color space, Hue channel value, Saturation channel value, Value channel value in HSV color space, etc. for the first and second regions of interest for obtaining blood pressure. , at least two color channel values are obtained, but are not limited thereto.

Further, for example, for the first and second regions of interest for obtaining blood pressure, the G-R value that is the difference value between the Red channel value and the Green channel value, the G-B value that is the difference value between the Green channel value and the Blue channel value, Hue At least two color channel values are obtained, such as, but not limited to, an H-V value that is the difference between the channel value and the Value channel value.

Also, the at least two color channel values may be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference value between the Green channel value where the absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value where the absorption by hemoglobin and oxyhemoglobin relatively decreases, and the relative Generally, the G-B value, which is the difference value between the Green channel value at which absorption is increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto.

Also, for example, in order to reduce noise, the difference value between the Green channel value that relatively reflects changes due to heart beat, the Red channel value that reflects relatively less changes due to heart beat, and the Blue channel value G-R and G-B values are selected that are, but are not limited to.

Also, referring to FIG. 33, at least one or more time-series data (3450) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, the time-series data (3450) can be acquired in consideration of the acquired biometric index.

More specifically, at least two or more time-series data can be acquired for the first region of interest. For example, a first time-series data and a second time-series data are acquired to acquire oxygen saturation for the first region of interest, and a third time-series data is acquired to acquire heart rate and blood pressure. be able to.

At this time, the first time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest is the Red channel value, the first time-series data is obtained based on the Red channel value, but is not limited thereto.

Also, the first time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the first time-series data is obtained based on the Red channel values obtained for the first image frame, the first time-series data is obtained for the first image frame group including the first image frame. obtained, but not limited to.

Also, the second time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest is the Blue channel value, the second time-series data is obtained based on the Blue channel value, but is not limited thereto.

Also, the second time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the second time-series data is obtained based on the Blue channel value obtained for the second image frame, the second time-series data is obtained for the second image frame group including the second image frame. obtained, but not limited to.

Also, the third time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R values and G-B values, the third time-series data are obtained based on the G-R values and G-B values, but are not limited thereto. .

Also, the third time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the third time-series data is obtained based on the G-R values and the G-B values obtained for the third image frame, the third time-series data is the third image frame group including the third image frame. is obtained for, but not limited to.

Also, the third time-series data can be obtained based on characteristic values obtained for the first region of interest. For example, the third time-series data is obtained based on characteristic values obtained based on G-R values and G-B values obtained for the first region of interest, It is not limited to this.

At least one time-series data is acquired for the second region of interest to acquire blood pressure, and third time-series data is acquired for the first region of interest, and for the second region of interest. Blood pressure can be obtained based on the obtained fourth time-series data.

At this time, the fourth time-series data can be obtained based on the color channel values obtained for the second region of interest. For example, if the color channel values obtained for the second region of interest are G-R values and G-B values, the fourth time-series data are obtained based on the G-R values and G-B values, but are not limited thereto. .

Also, the fourth time-series data may be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the fourth time-series data is obtained based on the G-R values and the G-B values obtained for the fourth image frame, the fourth time-series data is the fourth image frame group including the fourth image frame. is obtained for, but not limited to.

Also, the fourth time-series data can be obtained based on characteristic values obtained for the second region of interest. For example, the fourth time-series data is obtained based on the characteristic value obtained based on the G-R value and the characteristic value obtained based on the G-B value obtained for the second region of interest, It is not limited to this.

Also, the first, second, third and fourth image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may overlap each other at least partially. good.

Also, referring to FIG. 33, at least one feature (3460) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, at least one or more of the features (3460) can be obtained in consideration of the obtained biometrics.

More specifically, at least one feature can be obtained for the first region of interest. For example, a first feature can be acquired for the first region of interest to obtain oxygen saturation, and a second feature can be acquired for the first region of interest to obtain heart rate.

At this time, the first feature can be obtained based on color channel values or time-series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are a Red channel value and a Blue channel value, then the first feature is obtained based on the Red channel value and the Blue channel value. is not limited to

Also, for example, the first feature is acquired based on first time-series data and second time-series data acquired for the first region of interest, but is not limited thereto.

Also, the first feature is a feature for obtaining oxygen saturation. For example, the first feature may be an AC value and a DC value obtained based on the first time-series data, and an AC value and a DC value obtained based on the second time-series data. is not limited to

Further, for example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second It includes, but is not limited to, at least one of the difference between the average maximum value and the average minimum value obtained based on the time-series data, and the average value obtained based on the second time-series data.

Also, the first feature may include a plurality of features. For example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second time-series data, It includes, but is not limited to, at least two or more of the difference between the average of the maximum values and the average of the minimum values obtained based on the data, and the average value obtained based on the second time-series data.

Also, the second feature can be obtained based on color channel values or time series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R and G-B values, the second feature is obtained based on the G-R and G-B values, but is not limited thereto. .

Also, for example, the second feature is acquired based on the third time-series data acquired for the first region of interest, but is not limited thereto.

Also, the second feature is a feature for obtaining a heart rate. For example, the second feature may be, but is not limited to, the frequency value, the wavelength value, or the number of repetitions of the measurement time period obtained based on the third time-series data.

Also, at least one or more features can be obtained for the first and second regions of interest for obtaining blood pressure. For example, a third feature can be acquired for the first and second regions of interest to obtain blood pressure.

At this time, the third feature can be obtained based on color channel values or time-series data obtained for the first and second regions of interest. For example, if the color channel values obtained for the first and second regions of interest are G-R values and G-B values, the third feature is obtained based on the G-R values and the G-B values. is not limited to

Also, for example, the third feature is acquired based on the third time-series data acquired for the first region of interest and the fourth time-series data acquired for the second region of interest. is not limited to

Also, the third feature is a feature for obtaining blood pressure. For example, the third feature is the gradient value, maximum value, minimum value, local maximum average, local minimum average, local maximum average, and local minimum average difference obtained based on the third and fourth time-series data. value, etc., but is not limited thereto.

Also, for example, the third feature can be the time difference between the third and fourth time-series data, the time difference between maximum values, the time difference between minimum values, the time difference between inflection points, and the like. is not limited to
Also, the third feature may include a plurality of features. For example, the third feature may include at least two or more of the features described above, but is not limited thereto.

Also, referring to FIG. 33, at least one biometric index (3470) can be obtained to obtain a plurality of biometric indices according to one embodiment.

More specifically, each biometric index can be obtained based on the corresponding features. For example, oxygen saturation can be obtained based on the first feature, heart rate can be obtained based on the second feature, and blood pressure can be obtained based on the third feature. is not limited to

At this time, the above mathematical formula may be used to obtain the oxygen saturation based on the first characteristic, and repeated description will be omitted.

In addition, the above mathematical formula may be used to obtain the heart rate based on the second feature, and repeated description will be omitted.

Also, the above mathematical formula may be used to obtain the blood pressure based on the third feature, and the repeated description will be omitted.

In addition, when there are a plurality of biometric indices 3470, each biometric index is obtained at the same point in time, but is not limited thereto, and may be obtained at different times. For example, oxygen saturation and heart rate are obtained 6 seconds after measurement, and blood pressure is obtained 8 seconds after measurement, but are not limited thereto.

In addition, when the biometric index 3470 is plural, each biometric index can be obtained based on a group of image frames. For example, oxygen saturation can be obtained based on the fifth image frame group, heart rate can be obtained based on the sixth image frame group, and blood pressure can be obtained based on the seventh image frame group. can be done.

In addition, each image frame group for obtaining the biometric index (3470) may be the same as each other, but is not limited thereto, and may be different from each other, and at least partially overlap each other. may For example, the fifth, sixth and seventh image frame groups may be identical to each other, different from each other, or at least partially overlapped with each other.

Also, at least one preliminary biometric index may be obtained to obtain the biometric index (3470). For example, at least four preliminary heart rates can be obtained to obtain the heart rate.

However, the method of obtaining the heart rate or the biometric index using such a preliminary heart rate or biometric index is the same as described above, so a repeated explanation will be omitted.

Also, the number of preliminary biometrics for acquiring the biometrics (3470) may be the same or different for each biometric. For example, the heart rate reserve for obtaining the heart rate may be at least 4, and the oxygen saturation reserve and blood pressure reserve for obtaining the oxygen saturation and the blood pressure may be at least 2. Good, but not limited to this.

7.4 Multiple Biometrics Acquisition Method According to an Embodiment FIG. 34 is a diagram for explaining a multiple biometrics acquisition method according to an embodiment.

Referring to FIG. 34, an image frame (3510) can be acquired to acquire multiple biometrics according to one embodiment.

At this time, the image frame 3510 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, the image frame 3510 includes a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 34, at least one region of interest 3520 can be set to acquire a plurality of biometric indices according to one embodiment. More specifically, a first region of interest can be set.

At this time, the first region of interest is a region of interest for obtaining oxygen saturation, heart rate and blood pressure.

Also, the size and area of the first region of interest can be set based on the biometrics to be acquired. For example, to obtain heart rate and oxygen saturation, the first region of interest is sized to include the subject's cheek region to better sense changes in blood due to heartbeat. However, the first region of interest may be set to various sizes and regions, but not limited thereto.

Also, referring to FIG. 34, at least one pixel value (3530) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, the pixel values 3530 may be obtained for at least one image frame among the plurality of obtained image frames.

More specifically, at least some of the Red channel pixel values, the Green channel pixel values, and the Blue channel pixel values in the RGB color space are obtained for the first region of interest, but the invention is not limited thereto. .

Also, referring to FIG. 34, at least one color channel value (3540) can be obtained to obtain a plurality of biometrics according to one embodiment.

At this time, the color channel value (3540) may mean the average value of the pixel values of the color channel or may mean the processed value, and the detailed description thereof will be repeated as described above. will be omitted.

Also, the color channel values (3540) can be obtained considering the biometrics obtained.

More specifically, at least two or more color channel values can be acquired for the first region of interest for acquiring oxygen saturation, heart rate and blood pressure.

For example, for the first region of interest, at least two of color channel values such as a Red channel value, a Green channel value, a Blue channel value in an RGB color space, a Hue channel value in an HSV color space, a Saturation channel value, and a Value channel value. The above color channel values are obtained, but are not limited thereto.

Also, for example, for the first region of interest, the G-R value that is the difference between the Red channel value and the Green channel value, the G-B value that is the difference between the Green channel value and the Blue channel value, and the difference between the Hue channel value and the Value channel value At least two or more color channel values are obtained, such as, but not limited to, H-V values.

Also, the at least two or more color channel values can be selected considering the absorbance of hemoglobin and oxyhemoglobin to obtain oxygen saturation.

For example, a Blue channel in which the oxyhemoglobin absorbance is higher than the hemoglobin absorbance and a Red channel in which the oxyhemoglobin absorbance is lower than the hemoglobin absorbance may be selected, whereby the at least two color channel values are the Red channel values. and Blue channel values, but are not limited thereto.

Also, the at least two or more color channel values for obtaining heart rate and blood pressure can be selected to reduce noise due to motion, external light, and the like.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value in which absorption by hemoglobin and oxyhemoglobin relatively increases and the Red channel value in which absorption by hemoglobin and oxyhemoglobin relatively decreases, and The G-B value, which is the difference between the Green channel value at which absorption is relatively increased by hemoglobin and oxyhemoglobin and the Blue channel value at which absorption is relatively decreased by hemoglobin and oxyhemoglobin, is selected, but is not limited thereto. .

Also, for example, in order to reduce noise, the difference between the Green channel value, which relatively reflects changes due to heart beat, and the Red channel value, which reflects relatively less changes due to heart beat, and the Blue channel value Values G-R and G-B are selected, but are not limited thereto.

Also, referring to FIG. 34, at least one or more time-series data (3550) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, the time-series data (3550) can be acquired in consideration of the acquired biometric index.

More specifically, at least two or more time-series data can be acquired for the first region of interest. For example, a first time-series data and a second time-series data are acquired to acquire oxygen saturation for the first region of interest, and a third time-series data is acquired to acquire heart rate and blood pressure. be able to.

At this time, the first time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest is the Red channel value, the first time-series data is obtained based on the Red channel value, but is not limited thereto.

Also, the first time-series data can be acquired for at least a partial image frame group among the plurality of acquired image frames.

For example, when the first time-series data is obtained based on the Red channel values obtained for the first image frame, the first time-series data is obtained for the first image frame group including the first image frame. obtained, but not limited to.

Also, the second time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel value obtained for the first region of interest is the Blue channel value, the second time-series data is obtained based on the Blue channel value, but is not limited thereto.

Also, the second time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the second time-series data is acquired based on the Blue channel value acquired for the second image frame, the second time-series data is acquired for the second image frame group including the second image frame. is obtained for, but not limited to.

Also, the third time-series data can be obtained based on the color channel values obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R values and G-B values, the third time-series data are obtained based on the G-R values and G-B values, but are not limited thereto. .

Also, the third time-series data can be obtained for at least a partial image frame group among the plurality of obtained image frames.

For example, when the third time-series data is obtained based on the G-R values and the G-B values obtained for the third image frame, the third time-series data is the third image frame group including the third image frame. is obtained for, but not limited to.

Also, the third time-series data can be obtained based on characteristic values obtained for the first region of interest. For example, the third time-series data is obtained based on characteristic values obtained based on G-R values and G-B values obtained for the first region of interest, It is not limited to this.

Also, the first, second and third image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may overlap each other at least partially. .

Also, referring to FIG. 34, at least one feature (3560) can be obtained to obtain a plurality of biometrics according to one embodiment.

Also, at least one or more of the features (3560) can be obtained in consideration of the biometrics obtained.

More specifically, at least one feature can be obtained for the first region of interest. For example, a first feature can be acquired for the first region of interest to obtain oxygen saturation, a second feature can be acquired for the first region of interest to obtain heart rate, and blood pressure can be obtained. A third feature is obtained for the first region of interest for, but not limited to, the first region of interest.

At this time, the first feature can be obtained based on color channel values or time-series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are a Red channel value and a Blue channel value, then the first feature is obtained based on the Red channel value and the Blue channel value. is not limited to

Also, for example, the first feature is acquired based on first time-series data and second time-series data acquired for the first region of interest, but is not limited thereto.

Also, the first feature is a feature for obtaining oxygen saturation. For example, the first feature may be an AC value and a DC value obtained based on the first time-series data, and an AC value and a DC value obtained based on the second time-series data. is not limited to

Further, for example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second It includes, but is not limited to, at least one of the difference between the average maximum value and the average minimum value obtained based on the time-series data, and the average value obtained based on the second time-series data.

Also, the first feature may include a plurality of features. For example, the first feature may be the difference between the average of local maximum values and the average of local minimum values obtained based on the first time-series data, the average value obtained based on the first time-series data, the second time-series data, It includes, but is not limited to, at least two or more of the difference between the average of the maximum values and the average of the minimum values obtained based on the data, and the average value obtained based on the second time-series data.

Also, the second feature can be obtained based on color channel values or time series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R and G-B values, the second feature is obtained based on the G-R and G-B values, but is not limited thereto. .

Also, for example, the second feature is acquired based on the third time-series data acquired for the first region of interest, but is not limited thereto.

Also, the second feature is a feature for obtaining a heart rate. For example, the second feature may be, but is not limited to, the frequency value, the wavelength value, or the number of repetitions of the measurement time period obtained based on the third time-series data.

Also, the third feature can be obtained based on color channel values or time series data obtained for the first region of interest. For example, if the color channel values obtained for the first region of interest are G-R and G-B values, the third feature is obtained based on the G-R and G-B values, but is not limited thereto. .

Also, for example, the third feature is acquired based on the third time-series data acquired for the first region of interest, but is not limited thereto.

Also, the third feature is a feature for obtaining blood pressure. For example, the third feature is a slope value, maximum value, minimum value, average maximum value, average minimum value, difference value between the average maximum value and the average minimum value, etc. obtained based on the third time-series data. may be, but is not limited to.

Also, the third feature may include a plurality of features. For example, the third feature is at least one of the slope value, maximum value, minimum value, average maximum value, average minimum value, average maximum value, and average minimum value obtained based on the third time-series data. Including but not limited to two or more.

Also, referring to FIG. 34, at least one biometric index (3570) can be obtained to obtain a plurality of biometric indices according to one embodiment.

More specifically, each biometric index can be obtained based on the corresponding features. For example, oxygen saturation can be obtained based on the first feature, heart rate can be obtained based on the second feature, and blood pressure can be obtained based on the third feature. is not limited to

At this time, the above mathematical formula may be used to obtain the oxygen saturation based on the first characteristic, and repeated description will be omitted.

In addition, the above mathematical formula may be used to obtain the heart rate based on the second feature, and repeated description will be omitted.

Also, the above mathematical formula may be used to obtain the blood pressure based on the third feature, and the repeated description will be omitted.

In addition, when there are a plurality of biometric indices 3570, each biometric index is obtained at the same point in time, but is not limited thereto, and may be obtained at different times. For example, oxygen saturation and heart rate are obtained 6 seconds after measurement, and blood pressure is obtained 8 seconds after measurement, but are not limited thereto.

In addition, when the biometric index 3570 is plural, each biometric index can be obtained based on a group of image frames. For example, oxygen saturation can be obtained based on the fourth image frame group, heart rate can be obtained based on the fifth image frame group, and blood pressure can be obtained based on the sixth image frame group. can be done.

In addition, each image frame group for obtaining the biometric index (3570) may be identical to each other, but is not limited thereto, may be different from each other, and may overlap each other at least partially. be able to. For example, the fourth, fifth and sixth image frame groups may be identical to each other, different from each other, and at least partially overlapped with each other.

Also, at least one preliminary biometric index may be obtained to obtain the biometric index (3570). For example, at least four preliminary heart rates can be obtained to obtain the heart rate.

However, the method of obtaining the heart rate or the biometric index using such a preliminary heart rate or biometric index is the same as described above, so a repeated explanation will be omitted.

Also, the number of preliminary biometrics for obtaining the biometrics (3570) may be the same or different for each biometric. For example, the heart rate reserve for obtaining the heart rate may be at least 4, and the oxygen saturation reserve and blood pressure reserve for obtaining the oxygen saturation and the blood pressure may be at least 2. Good, but not limited to this.

7.5 Multiple Biometrics Acquisition Method According to an Embodiment FIG. 35 is a diagram for explaining a multiple biometrics acquisition method according to an embodiment.

Referring to FIG. 35, multiple image frames (3600) can be acquired to acquire multiple biometrics according to one embodiment.

At this time, the image frame 3600 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, referring to FIG. 35, at least one or more biometric indices may be obtained based on at least one or more features.

For example, as shown in FIG. 35, a first biometric index (3660) is obtained based on the first feature (3610), a second biometric index (3670) is obtained based on the second feature (3620), A third biometric index (3680) is obtained based on the third feature (3630), and a fourth biometric index (3690) is obtained based on the fourth feature (3640), but is not limited thereto. One biometric index may be obtained based on one feature, and a plurality of biometric indices may be obtained based on one feature.

At this time, the first to fourth features are the AC value, the DC value, the difference between the average maximum value and the average minimum value, the average value, the frequency component value, and the wavelength component value obtained based on at least one time-series data. , value for the number of cycles repeated during the measurement time, slope value, maximum value, minimum value, mean maximum, mean minimum, acquisition time difference, time difference between maximum values, time difference between minimum values, inflection It includes at least one of features such as time difference between points, skin temperature, etc., but is not limited thereto.

Also, the first to fourth bio-indexes include, but are not limited to, at least one of bio-indexes such as heart rate, oxygen saturation, blood pressure, body temperature, and blood flow.

However, as described above, detailed descriptions of the features and biometric indices will be omitted.

Also, the first to fourth features and/or the biometric index can be obtained based on at least a partial image frame group among the plurality of image frames.

For example, the first feature and/or the first biometric index is obtained based on a first group of image frames, and the second feature and/or the second biometric index is obtained based on a second group of image frames, the third feature and/or the third biometric index is obtained based on a third group of image frames, and the fourth feature and/or the fourth biometric index is obtained based on a fourth group of image frames, It is not limited to this.

At this time, the first to fourth image frame groups may include at least two image frames.

For example, as shown in FIG. 35, the first group of image frames may include N image frames, the second group of image frames may wrap M image frames, and the third group of image frames may include N image frames. may include K image frames, and the fourth group of image frames includes, but is not limited to, L-dog image frames.

Also, the number of image frames included in the first to fourth image frame groups may be the same.

For example, the first group of image frames may contain 180 image frames, the second group of image frames may contain 180 image frames, and the third group of image frames may contain 180 image frames. Alternatively, the fourth image frame group includes 180 image frames, but is not limited thereto.

In this case, the first to fourth bioindexes are obtained or output at the same time, but are not limited thereto, and may be obtained or output at different times.

Also, in this case, the first to fourth bioindexes can be acquired from the same state of the subject, and the respective bioindexes can be associated with each other.

Also, the number of image frames included in the first to fourth image frame groups may be different from each other.

For example, the first group of image frames may contain 180 image frames, the second group of image frames may contain 90 image frames, and the third group of image frames may contain 240 image frames. Alternatively, the fourth image frame group includes 360 image frames, but is not limited thereto.

In this case, the first to fourth bioindexes are obtained or output at different times, but are not limited thereto, and may be obtained or output at the same time.

Also, the number of image frames included in the first to fourth image frame groups may vary according to a biometric index to be obtained.

For example, if the first biometric index is heart rate, the second biometric index is oxygen saturation, the third biometric index is blood pressure, and the fourth biometric index is body temperature, the first image frame group includes 180 image frames. The second group of image frames may contain 120 image frames, the third group of image frames may contain 60 image frames, and the fourth group of image frames may contain 60 image frames. However, it is not limited thereto, and the number of image frames included in the image frame group can be set in consideration of the biometric index to be obtained.

FIG. 36 is a diagram for explaining a method of obtaining multiple relevant biometrics according to one embodiment.

Referring to FIG. 36, according to one embodiment, multiple image frames (3700) can be acquired to acquire relevant multiple biometrics.

At this time, the image frame 3700 may be an image frame obtained from a visible light image, an infrared image, etc., but is not limited thereto.

Also, referring to FIG. 36, at least one or more biometric indices can be obtained based on at least one or more image frame groups.

More specifically, as shown in FIG. 36, a first biometric index (3760) is obtained based on a first group of image frames (3710) and a second biometric index (3770) is obtained based on a second group of image frames (3720). ), a third biometric index (3780) is obtained based on the third group of image frames (3730), and a fourth biometric index (3790) is obtained based on the fourth group of image frames (3740). can be obtained based on

At this time, since the above-described contents are applied to each image frame group and each biometric index, repeated description will be omitted.

Referring to FIG. 36, the first biometric index (3710), the second biometric index (3720), the third biometric index (3730) and the fourth biometric index (3740) are related to each other. To be acquired, the first group of image frames (3710), the second group of image frames (3720), the third group of image frames (3730) and the fourth group of image frames (3740) are at least one of each other. Parts may overlap.

For example, as shown in FIG. 36, the first image frame group (3710) may include the first image frame to the 13th image frame, and the second image frame group (3720) may include the sixth image frame to the thirteenth image frame. The third image frame group (3730) may include the 4th to 17th image frames, and the fourth image frame group (3740) may include the 9th to 23rd image frames. It may include, but is not limited to, frames.

Therefore, the first to fourth image frame groups (3710, 3720, 3730, 3740) can commonly include the ninth to thirteenth image frames.

In addition, as described above, the first to fourth biometric indices can be acquired in association with each other by at least partially overlapping each other in the first to fourth image frame groups.

For example, since the first to fourth image frame groups include the ninth to thirteenth image frames in common, the first to fourth biometric indices are acquired including the same state of the subject, The first to fourth bioindexes may be obtained in association with each other.

As a more specific example, when the state of the subject is set to the first state at the time when the ninth image frame is acquired, the first to fourth biometric indices are the first state of the subject. The first to fourth bio-indexes can be acquired while reflecting the state in common, and accordingly, the first to fourth bio-indexes can be acquired in association with each other.

In addition, when the first to fourth biometric indices (3760, 3770, 3780, 3790) are obtained in association with each other, it is possible to obtain a more accurate biometric index of the subject.

Also, at least one biometric information can be obtained based on the first to fourth biometric indices (3760, 3770, 3780, 3790).

In addition, when the biometric information is obtained based on a related biometric index, the accuracy of the biometric information can be improved. For example, when using the heart rate and blood pressure included in the first to fourth bioindexes to obtain hypertension information included in the biometric information, when the heart rate and blood pressure are related to each other, High blood pressure information becomes more accurate.

More specifically, when blood pressure is measured while the subject is exercising or excited, heart rate is measured when the subject is stable, and hypertension information is obtained based on the measured heart rate, A person may be perceived as hypertensive even if they are not hypertensive. However, on the contrary, when the subject is exercising or excited, blood pressure and heart rate are measured, and hypertension information is obtained based on the measured blood pressure and hypertension information, the hypertension information can be obtained more accurately.

FIG. 37 is a diagram for explaining a method for obtaining a plurality of biometric indices according to one embodiment.

Referring to FIG. 37, at least one biometric index can be obtained based on at least one preliminary biometric index.

For example, as shown in FIG. 37, a first bioindex is obtained based on N preliminary first bioindexes, a second bioindex is obtained based on M preliminary second bioindexes, and a third A bioindex can be obtained based on the K preliminary third bioindexes, and a fourth bioindex can be obtained based on the L preliminary fourth bioindexes.

However, since the above-described method is applied to the method of obtaining the biometric index using the preliminary biometric index, repeated description will be omitted.

The number of preliminary first to fourth bioindexes for obtaining each of the first to fourth bioindexes may be the same.

For example, the preliminary first bioindex for obtaining the first bioindex may be four, and the preliminary second bioindex for obtaining the second bioindex may be four. The preliminary third bioindex for obtaining the third bioindex may be four, and the preliminary fourth bioindex for obtaining the fourth bioindex may be four. Good, but not limited to this.

Also, the number of preliminary first to fourth bioindexes for obtaining the first to fourth bioindexes may be different from each other.

For example, the preliminary first bioindex for obtaining the first bioindex may be four, and the preliminary second bioindex for obtaining the second bioindex may be two. The preliminary third bioindex for obtaining the third bioindex may be two, and the preliminary fourth bioindex for obtaining the fourth bioindex may be one. Well, you are not limited to this.

In addition, the number of preliminary first to fourth bioindexes for obtaining each of the first to fourth bioindexes may vary depending on the bioindex to be obtained.

For example, if the first bioindex is heart rate, the second bioindex is oxygen saturation, the third bioindex is blood pressure, and the fourth bioindex is body temperature, the preliminary first bioindex for obtaining the heart rate may be four, the preliminary second bioindex for obtaining the oxygen saturation may be two, and the preliminary third bioindex for obtaining the blood pressure may be two. There may be one, but not limited to, one preliminary fourth bioindex for obtaining the body temperature, and the underlying preliminary bioindex in consideration of the bioindex to be obtained. can be set.

8. Drowsiness sensing device and method
8.1 Drowsiness Sensing Device A drowsiness sensing device provided herein may mean a device that senses a drowsiness-related condition of a subject. Specifically, the drowsiness sensing device can sense whether the subject is drowsy and the degree of drowsiness.

A person can feel drowsiness during daily life. There are situations in which drowsiness poses a safety hazard. For example, if a driver feels drowsy while driving, he/she will be unable to concentrate on driving, which may cause a traffic accident. In this case, the drowsiness sensing device senses the drowsiness of the driver and informs the driver or manager of the drowsiness, thereby preventing traffic accidents and situations that endanger the safety of the driver. . As such, it goes without saying that drowsiness detection devices are used not only in situations where drowsiness poses a safety hazard, but also in places and fields where drowsiness acts as important information, such as libraries and infant monitoring.
The drowsiness sensing device can sense a drowsiness-related state by means of a method that utilizes a contact-type device that involves physical contact with the person to be measured together with an adhesive electrode sensor or a wearable device. In addition, the drowsiness sensing device can sense the drowsiness of the person to be measured in a non-contact manner using a camera. When the non-contact method is used, the subject can sense the drowsiness of the subject without being restrained by the contact-type device, thereby increasing the convenience of the user and preventing drowsiness in various places and fields. A sensing device is used.
The non-contact drowsiness sensing device can detect drowsiness by detecting abnormal states of the subject, such as measuring the number of times the subject blinks or tracking the position of the pupil. . However, when the subject consciously controls the blink of the eye or the position of the pupil, it becomes difficult to accurately perceive the state of drowsiness. In order to supplement the accuracy, if the information that appears in the subject's body that appears when the subject feels sleepy is used, the sleepiness-related state can be sensed more accurately. In addition, if the subject does not feel drowsiness but will soon feel drowsiness using biometric information, the state of drowsiness can be sensed in advance in the initial stage.

Due to these advantages, the drowsiness sensing device (6000) provided by the specification uses the non-contact biometrics of the subject to determine whether the subject is biocognitive about drowsiness. It is possible to sense whether and to what extent drowsiness corresponds to.

Referring to FIG. 38 according to one embodiment, the drowsiness detection device (6000) includes a heart rate information acquisition unit (6100) that acquires the heart rate of the subject, and the subject based on the acquired heart rate of the subject. A drowsiness sensing unit (6200) that senses drowsiness of the measurer can be included.
Of course, the drowsiness sensing device (6000) of the present invention can further include other configurations of the heartbeat information acquiring section (6100) and the drowsiness sensing section (6200) described above. For example, the drowsiness sensing device (6000) may further include a heartbeat information acquiring unit (6100), a drowsiness sensing unit (6200), and a notification unit (6300). Here, the notification unit (6300) can give a notification to make the person to be measured sleepy or an individual other than the person to be measured based on the result detected by the drowsiness detection unit (6200). Here, an individual who is not a person to be measured may mean a manager who manages the person to be measured or a third party around the person to be measured.

In one embodiment, the drowsiness sensing device 6200 can sense drowsiness of the subject at various times. That is, a part or the entire configuration included in the drowsiness sensing device 6000 can operate at various times, and each step or all steps of various drowsiness sensing methods described below can also be implemented at various times. can be done.

For example, the drowsiness sensing device (6000) can sense the drowsiness of the subject at a predetermined cycle (eg, every minute, every hour). In another example, the drowsiness sensing device (6000) can sense the drowsiness of the subject without following a predetermined time period. For example, the drowsiness sensing device 6000 can sense the drowsiness of the subject in real time whenever information on the heartbeat of the subject is obtained. In addition, the drowsiness sensing device may continuously sense the drowsiness of the subject, and when requesting to sense the drowsiness by inputting a signal for triggering the drowsiness sensing, the drowsiness of the subject may be detected. can be sensed.

In another embodiment, the drowsiness sensing device 6000 can also initiate drowsiness sensing upon first heart rate acquisition. For example, the heart rate information obtaining section (6100) can first obtain the heart rate several seconds or minutes after starting the measurement for obtaining the heart rate of the subject. At this time, the drowsiness sensing device 6000 can start drowsiness sensing of the person to be measured when the heart rate is first obtained.

Hereinafter, with reference to FIG. 38 according to one embodiment, the configuration included in the drowsiness sensing device (6000) will be described in detail.

8.1.1 Heartbeat Information Acquisition Unit In one embodiment, the heartbeat information acquisition unit (6100) acquires information on the heart rate of the subject, such as heart rate or sympathetic nerve activity versus subsympathetic nerve activity of the subject. At least one of the ratios can be obtained.

Here, the heart rate may mean the number of times the heart beats during the reference time. Heart rate generally means the number of heart beats during a reference time of 1 minute, but the reference time can be set arbitrarily. For example, it can be set to 30 seconds or some other length. Also, the ratio of sympathetic to sub-sympathetic activity is attributed to the heartbeat signal, where heartbeat signal may mean a signal that can vary with heart beats, and that heart beats vary. It may mean a signal that can be estimated. In general, the degree of sympathetic nerve activity, the degree of subsympathetic nerve activity, and the drowsiness state of the subject are highly related to the subject. Therefore, the drowsiness sensing device can sense the drowsiness of the subject using the ratio of sympathetic nerve activity to sub-sympathetic nerve activity resulting from heartbeat signals.

In addition, the activity of the sympathetic nerve can appear in the low frequency band of the heartbeat signal, and the activity of the sub-sympathetic nerve can appear in the high frequency band of the heartbeat signal. Accordingly, the ratio of sympathetic to sub-sympathetic activity of the heartbeat signal can be denoted herein as LF/HF. LF may mean a characteristic value in a low frequency band (Low Frequency, LF) of the heartbeat signal, and HF may mean a characteristic value in a high frequency band (High Frequency, HF) of the heartbeat signal.
In one embodiment, a low frequency band and a high frequency band can be divided based on a reference frequency. In one example, the low frequency band and the high frequency band are divided based on a reference frequency of 0.15 Hz, the low frequency band includes signals having frequencies in the range of 0.04-0.15 Hz, and the high frequency band is in the range of 0.15-0.4 Hz. can include a signal having a frequency of Of course, the reference frequency can be set to other numerical values besides 0.15 Hz.

In one embodiment, the heartbeat information obtaining unit 6100 can transform the heartbeat signal into a signal in the frequency domain through various methods such as Fourier transform. By extracting the frequency band (0.15-0.4Hz) signal, the LF/HF values can be obtained.
In one embodiment, the heart rate information acquiring unit 6100 can invasively or non-invasively acquire information on heart rate.
Also, according to another embodiment, the heartbeat information acquisition unit 6100 can acquire information on the heartbeat rate through a contact method or a non-contact method.

Specifically, the heartbeat information acquisition unit (6100) can acquire information on the heartbeat rate by a non-contact method using a device that does not physically contact the subject. For example, the heartbeat information acquiring unit 6100 can acquire an image of the subject using a camera and analyze the acquired image to acquire information on the heart rate of the subject. As described above, a detailed description of the method and apparatus for acquiring information on heart rate using a camera in a non-contact manner will be omitted below.

In addition, the heartbeat information acquisition unit (6100) naturally acquires information on the heartbeat rate by a contact method using a device that physically contacts the subject. For example, the heart rate information acquisition unit 6100 may acquire information on heart rate using a wearable sensor for heart rate measurement or a wearable device including various wearable sensors and optical sensors. . For convenience of explanation, the configuration of the drowsiness sensing device and the drowsiness sensing method will be described below, focusing on an example of obtaining information on heart rate using non-contact means.

In one embodiment, the heart rate information acquisition unit 6100 can acquire information on heart rate by directly measuring it. Specifically, the heartbeat information acquisition unit includes a device for measuring heartbeat, and can acquire information on the heartbeat rate measured by the device for measuring heartbeat. For example, the heart rate information acquisition unit 6100 includes a camera, and can directly measure information on heart rate by analyzing images acquired by the camera.

In addition, the heartbeat information acquisition unit (6100) can also acquire information on the heartbeat rate by receiving information on the heartbeat rate measured by an external device. For example, heart rate information is measured by a device including an electrode sensor for measuring heart rate, and the heart rate information acquisition unit (6100) can receive the measured heart rate information. In addition, the heartbeat information acquisition unit 6100 can receive information on the measured heart rate through another device (eg, server) that mediates between the heartbeat information acquisition unit 6100 and an external device.
In one embodiment, when the drowsiness sensing device (6000) is included in the biometric acquisition device (10), the heart rate information acquisition unit (6100) reads the heart rate stored in the memory of the biometric acquisition device (10). heart rate can be obtained by In a different embodiment, if the drowsiness sensing device (6000) is not included in the biometric acquisition device (10), the heart rate information acquisition unit (6100) may use the heart rate measured by the bioindex acquisition device (10) as a measure of drowsiness. It can be received through a communication unit (not shown) included in the sensing device (6000). Of course, the drowsiness sensing device (6000) can also acquire heart rate from other devices besides the biometric acquisition device (10).

8.1.2 Drowsiness Detector In one embodiment, the drowsiness detector 6200 detects whether the subject is asleep or the degree of drowsiness felt by the subject based on the information on the heart rate. can be done. Specifically, the drowsiness sensing unit 6200 can sense the drowsiness state and steady state of the subject based on information on the heart rate. Here, the drowsiness state can be divided into a plurality of stages.

Herein, a drowsy state may mean a state in which the subject temporarily sleeps or is likely to fall asleep within a predetermined time before falling asleep, and a steady state is a drowsiness state. It may mean a state in which the subject is unlikely to fall asleep within a predetermined period of time.

In one embodiment, the drowsiness state can be divided into multiple levels according to the intensity of drowsiness. The drowsiness state can optionally be classified from the first stage to the Nth stage. For convenience of explanation, the stage of the lowest drowsiness intensity is indicated by 1 stage, and the stage of the highest drowsiness intensity is indicated by N stage in this specification. For example, drowsiness can be divided into stages 1-3. In this case, the first stage may refer to the lowest drowsiness intensity stage and the third stage may refer to the highest drowsiness intensity stage.
In one embodiment, the first-stage drowsiness state may mean that the subject has physically entered the drowsiness state, but the subject is unaware of the drowsiness. For example, information about the subject's heart rate in the stage 1 drowsiness state. It objectively indicates that the subject is in a drowsy state, but there is a possibility that the subject himself/herself cannot feel drowsiness. According to the present specification, the drowsiness state can be indicated as an unconscious drowsiness state or an unconscious drowsiness state.

In one embodiment, the second stage drowsiness state and the third stage drowsiness state may refer to a state in which the subject physically enters the drowsiness state and the subject is aware of the drowsiness. Herein, the condition can be designated as a subjective sleepiness state or a conscious sleepiness state.
In general, the subject will enter an unconscious drowsy state before entering a conscious drowsy state. If the subject receives a drowsiness notification after the subject has entered a subjective drowsiness state, the subject receives the notification while he or she is already asleep, creating a dangerous situation for the subject. Probability is high. In addition, it may take a long time for the person to be measured to get out of the drowsy state even if the drowsy state notification is received before the dangerous situation occurs.

In order to prevent such a dangerous situation, the drowsiness detection device (6000) can sense the non-conscious drowsiness state before the subject enters the subjective drowsiness state, and notify the outside based on the detection result. can be sent to The drowsiness sensing device (6000) can be conscious of the drowsiness state from the time of the unconscious drowsiness state in which the subject cannot feel sleepiness through the notification.

Therefore, the detection and notification of unconscious drowsiness can increase the possibility of getting the subject out of the drowsiness before a dangerous situation occurs, and more accurately guide the subject out of the dangerous situation caused by the drowsiness. can be protected against

In one embodiment, the drowsiness sensing unit 6200 can measure the drowsiness state of the subject using information on heartbeat. The drowsiness state according to the present specification is a state in which the subject temporarily falls asleep or is likely to fall asleep within a predetermined time immediately before falling asleep, and compared to the steady state, relatively It is a state in which the sub-sympathetic nervous system is activated, and when the sub-sympathetic nervous system is activated, the heart rate decreases. In other words, the change in heart rate can serve as a biomarker that can estimate the drowsiness state of the subject. For example, if the subject is drowsy, the heart rate of the subject may be reduced below a certain level, and the reduced heart rate may be maintained for a certain period of time. In some cases, the drowsiness sensing unit (6200) can estimate the drowsiness state of the subject through the duration of the reduced heart rate.

In one embodiment, the drowsiness sensing unit (6200) can measure the subject's steady state using information about the heart rate. For example, when the person to be measured enters a steady state, the sympathetic nerve is activated to increase the heart rate of the person to be measured over a certain level, and the increased heart rate can be maintained for a certain period of time. In this case, the drowsiness sensing unit (6200) can estimate the subject's steady state through the duration of the increased heart rate.

Also, in a different embodiment, the drowsiness sensing unit 6200 can measure the drowsiness state and steady state of the subject using LF/HF. When the person to be measured enters a drowsy state, the subsympathetic nervous system of the person to be measured is activated, and the LF value of the heartbeat signal decreases with respect to the HF value. Also, when the person to be measured enters a steady state, the sympathetic nervous system of the person to be measured is activated and the LF value increases with respect to the HF value. Therefore, the drowsiness sensing unit can also estimate the drowsiness state and steady state of the subject through LF/HF.

Also, in one embodiment, the drowsiness sensing unit 6200 can estimate the drowsiness state or steady state of the subject by simultaneously considering the change in heart rate and the LF/HF.

A method for detecting drowsiness will be described in detail below.

8.1.3 Notifier In one embodiment, the notifier 6300 can provide information about externally perceived drowsiness. Specifically, the notification unit 6300 may provide various information related to whether the subject is drowsy, drowsiness level, and other information related to the drowsiness state. For example, the notification unit 6300 can provide information such as how long the sleepy state was maintained, the time when each sleepy state stage was entered, and the recovery axis of the sleepy state.
In one embodiment, the notification unit (6300) can provide the subject with information regarding sleepiness. For example, the notification unit (6300) can transmit a notification of intensity corresponding to the drowsiness stage to the subject. The notification can induce the subject to recover from the drowsy state by applying a constant stimulus to the subject who is in the drowsy state.

In other embodiments, other individuals who are not the subject can be provided with information regarding the sleepiness state of the subject. For example, the notifier (6300) can provide information about drowsiness to the administrator or server. Here, the manager may mean the subject's manager or an individual who manages the subject's drowsiness state. Providing such information can assist the administrator to effectively manage the safety of the subject based on the information on the drowsiness state.

In one embodiment, the notification unit 6300 can provide information on all levels of sleepiness detected to the outside. For example, the notification unit (6300) can provide the subject with information on the drowsiness state for the 1st to 3rd stages of drowsiness state.
Also, in another embodiment, the notification unit 6300 may provide information on at least some of the detected drowsiness states of all stages to the outside.
For example, the notification unit 6300 may provide the administrator with information about the time and stage when the third level of drowsiness is detected only when the third level of drowsiness is detected. If notifications about all levels of drowsiness disturb the subject, the notification unit 6300 provides only information about the high-risk drowsiness level (for example, information about the third level of drowsiness). By not providing information on the drowsiness stages with low risk (for example, information on the first stage drowsiness state), the work efficiency of the subject can be improved.
Also, in one embodiment, the notification unit 6300 may provide information on the detected steady state when the subject is detected as the steady state. Specifically, the notification unit 6300 can provide various information related to the detected steady state including whether the subject is in a steady state. For example, it is possible to provide the time when the steady state was maintained, the time when the steady state was entered, and the like.
In one embodiment, the notification unit (6300) can provide the subject with information about the steady state. For example, the notification unit (6300) can output a message to inform the fact that a subject who is sleepy has recovered to a steady state.

In another embodiment, the notification unit (6300) can provide information about the steady state to the subject to another individual who is not the subject. For example, the notifier (6300) can provide information about steady state to the administrator or server. Providing such information assists the administrator in recognizing that the subject is in a safe situation.

In one embodiment, the notification unit (6300) generates a notification signal indicating information related to the drowsiness state, and outputs the information regarding the steady state and the drowsiness state to the outside of the drowsiness sensing device. (not shown) can provide a notification signal.
Here, the output unit is a device for informing information to the outside. At least one of the parts can be included. The output unit may be included in the drowsiness sensing device (6000), or may be included in an external device of the drowsiness sensing device (6000).

When the output unit is included in another device outside the drowsiness detection device 6000, the drowsiness detection device 6000 further includes a communication unit (not shown), and the notification unit 6300 outputs a notification signal through the communication unit. can be provided to the output. The output unit can receive the notification signal provided by the notification unit (6300) and output information on drowsiness to the outside according to the notification signal.

For example, the notification unit (6300) can provide auditory notification through the audio unit. Such audible notification is effective because it can immediately stimulate the recipient of the notification and the strength of the stimulation varies according to the strength of the voice.

In another example, the notification unit 6300 can provide notifications by displaying messages on the display through the display unit. Such visual notifications can minimize situations where individuals other than the recipient of the notification are disturbed by the notification.

In other examples, the notifier (6300) can provide notifications through the tactile stimulus, such as vibration, electrical stimulation, thermal elements, or cooling elements. Such a tactile notification is effective because it does not disturb individuals other than the recipient of the notification, provides an immediate stimulus, and varies the intensity of the stimulus according to the strength of the stimulus.
Also, in one embodiment, the notification unit 6300 can provide notifications of other types and/or intensities according to the level of drowsiness detected by the subject. Along with this, the notification unit (6300) provides notifications in a type suitable for the subject to perceive that he or she is in a drowsy state and/or with an intensity necessary for the subject to get out of the drowsy state. be able to.

For example, when the first stage drowsiness state, which is a low intensity drowsiness state, is detected, the notification unit 6300 notifies the subject of the drowsiness state by using a visual message. can be made In addition, when the third level drowsiness, which is a high-intensity drowsiness state, is detected, the notification unit 6300 notifies the subject to use electrical stimulation to help the subject escape from the drowsiness state. can be

For example, when the first stage drowsiness state, which is a low intensity drowsiness state, is detected, the notification unit 6300 provides a low intensity alarm to the subject, and when the third stage drowsiness is detected. , the notification unit (6300) can provide a high-intensity alarm to the subject.
Of course, depending on the embodiment, the notifier 6300 may provide the subject with the same type and/or intensity of notification regardless of the perceived drowsiness stage.

In addition, the notifying unit 6300 can notify at least one of the subject and the non-subject according to the level of the sensed drowsiness state.
In one embodiment, the notification unit 6300 can notify the subject when the first and second stages of drowsiness are detected. At this time, the notification unit 6300 can give at least one of auditory notification, visual notification, and tactile notification to the subject.

In addition, the notification unit 6300 can notify not only the subject but also individuals other than the subject when the three-level drowsiness state is detected. Specifically, the notifying unit 6300 can notify the non-measured individual through a device or server that mediates the notifying unit 6300 and the non-measured individual. For example, if an individual who is not the subject is the manager managing the subject, such notification may be effective in situations where the subject turns off or ignores the notification through the output. is. The drowsiness sensing method performed by the drowsiness sensing device 6000 will now be described in detail.

8.2 Heart Rate-Based Drowsiness Detection Method Hereinafter, a heart rate-based drowsiness detection method according to an embodiment will be described with reference to FIGS. 39 to 43. FIG.
In one embodiment, the heart rate-based drowsiness sensing method includes steps of acquiring a heart rate (S6110), comparing the acquired heart rate of the subject and a reference heart rate (S6120), and according to the comparison result, a duration measurement step (S6130) of measuring the duration of the heart rate change, and a step-by-step drowsiness detection step (S6140) of detecting drowsiness in steps based on the measured duration of the heart rate change; can include
In the heart rate acquisition step (S6110), the drowsiness sensing device (6000) can acquire the subject's heart rate using the heart rate information acquisition unit (6100).
In one embodiment, the drowsiness sensing device (6000) can acquire the subject's heart rate in a predetermined time period. Here, the predetermined time period may include various time periods such as 1 second, 1 minute, and 5 minutes. Also, the predetermined time period may be fixed or variable.

In different embodiments, the drowsiness sensing device (6000) can acquire the subject's heart rate without following a predetermined time period. For example, the drowsiness sensing device (6000) can acquire the subject's heart rate every time the subject's heart rate is measured by the biometric acquisition device (10) or other device, that is, in real time. can.
Of course, if the subject's heart rate is measured by the bioindex acquisition device 10 or other devices in a predetermined time period, the drowsiness sensing device 6000 can be measured by the bioindex acquisition device 10 or other devices. The heart rate of the subject can be obtained based on the predetermined time period.
In another example, the drowsiness sensing device (6000) may request the biometric index acquisition device (10) or other device to provide a heart rate by an external input or request, and makes the request. The heart rate of the subject can be obtained each time.
In one embodiment, the drowsiness sensing device (6000) can obtain the subject's average heart rate for a certain period of time. The certain time interval can be arbitrarily determined. Here, the drowsiness sensing device 6000 can calculate the average heart rate for a certain period of time using the subject's heart rate received from an external device. Also, the drowsiness sensing device (6000) can receive the average heart rate of the person being measured for a certain period of time from an external device.

In the following, further effects of obtaining the average heart rate as the subject's heart rate will be described with reference to FIG.
Referring to FIG. 40 according to one embodiment, the drowsiness sensing device (6000) can obtain the heart rate with noise removed by obtaining the average heart rate (6003).

Here, the noise indicates an error caused by various factors such as the movement of the person being measured or external light. It can be difficult.

For example, when the heart rate information acquisition unit (6100) acquires the heart rate at intervals of every second from 1 second to 10 seconds, if noise occurs at 5 seconds, the drowsiness detection device (6000) accurately detects drowsiness. Sometimes you can't. However, when the heart rate information acquiring unit (6100) acquires the average heart rate (6003), the difference between noise and different values is corrected in the process of calculating the average heart rate (6003). ) can detect drowsiness more accurately.

As described in Table of Contents 8.1.1, a detailed description of how the heartbeat information acquisition unit (6100) acquires the heartbeat rate of the subject will be omitted.

In the following, referring to FIG. 41, the step of comparing the heart rate of the subject and the reference heart rate (S6120), and the duration of the acquired heart rate of the subject while decreasing/increasing. The measuring duration measuring step (S6130) will be described.
Referring to FIG. 41 according to one embodiment, in the step of comparing the heart rate of the subject and the reference heart rate (S6120), the drowsiness sensing device (6000) detects A measurer's heart rate (6001) and a reference heart rate (6002) can be compared.
Here, the reference heart rate (6002) may mean a heart rate set as a reference for distinguishing between the drowsy state and the steady state. As described above, activation of the subsympathetic nervous system reduces the heart rate in the drowsy state as compared to the heart rate in the normal state. Therefore, a heart rate that is the same as or similar to the decreased heart rate can be set as the reference heart rate (6002) in order to distinguish between the drowsy state and the steady state. In this case, when the subject's heart rate is below the reference heart rate (6002), it can be determined that the subject is drowsy, and the subject's heart rate (6001) is above the reference heart rate (6002). When , it can be grasped that the person to be measured is in a steady state.

Various methods for setting the reference heart rate (6002) will be described below.

In one embodiment, the drowsiness sensing device (6000) can set the reference heart rate (6002) by using the average heart rate obtained during a certain period of time from the time when the heart rate was first obtained. can. The predetermined time can be set to various values such as 1 minute and 5 minutes. In general, there is a high possibility that the person to be measured is in a steady state, not in a drowsy state, for a certain period of time after the heart rate of the person to be measured is first obtained. Therefore, the average heart rate of the person to be measured is obtained during the certain period of time, and the average heart rate is a predetermined value a (for example, the a represents a real number of 1 or less. More specific example , the value obtained by multiplying a by 0.9) can be obtained as the reference heart rate (6002). The predetermined value a is not limited to the above examples.
In another embodiment, the drowsiness sensing device (6000) can use resting heart rate and/or active heart rate to set the reference heart rate (6002). Specifically, the rest period heart rate may mean the heart rate when the person to be measured does not move (or when the amount of movement is small). In this specification, the heart rate when the person to be measured is awake but does not move (or when the amount of movement is small) is defined as the heart rate in the first resting period, and the heart rate when the person to be measured is in a sleeping state defined as the second resting heart rate.

Also, the heart rate in the active period may mean the heart rate when the subject moves actively.

In one embodiment, the drowsiness sensing device (6000) detects the heart rate measured when the subject is awake but does not move (or when the amount of movement is small). heart rate can be obtained. For example, the drowsiness detection device (6000) detects the heart rate for a certain period of time from the time when the person to be measured is in an environment where they cannot move actively (for example, when the person to be measured is sitting in the driver's seat). Assuming that the person to be measured is awake but not moving during the period, the heart rate measured during the fixed period of time can be set as the heart rate of the first resting period.

In addition, the drowsiness detection device (6000) acquires the heart rate measured when the person to be measured is in a sleeping state as the heart rate of the second resting period, and detects when the person to be measured is actively moving. A heart rate measured in a situation recognized as being active can be obtained as an active heart rate.

In addition, the drowsiness detection device 6000 can acquire the heart rate of the first rest period, the heart rate of the second rest period, and the heart rate of the active period using a camera or a wearable device.
For example, the drowsiness sensing device 6000 can obtain information on whether the subject is moving and whether the subject's pupils are being tracked using a camera. For example, the drowsiness sensing device (6000) can acquire the heart rate measured when the subject moves as the active heart rate. In addition, the drowsiness detection device (6000) uses a camera to record the heart rate measured when the subject's pupils are continuously tracked even if the subject does not move. and the heart rate measured when the subject is not moving and the subject's pupils are not continuously tracked can be obtained as the heart rate of the second resting period.

As another example, the drowsiness detection device 6000 uses the wearable device to detect whether the subject moves and whether the wearable device is worn on a body part (for example, a wrist). can be obtained. For example, the drowsiness detection device 6000 can obtain the heart rate measured when the subject's movement is detected as the active heart rate using a wearable device. In addition, the drowsiness detection device (6000) obtains the heart rate measured when the subject's movement is not detected but the body part wearing the wearable device is detected as the heart rate of the first rest period. A heart rate measured when the subject's movement is not detected and the movement of the body part is not detected for a certain period of time can be obtained as the second rest period heart rate.

In addition, the drowsiness sensing device (6000) may preset the heart rate of the first rest period, the heart rate of the second rest period, and the heart rate of the active period before starting the drowsiness detection. The heart rate of the first resting period, the heart rate of the second resting period and the heart rate of the active period can be received later from the outside.

According to one embodiment, the reference heart rate (6002) may be set based on the heart rate during the first rest period. Specifically, the reference heart rate (6002) may be set as a constant percentage of the heart rate in the first rest period. For example, the reference heart rate (6002) is calculated by multiplying the heart rate in the first rest period by a predetermined value a (for example, a is a real number of 1 or less, and as a more specific example, a is 0.9). can be Of course, the predetermined value a is not limited to the above example.

According to another embodiment, the reference heart rate (6002) can be set based on the heart rate of the second rest period. Specifically, the reference heart rate (6002) is set to be the same as the heart rate of the second rest period or a predetermined value b (for example, the b is a real number greater than 1) to the heart rate of the second rest period. , and as a more specific example, b can be calculated by multiplying by 1.1). Of course, the predetermined value b is not limited to the above examples.

Also, according to another embodiment, the reference heart rate (6002) can be set based on the heart rate during the active period. Specifically, the reference heart rate (6002) is calculated by multiplying the heart rate in the active period by a predetermined value c (for example, c is a real number of 1 or less, and as a more specific example, c is 0.8). can be In addition, the predetermined value c can be set lower than the predetermined value a, considering that the heart rate in the active period is generally higher than the heart rate in the first rest period. . Of course, the predetermined value c is not limited to the above examples.

The method of setting the reference heart rate (6002) is not limited to the method of calculating the heart rate of the first and second resting periods and the heart rate of the active period at a constant rate, and various calculations such as sum and difference are applied. Of course.

Specifically, in one embodiment, the drowsiness detection device (6000) sets the reference heart rate (6002) by adding or subtracting a predetermined value to the heart rate of the first and second rest periods or the heart rate of the active period. can be done. For example, if the heart rate in the first rest period is 75, the drowsiness detector will set the reference heart rate (6002) based on this to (heart rate in the first rest period - 10) = 65 (beats/minute). can be set. For example, if the heart rate in the second rest period is 65, the drowsiness detector (6000) will calculate the reference heart rate (6002) based on this (heart rate in the second rest period + 10) = 65 (times/ minutes).

Also, in another embodiment, the drowsiness sensing device (6000) can change an already set reference heart rate (6002) to a new reference heart rate (6002). That is, the drowsiness sensing device (6000) can update the preset reference heart rate (6002) to set a new reference heart rate (6002).

For example, as described above, the drowsiness sensing device (6000) sets the reference heart rate (6002) based on the heart rate acquired during a certain period of time from the time when the subject's heart rate was first acquired. can be done. The drowsiness detection device (6000) senses a steady state based on the heart rate of the person to be measured after the predetermined period of time, and calculates the average heart rate in the time interval sensed in the steady state as a new reference heart rate (6002 ) can be changed to In addition, the drowsiness detection device (6000) changes the heart rate of the first, second rest period, and active period obtained for the same subject after the predetermined time to a new reference heart rate (6002). be able to. As a result, the drowsiness sensing device (6000) can set the reference heart rate (6002) that reflects the subject's latest condition, and therefore the drowsiness sensing device (6000) can set a more accurate reference heart rate (6002). Based on this, it is possible to sense the drowsiness of the subject.

In another example, the drowsiness sensing device (6000) can preset the reference heart rate (6002) based on the heart rates of the first, second resting and active periods, as described above. At this time, the drowsiness sensing device (6000) detects the first and second rest periods or A new active heart rate can be acquired. The drowsiness detection device (6000) uses the newly acquired heart rate of the first and second rest periods or active periods to set the new reference heart rate (6002) by the method of setting the reference heart rate (6002) described above. ) can be set.

Referring to FIG. 41 according to an embodiment, in the step of comparing the heart rate (6001) of the subject and the reference heart rate (6002) (S6120), the drowsiness sensing device (6000) detects the heart rate of the subject. number (6001) and said reference heart rate (6002) can be compared.
In one embodiment, the drowsiness detection device (6000) compares the subject's heart rate (6001) with a reference heart rate (6002), and determines that the subject's heart rate (6001) is less than or equal to the reference heart rate (6002)/ It is possible to detect the time (t1) when the The sensed time (t1) can be a starting point for measuring the sleepiness state and the steady state according to stages.
That is, in the above embodiment, the sensing of the time (t1) when the subject's heart rate (6001) is below/above the reference heart rate (6002) is the trigger ( trigger).

In the duration measurement step (S6130), the drowsiness detection device (6000) detects the heart rate (6001) of the person to be measured as the time when the heart rate (6001) of the person to be measured falls below the reference heart rate (6002). It is possible to measure the time during which the heart rate (6002) or less continues. In addition, the drowsiness detection device (6000) detects the time when the heart rate (6001) of the person to be measured becomes equal to or greater than the reference heart rate (6002). It is possible to measure how long the above state lasts.

Heart rate can fluctuate due to many physical factors besides drowsiness. Since the decrease/increase in heart rate due to drowsiness persists in a decrease/increase state among other physical factors, a sustained decrease or increase in heart rate for several seconds to minutes may indicate a drowsy state. can be a good biomarker to demonstrate.
For example, the heart rate temporarily decreases when a person's tension is released, but the decreased heart rate can be easily restored. If the drowsiness detection device 6000 detects the drowsiness state based on the time the heart rate continues to decrease, the drowsiness detection device 6000 detects the drowsiness state in which the heart rate temporarily decreases due to the release of tension. You can avoid perceiving it as a state. Accordingly, the drowsiness sensing device (6000) should be able to sense drowsiness more accurately.

Also, the more intense the subject enters into a state of drowsiness, the longer the drowsiness-decreased heart rate lasts. Therefore, if the drowsiness sensing device 6000 measures the length of time during which the reduced heart rate continues, it is possible to know what level of drowsiness the subject has entered. That is, by classifying the drowsiness state according to the duration, the drowsiness sensing device 6000 can more specifically sense the drowsiness state of the subject.

Referring to FIG. 41 according to one embodiment, the drowsiness sensing device (6000) detects the subject's The duration of the heart rate (6001) below the reference heart rate (6002) can be measured.
For example, the reference heart rate (6002) can be set to 72 (beats/minute) corresponding to 90% of the heart rate in the first rest period. Here, if the subject's heart rate is 74 (beats/minute) at the first time point and the subject's heart rate is 72 (beats/minute) at the second time point immediately after the first time point, the drowsiness detector ( 6000) can measure the duration of the subject's heart rate below 72 (beats/minute) from the second time point.

Of course, according to other embodiments, the drowsiness detection device (6000) detects the duration of the state in which the heart rate of the subject is above the reference heart rate based on the time when the heart rate of the subject is above the reference heart rate. can also be measured.

For example, the reference heart rate (6002) can be set to 72 (beats/minute) corresponding to 90% of the heart rate in the first rest period. Here, the subject's heart rate (6001) is 70 (beats/minute) at the third time point, and the subject's heart rate (6001) is 72 (beats/minute) at the fourth time point immediately after the third time point. In this case, the drowsiness sensing device (6000) can measure the duration of the subject's heart rate (6001) being 72 (beats/minute) or more from the fourth time point.

Referring to FIG. 42 according to one embodiment, when the drowsiness sensing device (6000) measures the duration of the state in which the heart rate (6001) of the subject is below/above the reference heart rate (6002), The duration can be measured by correcting the time interval (Δtn) in which the heart rate containing noise was measured. In one example, the drowsiness sensing device (6000) can correct the noise-sensed heart rate based on the previously measured heart rate. In another example, a noisy heart rate measurement time interval (Δtn) may be excluded during duration measurement.

Here, noise indicates an error caused by various factors such as movement of the person being measured or external light, and the heart rate including noise may not reflect the state of the person being measured. Generally, a noisy heart rate is acquired temporarily and has a large deviation from previously acquired and subsequently acquired heart rates of the subject.
Based on this, in one embodiment, the drowsiness detection device (6000) detects the previously obtained heart rate (6001) of the subject and/or the subsequently obtained heart rate (6001) of the subject and a predetermined value. When a heart rate with such a large deviation is obtained, the heart rate of the person to be measured (6001) can be detected as a heart rate containing noise. For example, when the drowsiness sensing device acquires a heart rate that has a deviation of 20 or more from the previously sensed heart rate and/or subsequently acquired heart rate during a time interval of 3 seconds or less, the 3 seconds The measured person's heart rate (6001) acquired during the measurement can be sensed as a heart rate containing noise. Here, the time at which the heart rate including the noise is acquired and the deviation between the heart rate including the noise and the different heart rate are not limited to specific values.
According to one embodiment, the drowsiness sensing device (6000) determines that the subject's heart rate (6001) is equal to the reference heart rate (6002) even though the heart rate containing noise is greater than or equal to the reference heart rate (6002). ), the duration can be measured by treating the noise as a heart rate below the reference heart rate.
According to one embodiment, when the drowsiness sensing device (6000) measures the duration of the subject's heart rate above the reference heart rate, even though the heart rate containing noise is below the reference heart rate. , the noise can be treated as a heart rate equal to or greater than the reference heart rate and the duration can be measured.

According to one embodiment, the drowsiness sensing device (6000) measures the duration of the subject's heart rate below the reference heart rate, even though the noisy heart rate is below the reference heart rate. When the heart rate including noise is measured, it can be measured except for the time interval (Δtn).
According to one embodiment, when the drowsiness sensing device (6000) measures the duration of the subject's heart rate above the reference heart rate, even though the heart rate containing noise is below the reference heart rate. , excluding the time interval (Δtn) in which the heart rate containing noise was measured.

Referring to FIG. 41 according to an embodiment, in step S6140, the drowsiness sensing device 6000 senses drowsiness or a steady state by comparing the measured duration with the reference duration. can do.

The drowsiness sensing device (6000) measures the duration of the state in which the heart rate (6001) of the subject is below the reference heart rate, and the measured duration reaches the reference duration (Δta, Δtb, Δtc). You can detect drowsiness when you do it. In this case, the reference durations (Δta, Δtb, Δtc) are plural. For example, there may be three reference durations (Δta, Δtb, Δtc), the first reference duration (Δta) is 30 seconds, the second reference duration (Δtb) is 60 seconds, and the third reference duration is (Δtc) can be set to 90 seconds.
At this time, if the subject's heart rate (6001) remains below the reference heart rate (6002) for 30 seconds, the drowsiness detection device indicates that the subject is in the first stage of drowsiness. can be perceived.

At this time, if the subject's heart rate (6001) remains below the reference heart rate (6002) for 60 seconds, the drowsiness detection device (6000) detects that the subject is in the second stage of drowsiness. state can be perceived.
At this time, if the subject's heart rate (6001) remains below the reference heart rate (6002) for 90 seconds, the drowsiness detection device (6000) detects that the subject is in the third stage of drowsiness. state can be perceived.
In addition, the drowsiness sensing device 6000 can maintain the sensed drowsiness state until it senses another stage of drowsiness state or a steady state for the person whose drowsiness state has been sensed.

In another embodiment, if the duration of the heart rate (6001) of the subject being below the reference heart rate (6002) does not reach the first reference duration (Δta), the drowsiness sensing device (6000) can sense that the subject is in a steady state.
For example, if the first reference duration (Δta) is 30 seconds and the subject's heart rate (6001) is below the reference heart rate (6002) for 10 seconds, then the heart rate reaches or exceeds the reference heart rate. If recovered, the drowsiness sensing device (6000) can sense that the subject is in a steady state.

Hereinafter, a method for sensing recovery from drowsiness based on heart rate will be described with reference to FIG.
In the drowsiness state sensing step (S6140), the drowsiness sensing device (6000) detects that the subject is drowsy when the measured duration reaches the recovery reference duration (Δtd, Δte, Δtf). You can feel that you have recovered.

Restoration of the drowsy state may refer to the state in which the subject exits the perceived drowsiness state after entering the drowsiness state. Specifically, recovery of drowsiness may refer to a state in which a drowsiness level lower than a level of drowsiness level detected after a predetermined level of drowsiness level is detected is sensed. Restoration of a drowsy state may also refer to a situation in which a steady state is sensed after a drowsy state has been sensed.
According to one embodiment, the time during which the heart rate (6001) of the subject whose drowsiness state is detected continues to be equal to or higher than the recovery reference heart rate (6004) reaches the recovery reference duration (Δtd, Δte, Δtf). In this case, the drowsiness sensing device (6000) can sense recovery of drowsiness of the subject.
Specifically, referring to (a) of FIG. 46 according to one embodiment, the drowsiness sensing device (6000) detects that the subject's heart rate (6002) is equal to or higher than the recovery reference heart rate (6004) from time (t1). It is possible to measure the duration that the state continues, and sense that the subject has recovered from the drowsy state at the time (t1, t2, t3) when the duration reaches the recovery reference duration.
Here, the recovery reference duration of the recovery reference duration can be set variously. For example, the recovery reference duration of the recovery duration can be set in various units such as seconds, tens of seconds, minutes, and tens of minutes.

For example, the recovery criterion duration can be set to 30 seconds. In this case, the drowsiness detection device (6000) detects when the subject's heart rate, which is detected in the 3-stage drowsiness state, continues for 30 times at the recovery reference heart rate (6004) or higher. can be sensed to have returned to steady state.
According to one embodiment, a plurality of recovery reference durations can be set. For example, there may be three recovery reference durations, the first recovery reference duration may be set to 20 seconds, the second recovery reference duration to 40 seconds, and the third recovery reference duration to 60 seconds. can. Here, the first to third recovery reference durations are not limited to the values suggested by the above examples.
Referring to (a) of FIG. 43 according to one embodiment, the drowsiness sensing device (6000) detects the heart rate (6001) of the subject detected in the third stage drowsiness state. ) is measured from the point of time (t1) at which the recovery reference heart rate (6004) or more is maintained at the recovery reference heart rate (6004) or more, and the time when the duration reaches the first recovery reference duration ( At t2), it can be sensed that the subject has recovered from the second stage drowsiness state.
In addition, after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the drowsiness sensing device (6000) detects when the subject's heart rate (6002) is equal to or higher than the recovery reference heart rate (6004). From (t1), measure the duration for which the heart rate (6001) of the person to be measured is equal to or higher than the recovery reference heart rate (6004), and measure the time when the duration reaches the second recovery reference duration (t3). 3, the drowsiness sensing device 6000 can sense that the subject has recovered from the first stage drowsiness state.
However, referring to FIG. 43(b) according to one embodiment, after the subject recovers from the third-stage drowsiness state to the second-stage drowsiness state, the heart rate (6002) of the subject increases to the recovery reference heart rate. Drowsiness is detected before the duration of the subject's heart rate (6001) being equal to or higher than the recovery reference heart rate (6004) from the point (t4) of the number (6004) or higher before reaching the second recovery duration. When the device (6000) detects the time (t6) when the subject's heart rate (6001) is equal to or less than the recovery reference heart rate (6004), the drowsiness detection device (6000) detects that the subject is in the second stage. It is possible to maintain the state of sleepiness and reset it to the third stage sleepiness state.

Of course, the drowsiness detection device (6000) can detect the recovery from drowsiness based on the time during which the average heart rate (6003) of the subject remains equal to or higher than the recovery reference heart rate (6004).

The recovery reference heart rate (6004) may be the same as or different from the reference heart rate (6002) for drowsiness detection.
The recovery reference duration may be the same as or different from the duration for sensing drowsiness.

8.3 Drowsiness Detection Method Based on LF/HF Hereinafter, a drowsiness detection method based on LF/HF according to an embodiment will be described with reference to FIGS. 44 to 47. FIG.

In one embodiment, the method for detecting drowsiness based on LF/HF includes the step of acquiring LF/HF (6005) of the subject (S6210), determining the LF/HF (6005) of the subject and the reference LF/HF (S6210). A step of comparing sizes (S6220) and a step of sensing sleepiness according to stages based on the comparison result (S6230) may be included.

The drowsiness detection method may further include the step of giving an external notification according to the drowsiness level according to an embodiment.

At the step of acquiring LF/HF (S6210), the heartbeat information acquisition unit (6100) can acquire LF/HF (6005) of the subject.

In one embodiment, the drowsiness sensing device (6000) can acquire the subject's LF/HF (6005) in a predetermined time period. In different embodiments, the drowsiness sensing device (6000) can acquire the subject's LF/HF (6005) without following a predetermined time period.
Of course, if the subject's LF/HF (6005) is measured by the bioindex acquisition device (10) or other device in a predetermined time period, the drowsiness detection device (6000) can be detected by the bioindex acquisition device (10) or The subject's LF/HF (6005) can be obtained based on a predetermined time period measured by another device.
In another example, the drowsiness sensing device (6000) requests the biometric acquisition device (10) or other device to provide LF/HF by an external input or request, and each time it makes the request The LF/HF (6005) of the subject can be obtained.
In one embodiment, the drowsiness sensing device (6000) can obtain the subject's average LF/HF for a certain time interval.

The drowsiness detector (6000) has the effect of accurately detecting drowsiness by obtaining average LF/HF with noise correction, which is the same effect obtained by obtaining average heart rate. , and detailed description is omitted.

Referring to FIG. 45 according to one embodiment, in the step of comparing the obtained LF/HF of the subject (6005) with the reference LF/HF (6006, 6007, 6008) (S6220), the drowsiness sensing device (6000) ) can compare the acquired subject's LF/HF (6005) with the reference LF/HF.
Here, the reference LF/HF (6006, 6007, 6008) may mean a value set as a reference for distinguishing between the drowsy state and the steady state. As mentioned above, the LF/HF in the drowsy state may be smaller than the LF/HF in the normal state due to the activation of the subsympathetic nervous system.

In one embodiment, the drowsiness sensing device (6000) uses the LF/HF average of the heartbeat signal acquired during a certain period of time from the time when the LF/HF of the heartbeat signal was first acquired to obtain the reference LF/HF ( 6006, 6007, 6008) can be set. The predetermined time can be set to various values such as 1 minute and 5 minutes. In general, there is a high possibility that the person being measured is in a steady state for a certain period of time after the LF/HF (6005) of the person being measured is first acquired. Therefore, the average LF/HF of the person to be measured acquired during the certain period of time is obtained, and the average LF/HF is a predetermined value a (for example, the a is a real number of 1 or less, and a more specific example , the value obtained by multiplying a by 0.9) can be obtained as the reference LF/HF. Also, the drowsiness sensing device can set multiple reference LF/HF using other predetermined real numbers. The predetermined value a is not limited to the above examples.

In another embodiment, the drowsiness sensing device (6000) can use the resting period LF/HF and/or the active period LF/HF to set the reference LF/HF (6006, 6007, 6008). At this time, there are multiple reference LF/HF (6006, 6007, 6008).

Specifically, the drowsiness detection device (6000) uses a camera or a wearable device to measure heart rate during the first rest period, LF/HF, and heart rate during the second rest period. You can get the second resting period LF/HF, and the active period LF/HF based on the active period heart rate.

According to one embodiment, the reference LF/HF (6006, 6007, 6008) can be set based on the first rest period LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) can be set at a constant ratio of the first rest period LF/HF. For example, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006), which is the first rest period LF/HF multiplied by a predetermined value of 0.9, and the second reference LF, which is multiplied by 0.8. /HF (6007), and a third reference LF/HF (6008) multiplied by 0.7. Of course, the predetermined values are not limited to the above examples.

According to another embodiment, the reference LF/HF (6006, 6007, 6008) can be set based on the second rest period LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) can be set at a constant ratio of the first rest period LF/HF. For example, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006), which is the first rest period LF/HF multiplied by a predetermined value of 1.5, and the second reference LF, which is multiplied by 1.3. /HF (6007), and a third reference LF/HF (6008) multiplied by 1.1. Of course, the predetermined values are not limited to the above examples.

Also, according to another embodiment, the reference LF/HF (6006, 6007, 6008) can be set based on the active LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006), which is the active period LF/HF multiplied by a predetermined value of 0.8, and the second reference LF, which is multiplied by 0.7. /HF (6007), and a third reference LF/HF (6008) multiplied by 0.6. Of course, the predetermined values are not limited to the above examples.

The method of setting the reference LF/HF (6006, 6007, 6008) is not limited to the method of calculating at a constant ratio of the first and second rest period LF/HF and active period LF/HF, addition, subtraction, etc. Of course, various operations can be applied. Also, in another embodiment, the drowsiness detection device (6000) can change from the already set reference LF/HF (6006, 6007, 6008) to a new reference LF/HF (6006, 6007, 6008). .
For example, as described above, the drowsiness detection device (6000) calculates the reference LF/HF ( 6006, 6007, 6008) can be set. The drowsiness detection device (6000) can detect a steady state based on the subject's LF/HF (6005) obtained after the given period of time, and the average LF/HF in the time interval detected in the steady state is updated. Can be changed to standard LF/HF (6006, 6007, 6008). In addition, the drowsiness detection device (6000) generates a new reference LF/HF (6006, 6007 ,6008).
In another example, the drowsiness sensing device (6000) can preset the reference LF/HF (6006, 6007, 6008) based on the first, second rest period and active period LF/HF as described above. can. At this time, the drowsiness detection device (6000) detects the same New reference LF/HF (6006, 6007, 6008) can be set based on newly set 1st, 2nd rest period LF/HF and active period LF/HF for the same subject by the method. .

In addition, the drowsiness detection device (6000) shall set a new standard LF/HF (6006, 6007, 6008) based on the LF/HF acquired within a certain period of time from the first acquisition of LF/HF. can be done.
In addition, the drowsiness detection device (6000) sets a new reference LF/HF (6006, 6007, 6008) based on the average LF/HF during the time interval when the subject is detected in steady state. can be done.
Referring to FIG. 45 according to one embodiment, the drowsiness sensing device (6000) detects that the subject's LF/HF (6005) is greater than or equal to any one of a plurality of reference LF/HF (6006, 6007, 6008). It is possible to sense the time points (t1, t2, t3, t4) that are: The sensed time point may be a time point for measuring drowsiness state and steady state.
In one embodiment, the drowsiness detection device (6000) detects the drowsiness state at the time (t1, t2, t3) when the subject's LF/HF (6005) becomes equal to or less than the reference LF/HF (6006, 6007, 6008). can be sensed.

In one embodiment, the drowsiness detection device (6000) detects the time (t1 , t2, t3), the level of drowsiness corresponding to the one reference LF/HF can be sensed. Here, the drowsiness detection device (6000) detects the drowsiness state based on the result of comparing the subject's LF/HF (6005) with not only the one reference LF/HF but also other reference LF/HF. can be sensed.

Specifically, multiple reference LF/HF (6006, 6007, 6008) are set to 1st reference LF/HF (6006), 2nd reference LF/HF (6007), and 3rd reference LF/HF (6008). can be In the following, for convenience of explanation, the first reference LF/HF (6006) is defined as the reference LF/HF for sensing the drowsiness state with the lowest intensity.
For example, if the subject's LF/HF (6005) is less than the first standard LF/HF (6006) and greater than the second standard LF/HF (6007) at the first time point, the drowsiness detection device (6000) , the drowsiness state of the subject can be determined as the first stage drowsiness state.

Also, at the first time point, if the subject's LF/HF (6005) is less than the second standard LF/HF (6007) and more than the third standard LF/HF (6008), the drowsiness detection device (6000) , the drowsiness state of the subject can be determined as the second stage drowsiness state.

Also, at the first time point, if the subject's LF/HF (6005) is less than or equal to the third reference LF/HF (6008), the drowsiness detection device (6000) puts the subject's drowsiness state into the third stage. It can be judged as a drowsy state.

Hereinafter, a method for sensing recovery from drowsiness based on LF/HF will be described with reference to FIG.
Referring to (a) of FIG. 46 according to one embodiment, in the step of sensing the state of drowsiness (S6230), the drowsiness sensing device (6000) detects that the measured LF/HF (6005) of the subject is It can be sensed that the subject has recovered from the drowsy state at the time points (t1, t2, t3, t4) that exceed the recovery criteria LF/HF (6006, 6007, 6008, 6010). Here, the drowsiness detection device (6000) determines the drowsiness state based on the result of comparing the subject's LF/HF (6005) with the one recovery reference LF/HF and a different recovery reference LF/HF. You can feel the recovery.

In one embodiment, the plurality of criteria LF/HF (6006, 6007, 6008, 6010) are a first recovery criteria LF/HF (6006), a second recovery criteria LF/HF (6007), a third recovery criteria LF/HF Can be set to HF (6008). Hereinafter, for convenience of explanation, the first recovery criterion LF/HF (6006) is set to be the recovery criterion LF/HF having the largest value among recovery criteria LF/HF equal to or less than 1. For example, the subject's LF/HF (6005) is below the third recovery criterion LF/HF (6008) at the first time, and the subject's LF/HF (6005) is the third recovery at the second time. If the reference LF/HF (6008) or more and the second recovery reference LF/HF (6007) or less, the drowsiness detection device (6000) detects that the drowsiness state of the subject at the second time point is from the third stage drowsiness state. It can be determined that the patient has recovered to the second stage drowsiness state. At the third point after the subject recovers from the third-stage drowsiness state to the second-stage drowsiness state, the drowsiness sensing device (6000) detects that the subject's LF/HF (6005) is the second recovery criterion. If the LF/HF (6007) or more and the first recovery reference LF/HF (6006) or less, the drowsiness detection device (6000) detects that the subject's drowsiness state has changed from the second-stage drowsiness state to the first-stage drowsiness state at the third time point. It can be judged that the state has recovered to the 1st stage sleepiness state.

In one embodiment, at a first time point the subject's LF/HF (6005) is less than or equal to a third recovery criterion LF/HF (6008), and at a subsequent second time point the subject's LF/HF (6005) is equal to or greater than the second recovery criterion LF/HF (6007) and equal to or less than the first recovery criterion LF/HF (6006), the drowsiness detection device (6000) detects that the subject's drowsiness state is the third It can be determined that the staged drowsiness state has recovered to the first stage drowsiness state.
However, referring to (b) of FIG. 46 according to one embodiment, after the time (t6) when the subject recovers from the third stage sleepiness state to the first stage sleepiness state, the drowsiness sensing device (6000) If the subject's LF/HF (6005) decreases to the second recovery criterion LF/HF (6007) or less (t7) within a certain period of time, the drowsiness detection device (6000) It can be perceived as being in the second stage drowsiness state. Also, it can be sensed that the person to be measured has reset to the third stage drowsiness state.
The first recovery reference LF/HF (6006) is the first reference LF/HF (6006) in FIG. 45, and the second recovery reference LF/HF (6010) is the second reference LF/HF (6007) in FIG. ) and the third recovery reference LF/HF (6011) may be the same as or different from the third reference LF/HF (6006) in FIG.

In addition, a fourth recovery criterion LF/HF (6010) having a value greater than the first to third recovery criteria LF/HF is used to detect recovery of the drowsiness detection device (6000) from the drowsiness state to the steady state. can also be set.
For example, if the drowsiness detection device (6000) senses that the subject's LF/HF (6005) is greater than the fourth recovery criterion LF/HF (6010), the subject's It can also be determined that the patient has recovered from the drowsy state to a steady state. Generally, LF/HF is measured within the range of 0.9 to 1 when the subject is in a normal state. Therefore, even if the person to be measured is sensed to be in a drowsiness state of very low intensity, the LF/HF of the person to be measured is less likely to have a value of 1 or more. That is, by setting the fourth recovery reference LF/HF (6010) having a value of 1 or more as the recovery reference LF/HF, the drowsiness detection device (6000) detects when the subject has completely recovered from the drowsiness state. can also be determined to be in a steady state.

8.4 Drowsiness Detection Method Based on Heart Rate and LF/HF A drowsiness detection method based on heart rate and LF/HF will now be described with reference to FIG.

In one embodiment, step-by-step drowsiness sensing steps (S6100, S6200) based on heart rate and LF/HF, and step (S6300) of determining the final drowsiness state may be included.

The drowsiness detection method may further comprise, according to an embodiment, informing the subject/non-subject individual according to the stage of drowsiness.
In the table of contents of this specification, a drowsiness detection method to which the heart rate-based drowsiness detection method described in Table of Contents 8.3 and the LF/HF-based drowsiness detection method described in Table of Contents 8.4 are applied will be described.

Heart rate and LF/HF are derived from heart rate information obtained from the same subject, but in some cases the drowsiness stage determined based on heart rate and the drowsiness stage determined based on LF/HF can be different. For example, the drowsiness sensing device can detect 3 levels of drowsiness based on heart rate and 2 levels of drowsiness based on LF/HF from the same subject at the same time. In this way, although the drowsiness detection device determined the drowsiness state of the same subject at the same time, the fact that different drowsiness stages were determined by the drowsiness detection method means that the drowsiness detected by any one of the drowsiness detection methods. It may mean that the state determination result may be wrong.

Therefore, in order to reduce such errors and detect drowsiness more accurately, a drowsiness detection method based on both heart rate and LF/HF will be described below.

In one embodiment, the drowsiness-related state sensed by the heart rate and LF/HF-based drowsiness sensing method can be indicated by a final drowsiness state and a final steady state. In one embodiment, the final drowsiness state can be divided into multiple stages according to the drowsiness intensity. For example, it can be divided into a first terminal sleepiness state, a second terminal sleepiness state and a third terminal sleepiness state. Here, the first final drowsiness state may mean the lowest intensity final drowsiness state and the third final drowsiness state may mean the highest final drowsiness state.

In one embodiment, the final drowsiness state and final normal state correspond to the drowsiness state and steady state sensed based on the heart rate-based drowsiness detection method and the LF/HF-based drowsiness detection method, respectively. It can mean state. That is, the final drowsiness state may mean a state in which the subject temporarily falls asleep or a state in which there is a high possibility of falling asleep within a predetermined time before falling asleep, and the final steady state is the final drowsiness state. It may mean a state in which the subject is unlikely to fall asleep within a predetermined period of time.

As described above, the method of detecting drowsiness based on heart rate changes and LF/HF values has four stages in total: steady state, first-stage drowsiness, second-stage drowsiness, and third-stage drowsiness, depending on the intensity of drowsiness. can be divided into Here, the first-stage drowsiness state may mean unconscious/unconscious drowsiness state, and the second and third-stage drowsiness states may mean subjective/conscious drowsiness state.

In one embodiment, the final drowsiness state divided into a plurality of stages according to drowsiness intensity is detected based on each of a heart rate-based drowsiness detection method and an LF/HF-based drowsiness detection method. It may refer to a state corresponding to a first-stage drowsiness state, a second-stage drowsiness state, and a third-stage drowsiness state. Therefore, the first terminal drowsiness state may refer to an unconscious/unconscious drowsiness state. Also, the second final sleepiness state and the third final sleepiness state may mean subjective/conscious sleepiness states. In one embodiment, the drowsiness sensing device can acquire the final drowsiness state of the subject in a predetermined time period. Of course, the predetermined time period may be fixed or may be varied.

In different embodiments, the drowsiness sensing device can obtain the final drowsiness state of the subject according to a predetermined time period. For example, the drowsiness sensing device acquires the drowsiness of the subject in real time, i.e., every time the subject's heart rate and/or LF/HF are measured by the biometric acquisition device (10) or other device. be able to.

Of course, if the subject's heart rate or LF/HF is measured by the bioindex acquisition device (10) or other device in a predetermined time period, the drowsiness sensing device can be detected by the bioindex acquisition device (10) or other device. The final drowsiness state of the subject can be obtained based on the measured predetermined time period.
As another example, the drowsiness sensing device can request the biometric device (10) or other device to provide LF/HF and heart rate by external input or request, and each time it makes said request, The subject's final drowsiness state can be obtained.
In one embodiment, the drowsiness sensing device can obtain the subject's average heart rate and average LF/HF for a certain period of time.

The drowsiness detection device acquires the average heart rate and average LF/HF and has the effect of accurately detecting drowsiness by correcting the noise, which is the same as the effect obtained by acquiring the average heart rate and average LF/HF. Therefore, detailed description will be omitted.

In one embodiment, the drowsiness detection device can determine the final drowsiness level based on the drowsiness level detected based on heart rate and the drowsiness level detected based on LF/HF. Depending on the case, the drowsiness stage sensed based on heart rate and the drowsiness stage sensed based on LF/HF may or may not match.

For convenience of explanation, the steady state detected based on the heart rate is defined as a drowsiness state lower than the first drowsiness state. Also, the steady state detected based on LF/HF is defined as a drowsiness state lower than the first-stage drowsiness state.

If the level of drowsiness detected based on heart rate matches the level of drowsiness detected based on LF/HF, the drowsiness sensing device may determine the matching level as the final drowsiness level. can. For example, if the drowsiness detection device senses the first stage drowsiness state based on the heart rate of the subject and detects the first stage drowsiness state based on LF/HF, the drowsiness detection device detects the first stage drowsiness state as the final drowsiness state. You can decide the stage 1 final drowsiness state. In addition, if the drowsiness detection device detects the 2nd stage drowsiness state based on the heart rate of the subject and detects the 2nd stage drowsiness state based on the LF/HF, the drowsiness detection device detects the drowsiness state as the final drowsiness state. You can decide the 2 stage final drowsiness state. In addition, if the drowsiness detection device detects the 3rd stage drowsiness state based on the heart rate of the subject and detects the 3rd stage drowsiness state based on the LF/HF, the drowsiness detection device determines the final drowsiness state as the 3rd stage drowsiness state. You can decide the final drowsiness state in 3 stages.

If the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, the drowsiness detection device can determine the final drowsiness level by considering various situations. can.

In one embodiment, if the drowsiness level sensed based on the heart rate and the drowsiness level sensed based on the LF/HF do not match, the drowsiness sensing device determines the higher of the two levels. It can be determined as a stage of drowsiness. For example, if the drowsiness detection device senses the first level of drowsiness based on heart rate and the second level of drowsiness based on LF/HF, the drowsiness detection device determines the final drowsiness state. You can decide the 2 stage final drowsiness state.

In another embodiment, if the drowsiness level sensed based on the heart rate and the drowsiness level sensed based on the LF/HF do not match, the drowsiness sensing device selects the lower of the two levels. can be determined as the final drowsiness stage. For example, if the drowsiness detection device senses the first level of drowsiness based on heart rate and the second level of drowsiness based on LF/HF, the drowsiness detection device determines the final drowsiness state. You can decide the stage 1 final drowsiness state.

In another embodiment, when the drowsiness stage sensed based on the heart rate and the drowsiness stage sensed based on the LF/HF do not match, the drowsiness sensing device calculates the average of the two stages as the final drowsiness level. It can be defined as a stage of the state. Here, if the average is not an integer, it can be rounded down or up to a small number of first positions. For example, if the drowsiness detection device detects the 1st stage drowsiness based on the heart rate of the subject and detects the 3rd stage drowsiness based on the LF/HF, the drowsiness detection device detects the drowsiness as the final drowsiness. A stage 2 final drowsiness state can be determined which is the average of the two stages. For example, when the first stage drowsiness is detected based on the heart rate and the second stage drowsiness is detected based on the LF/HF, the drowsiness detection device determines the average of the two stages as the final drowsiness. By rounding off the first decimal place for a certain 1.5, the first stage final drowsiness state can be determined. The drowsiness sensing device can also determine the final drowsiness level of the second stage by rounding up the final drowsiness level of 1.5, which is the average of the two stages, to the first decimal place.

Of course, in some cases, the drowsiness sensing device may newly define a drowsiness stage between the drowsiness stage sensed on the basis of heart rate and the drowsiness stage sensed on the basis of LF/HF. The drowsiness stage can be determined as the final drowsiness stage. For example, when the first stage drowsiness is detected based on the heart rate and the second stage drowsiness is detected based on the LF/HF, the drowsiness sensing device detects the first stage drowsiness and the second stage drowsiness. An intermediate drowsiness state can be defined that indicates a drowsiness intensity between and determined as the final drowsiness stage.

In another embodiment, if the drowsiness level sensed based on the heart rate and the drowsiness level sensed based on the LF/HF do not match, the drowsiness sensing device determines a previously determined final drowsiness level or A final steady state can be maintained as the state for drowsiness. For example, in a situation where the drowsiness detection device senses the final drowsiness state of the subject in the first stage, the drowsiness state in the first stage is detected based on the heart rate, and the drowsiness state in the third stage is detected based on the LF/HF. , the drowsiness sensing device can maintain a previously determined final drowsiness state, the first stage final drowsiness state, as the drowsiness-related state. For example, when the drowsiness detector senses the final steady state of the person being measured, if it senses the 1st level of drowsiness based on the heart rate and the 3rd level of drowsiness based on the LF/HF, the drowsiness The sensing device can maintain a previously determined final steady state as the drowsiness state.

In the following, a case where the drowsiness level sensed based on the heart rate and the drowsiness level sensed based on the LF/HF do not match will be described in detail.

8.4.1 Specific Embodiment In one embodiment, if the drowsiness stage sensed based on the heart rate and the drowsiness stage sensed based on the LF/HF are consistent with the steady state, the drowsiness sensing device can determine that the subject is in the final steady state.
In one embodiment, if the drowsiness stage sensed based on the heart rate and the drowsiness stage sensed based on the LF/HF match the first-stage drowsiness state, the drowsiness sensing device detects that the subject is The first stage can be determined to be the final drowsiness state.

In one embodiment, if the drowsiness stage sensed based on the heart rate and the drowsiness stage sensed based on the LF/HF match the second-stage drowsiness state, the drowsiness sensing device detects that the subject is The second stage can be determined to be the final drowsiness state.

In one embodiment, if the drowsiness level sensed based on the heart rate and the drowsiness level sensed based on the LF/HF are consistent with the third level drowsiness level, the drowsiness detection device detects that the subject is sleepy. The third stage can be determined to be the final drowsiness state.

In one embodiment, if the drowsiness stage sensed based on heart rate and the drowsiness stage sensed based on LF/HF do not match, the drowsiness stage sensed based on heart rate and LF The method of determining the lowest level of the drowsiness state detected based on /HF as the final drowsiness level (hereinafter referred to as the first method) reduces the need for sensitive detection of the level to be detected. It can be applied as a method for sensing stages.

For example, when the 1st, 2nd and 3rd stages of drowsiness are detected, a notification may be given, but in some cases, the 1st and 2nd stages of drowsiness may have little impact on the subject's safety. . At this time, if the subject is notified frequently, the subject may feel inconvenient or may not operate the drowsiness detection device. In addition, the drowsiness detection device can more accurately detect drowsiness by determining the lower drowsiness level among the drowsiness level detected based on heart rate and the drowsiness level detected based on LF/HF as the final drowsiness level. It can also sense conditions related to drowsiness. That is, determining the final drowsiness state using the drowsiness detection device according to the first method may be advantageous in terms of user's convenience and accuracy.

In one embodiment, the drowsiness stage sensed based on heart rate and the drowsiness stage sensed based on LF/HF are not consistent, and one of the drowsiness stages indicates a steady state, If another stage of drowsiness is detected as one or two stages higher than the steady state, the drowsiness sensing device determines whether the subject is in the final steady state based on the steady state, which is the lower stage. can decide that there is

For example, if the drowsiness sensing device senses a steady state based on the heart rate and senses the first stage drowsiness state based on LF/HF, the drowsiness sensing device can determine that the subject is in the final steady state. .

In one embodiment, if the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, one of the levels of drowsiness is the first level of drowsiness. , and when another level of drowsiness is detected as one level higher than the steady state, the drowsiness detection device detects the first level of drowsiness based on the first level of drowsiness, which is the lower level. It can be determined that one is in a stage 1 drowsy state.

For example, if the drowsiness sensing device senses the 1st stage drowsiness based on the heart rate and the 2nd stage drowsiness based on the LF/HF, the drowsiness sensing device detects that the subject is in the 1st stage drowsiness. can be determined to be

In one embodiment, if the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, one of the levels of drowsiness is the second level of drowsiness. , and if another level of drowsiness is detected as a higher level of drowsiness than the steady state, the drowsiness detection device detects the second level of drowsiness based on the second level of drowsiness, which is a lower level. can be determined to be On the other hand, the drowsiness level detected based on heart rate and the drowsiness level detected based on LF/HF do not match, and the drowsiness level detected based on heart rate and LF The method of determining the highest stage of the drowsiness state detected based on /HF as the final drowsiness stage (hereinafter referred to as the second method) is to detect the stage that needs to be sensitively detected. can be applied as a method of
For example, a notification may be given when the 1st, 2nd and 3rd stages of drowsiness are detected. can have an impact. At this time, if the instant drowsiness sensing device does not give a strong notification to the subject at the same time that the third stage drowsiness state is detected, the subject may be in a serious danger situation. In addition, the drowsiness detection device can be further diversified by determining the higher stage of the drowsiness state sensed based on the heart rate and the drowsiness state sensed based on the LF/HF as the final drowsiness stage. The stage of trying to perceive the situation can be determined as the stage of the final drowsiness state. That is, determining the final drowsiness state using the drowsiness sensing device according to the second method can prepare for the user's risk.

In one embodiment, the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, and one of the levels of drowsiness is the third level of drowsiness. and if another level of drowsiness is detected as one level, two levels or three levels lower than the level 3 drowsiness level, the drowsiness detection device indicates a high level level 3 level drowsiness state. Based on this, it can be determined that the subject is in the third stage final drowsiness state.

For example, if the drowsiness detector detects level 3 drowsiness based on heart rate and detects level 1 drowsiness based on LF/HF, the drowsiness sensor detects that the subject is in level 3 drowsiness. can decide that there is

In another embodiment, the drowsiness sensing device can also rely on a second method to sense a second final drowsiness state. In some cases, the drowsiness sensing device needs to sense not only the third-stage drowsiness state but also the second-stage drowsiness state in more diverse situations. Specifically, if the subject is a driver of a freight vehicle or a public transportation vehicle, if the driver faces the danger of a traffic accident, the surrounding vehicle drivers and passengers other than the driver Severely dangerous situations may arise. Therefore, the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, and one of the levels of drowsiness indicates the second level of drowsiness, If another level of drowsiness is detected as a level of drowsiness that is one or two levels lower than the second level of drowsiness, the drowsiness sensing device is measured based on the second level of drowsiness, which is a higher level. It is also possible to determine that a person is in the stage 2 terminal drowsiness state.

For example, if the drowsiness detector detects the 2nd stage drowsiness based on the heart rate and detects the 1st stage drowsiness based on the LF/HF, the drowsiness detector detects that the subject is in the 2nd stage drowsiness. can decide that there is

9. Diverse applications using the biometric index acquisition device
9.1 Various Embodiments of Display Apparatus The biometric acquisition method or drowsiness detection method described above can be used in a variety of applications. For example, the biometric acquisition method or drowsiness detection method can be applied to a display device.

At this time, the display device includes a smart mirror including a mirror display, but is not limited thereto, and may mean a device including a display such as a smart phone or a tablet.

Also, the biometric index or biometric information acquired by the biometric index acquiring method may be output through the display, and the drowsiness information acquired by the drowsiness sensing method may also be output through the display.

Also, the display device may perform an operation corresponding to the biometric information or the biometric information acquired by the biometric acquisition method.

Also, the display device may perform an operation corresponding to the drowsiness information obtained by the drowsiness sensing method.

Also, the user of the display apparatus may set a biometric index of interest, and when the biometric index of interest is set, only the biometric index of interest can be acquired or output. For example, if the first user sets the heart rate and the second user sets the blood pressure, only the heart rate can be obtained for the first user and only the blood pressure for the second user. is not limited to

In addition, the display device is installed in public places such as hotel lobbies, hotel front desks, public toilets, etc., and is used to measure the biometrics of people coming and going, but is not limited thereto.

In addition, the operation of the biometric device will be described below, but the described contents can be executed by other processors such as an ECU installed in a vehicle, and can also be executed by a processor included in a server.

In addition, the device described by the biometric measuring device may mean an image acquisition device, and in this case, the operation according to the described content may be executed by another processor such as an ECU or a processor included in the server. can be done.

FIG. 48 is a diagram for explaining a smart mirror device according to one embodiment.

Referring to FIG. 48, a smart mirror device 7000 according to one embodiment may include at least one of an image sensor 7010, a mirror display 7020, and a controller 7030. For example, the smart mirror device (7000) includes the mirror display (7020) and the controller (7030), but is not limited thereto.

At this time, the image sensor 7010 is provided as a visible light camera for obtaining a visible light image, an infrared camera for obtaining an infrared image, etc. Without limitation, a hybrid type camera can also be provided for acquiring visible light images and infrared images.

In addition, the image sensor 7010 is provided as one package with the mirror display 7020, but is not limited thereto, and is provided as a separate unit separate from the mirror display 7020. can also

Also, the image sensor 7010 can acquire a plurality of image frames and transmit the acquired images to the controller 7030 .

In addition, the mirror display 7020 may refer to a medium capable of transmitting information using a mirror function, but is not limited thereto.

In addition, the mirror display 7020 may include a mirror functioning as a mirror and a display capable of transmitting information, and may be provided in the form of adding a mirror film to the display, but is not limited thereto.

Also, the mirror display 7020 includes, but is not limited to, a half mirror, which means a half-transparent mirror that transmits light in one direction and reflects light in another direction.

Also, the mirror display 7020 includes, but is not limited to, a polarizer.

In addition, the mirror display (7020) may include a mirror display that can be commonly understood in addition to the exemplary principles and exemplary elements described above, and more specifically, performs a mirror function to provide information. can be understood as a device that performs the function of transmitting

In addition, the controller 7030 can obtain biometrics and biometric information based on a plurality of image frames obtained from the image sensor 7010 .

At this time, the method for obtaining the biometric index and biometric information based on the plurality of image frames obtained by the control unit 7030 from the image sensor 7010 is the same as described above, so the description will be repeated. will be omitted.

In addition, the control unit 7030 can acquire biometrics and biometric information from an external sensor (not shown). For example, the control unit (7030) can acquire heartbeat information through an ECG sensor or the like worn by the subject, but is not limited thereto.

Also, the controller 7030 can obtain personal and statistical data through an input device (not shown). For example, the control unit 7030 can acquire personal and statistical data such as height, age, and weight of the person to be measured through a keyboard, but is not limited thereto.

At this time, the input device may be a keyboard, mouse, key pad, dome switch, touch pad (constant pressure/power failure), jog wheel, jog switch, etc. Good, but not limited to this.

Also, the control unit 7030 can acquire personal and statistical data through an external device (not shown). For example, the control unit 7030 can acquire personal and statistical data such as height, age, and weight of the subject through the subject's smart phone, but is not limited thereto.

At this time, the external device may be a mobile terminal such as a mobile phone, smart phone, laptop computer, digital broadcasting terminal, PDA (Personal Digital Assistants), PMP (Portable Multimedia Player), navigation, etc. including, but not limited to, fixed terminals such as digital TVs, desktop computers, and the like.

In addition, the controller (7030) can control the operation of the mirror display (7020). For example, the control unit 7030 may control the operation of the mirror display 7020 to output biometrics or biometric information acquired based on the image frame, but is limited to this. not.

Also, the controller 7030 can control the operation of the mirror display 7020 to output various information. For example, the controller 7030 can control the operation of the mirror display 7020 to output various information such as weather information, date information, calendar information, internal humidity information, internal temperature information, etc. but not limited thereto.

Also, the controller 7030 can control the operation of the mirror display 7020 to output various personal information. For example, the control unit 7030 can control the operation of the mirror display 7020 to output various personal information such as schedule information and medication information of the subject. Not limited.

FIG. 49 is a diagram for explaining a smart mirror device according to one embodiment.

Referring to FIG. 49, a smart mirror device (7100) according to one embodiment may include an image sensor (7110) and a mirror display (7120).

At this time, the image sensor 7110 and the mirror display 7120 are applied to the above-described operation, so the repeated description will be omitted.

According to one embodiment, basic information can be output to the mirror display (7120). For example, as shown in FIG. 49, at least one of basic information such as date information, time information, outside temperature information, weather information, calendar information, and news information is output to the mirror display 7120. , but not limited to.

Also, personal information can be output to the mirror display (7120). For example, as shown in FIG. 49, the mirror display 7120 outputs at least one of personal information such as age, height, and weight of the person to be measured, but is not limited thereto.

Also, for example, as shown in FIG. 49, at least one of personal information such as schedule information and medication information of the subject is output to the mirror display 7120, but the present invention is not limited thereto.

In addition, the subject's biometric index can be output to the mirror display (7120). For example, as shown in FIG. 49, at least one biometric among biometric indices such as heart rate, oxygen saturation, blood pressure, etc. is output to the mirror display 7120, but is not limited thereto.

At this time, the biometric index may be obtained based on the image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained by an external sensor. .

In addition, biological information of the subject can be output to the mirror display (7120). For example, as shown in FIG. 49, at least one biometric information such as condition information is output to the mirror display 7120, but the present invention is not limited thereto.

At this time, the biometric information may be obtained based on image frames obtained from the image sensor 7110, but is not limited thereto, and may be obtained from an external sensor or the like.

Also, the biometric information may be obtained based on at least one biometric among the biometrics, but is not limited thereto.

In addition, the subject's biosignals can be output to the mirror display (7120). For example, as shown in FIG. 49, at least one biological signal among biological signals such as a heartbeat signal is output to the mirror display 7120, but the embodiment is not limited thereto.

At this time, the biological signal may be obtained based on the image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained based on an external sensor.

Also, the mirror display (7120) may include an input device. For example, as shown in FIG. 49, the mirror display 7120 includes at least one input device such as a touch panel, but is not limited thereto.

Also, for example, although not shown in FIG. 49, the smart mirror device (7100) recognizes and receives from the user gestures such as trekking the user's pupil, but is limited to this. not.

Also, the mirror display 7120 can acquire information about the subject through the input device, and the obtained information about the subject can be output to the mirror display 7120 .

In addition, the drawings and description of the mirror display 7120 described above are merely examples, and the display is not limited to FIG. 49 and related descriptions, and it is obvious that various information can be output in various ways.

9.1.1 Various Embodiments of Display Apparatus Outputting Guide Area FIG. 50 is a diagram for explaining a smart mirror apparatus outputting a guide area according to an embodiment.

Referring to FIG. 50, a smart mirror device (7150) according to one embodiment may include an image sensor (7160) and a mirror display (7170), the image sensor (7160) and the mirror display (7170) , the above-mentioned contents are applied, so the repeated description is omitted.

Also, a guide area 7180 may be displayed on the mirror display 7170 according to one embodiment. At this time, the guide area 7180 can function to roughly grasp the measurement position of the object to be measured.

When the biometric index is obtained using an image sensor, the measurement accuracy may vary depending on the measurement position of the subject, so the guide area 7180 that can guide the measurement position of the subject is used. In this case, it is possible to improve the accuracy of the measurement of the subject's biometric index.

In addition, the guide area 7180 is displayed in a rectangular shape as shown in FIG. 50, but is not limited thereto, and can be displayed in various shapes such as a human face contour, a circle, and an ellipse. .

In addition, the guide area 7180 can change its displayed position depending on the subject. For example, the guide area (7180) for tall subjects can be located relatively above the mirror display (7170), and the guide area (7180) for short subjects can be It may be positioned relatively below the mirror display 7170, but is not limited thereto.

Also, the position of the guide area 7180 can be changed in real time. For example, when the person to be measured moves during measurement, the guide area 7180 changes its position in real time corresponding to the movement of the person to be measured, but is not limited thereto.

Also, the guide area 7180 can be displayed before measuring the biometrics of the subject. For example, the guide area 7180 can be displayed before the subject enters the measurement area and can function to teach the subject's general measurement position, but is not limited thereto.

In addition, the guide area (7180) can perform the function of teaching the person to be measured when there are multiple persons to be measured. For example, when person A and person B enter the measurement area, but only person B is the person to be measured, the guide area (7180) can be displayed corresponding to person B. Not limited.

9.1.2. Various Embodiments of Display Apparatus for Displaying Predetermined Information During Measurement of Biometric Index When the biometric index is obtained using an image sensor, measurement accuracy varies depending on the movement of the subject. There is a need for features to minimize subject movement during biometric measurements.

Therefore, when a display device displaying predetermined information during biometric index measurement is used, the subject's movement can be minimized using the displayed information.

FIG. 51 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment.

Referring to FIG. 51, the smart mirror device 7200 according to one embodiment displays different information depending on the biometric measurement time.

More specifically, at the first time point (7210), biometric measurement for the subject can begin.

At this time, the first point in time (7210) may be a point in time when the measurement target area of the subject is positioned within the measurement area, but is not limited thereto. It is the point in time when it is located within the angle of view of the image sensor.

Also, the first point in time (7210) is the point in time when the person to be measured inputs his or her will for measurement. For example, the first time point 7210 may be a time point when the subject touches the measurement button, but is not limited thereto.

Also, the first information can be displayed at the second time point (7220).

At this time, the second time point 7220 may be the time point when the face recognition of the subject is completed, but is not limited thereto.

In addition, the second point in time 7220 may mean a point in time after a predetermined period of time has elapsed after the start of bioindex measurement for the subject, but is not limited thereto.

Also, the first information is recognition information for the subject. For example, as shown in FIG. 51, the first information displayed at the second point in time (7220) includes, but is not limited to, greeting word information for the subject whose face has been recognized. .

Also, the first information is information about the measurement progress time. For example, if the second time point (7220) is 2 seconds after the start of measurement, information corresponding to 2 seconds can be displayed.

Also, the first information is information on the remaining time to perform the measurement. For example, if 6 seconds are required to measure the biometric index and the second time point (7220) is 2 seconds after starting the measurement, information corresponding to the remaining 4 seconds can be displayed.

Also, the second information can be displayed at the third time point (7230).

At this time, the third point in time 7230 may mean a point in time after a predetermined time has passed after the face recognition of the subject is completed, but is not limited thereto.

In addition, the third point in time 7230 may mean a point in time after a predetermined period of time has elapsed after the start of bioindex measurement for the subject, but is not limited thereto.

Also, the second information is health information for the subject. For example, as shown in FIG. 51, the second information displayed at the third time point (7230) may be the subject's average monthly biometric index, but is not limited thereto.

Also, the second information is information about the measurement progress time. For example, if the third time point (7230) is 4 seconds after the start of measurement, information corresponding to 4 seconds can be displayed.

Also, the second information is information on the remaining time to perform the measurement. For example, if 6 seconds are required to measure the biometric index and the third time point (7230) is 4 seconds after starting the measurement, information corresponding to the remaining 2 seconds can be displayed.

Also, the third information can be displayed at the fourth time point (7240).

At this time, the fourth point in time 7240 may mean the point in time when the biometric measurement of the subject is completed, but is not limited thereto.

Also, the third information is information related to the biometric index of the subject. For example, as shown in FIG. 51, the third information displayed at the fourth time point (7240) may be the subject's heart rate, oxygen saturation, and blood pressure. Not limited.

Also, the third information is information related to biological information of the subject. For example, although not shown in FIG. 51, the fourth information displayed at the fourth time point (7240) may be biological information such as the condition index of the subject, but is not limited thereto. .

9.1.3 Various Embodiments of Smart Mirror Device Displaying Images Acquired Through Image Sensor FIG. 52 is a diagram for explaining a smart mirror device according to one embodiment.

Referring to FIG. 52, a smart mirror device (7250) according to one embodiment may include an image sensor (7260) and a mirror display (7270).

At this time, the image sensor 7260 and the mirror display 7270 are applied to the above-described operation, so the repeated description will be omitted.

According to one embodiment, an image can be displayed on the mirror display (7270). For example, as shown in FIG. 52, the image acquired through the image sensor 7260 can be displayed on the mirror display 7270, but is not limited thereto.

In addition, the displayed image may display information about the subject to be measured. For example, an area displaying a measurement object included in the image is displayed, but is not limited thereto.

In this way, the subject can see himself/herself through the mirror display 7270, and the additional display of the image is useful for biometric measurement when there are a plurality of measurement subjects. By clearly displaying the person to be measured, it is possible to prevent crosstalk of multiple persons to be measured.

In addition, the image of the person to be displayed and the image of the person to be measured reflected through the mirror display 7270 are positioned at different locations, but are not limited to, and may overlap each other. can also

9.1.4. Various Embodiments of Display Apparatus for Measuring Biometrics in Real Time FIG. 53 is a diagram illustrating a display apparatus for measuring biometrics in real time according to an embodiment.

Referring to FIG. 53, a display device 7300 according to one embodiment can measure biometrics in real time, and display the biometrics on the display device 7300 in real time.

More specifically, the biometric index for the subject obtained in the first time interval at the first time point (7310) can be displayed.

At this time, the first time point 7310 is the time after the biometric index of the subject is acquired in the first time interval.

In addition, the biometric index of the subject obtained in the second time interval at the second time point (7320) can be displayed.

At this time, the second time point (7320) is the time after the biometric index of the subject is acquired in the second time interval.

Also, the second time interval may be different from the first time interval and may at least partially overlap.

Also, the second time point (7320) is later than the first time point (7310).

In addition, the biometric index of the subject obtained in the third time interval at the third time point (7330) can be displayed.

At this time, the third time point (7330) is the time after the biometric index of the subject is acquired in the third time interval.

Also, the third time interval may at least partially overlap, unlike the first and second time intervals.

Also, the third point in time (7330) is later than the first and second points in time (7310, 7320).

In addition, the biometric index of the subject obtained in the fourth time interval at the fourth time point (7340) can be displayed.

At this time, the fourth time point (7340) is a time point after the biometric index of the subject is acquired in the fourth time interval.

Also, the fourth time interval may at least partially overlap, unlike the first, second and third time intervals.

Also, the fourth time point (7340) is later than the first, second and third time points (7310, 7320, 7330).

Also, as described above, if the biometrics are acquired in real time, the biometrics acquired at a particular point in time can be used to save the final biometrics. For example, but not limited to, the bioindex obtained when the bioindex was obtained for the first time can be stored, and the bioindex obtained when the bioindex was last obtained can be stored; It is also possible to store bioindexes obtained at intermediate bioindex measurements.

Also, as described above, when acquiring bioindexes in real time, multiple bioindexes can be used to store the final bioindex. For example, an average of a plurality of biometrics obtained during a certain time interval may be stored as a final biometric, but is not limited thereto.

In addition, as described above, when acquiring bioindex in real time, even if noise occurs due to the movement of the person being measured between bioindex measurements, an accurate bioindex can be obtained by acquiring a plurality of bioindexes. be able to.

In addition, as described above, if the biometric index is obtained in real time, an update period for displaying the biometric index can be set. For example, the user can set a period for updating the biometric index to be displayed, and the biometric index is updated according to the set period, but is not limited thereto.

9.1.5. Various Embodiments of Display Device Displaying Predetermined Information During Biometric Measurement FIG. 54 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment. .

Referring to FIG. 54, the smart mirror device 7350 according to one embodiment can display different information depending on the biometric measurement time.

More specifically, the first point in time (7360) is the point in time before or at which the biometric measurement for the subject begins.

At this time, the first time point (7360) may be, but is not limited to, a time point when the measurement target area of the subject is positioned within the measurement area, and the person to be measured is positioned on the image sensor. This is the point in time within the angle of view.

Also, the first information can be displayed at the first time point (7360).

At this time, the first information may include basic information. For example, as shown in FIG. 54, it includes basic information such as date, time, and weather information, but is not limited thereto.

Also, the first information may include information about the guide area. For example, as shown in FIG. 54, the guide area can be displayed at the first time point 7360, but is not limited thereto.

Also, the second information can be displayed at the second time point (7370).

At this time, the second time point 7370 may be the time point of measuring the biometrics of the subject, but is not limited thereto.

In addition, the second point in time 7370 may mean a point in time after a predetermined period of time after the start of bioindex measurement for the subject, but is not limited thereto.

Also, the second information may include personal information about the subject. For example, as shown in FIG. 54, the second information displayed at the second time point (7370) includes, but is not limited to, the subject's main schedule, medication information, and other personal information. .

Also, the second information may include information related to biometric measurements. For example, as shown in FIG. 54, the second information displayed at the second time point (7370) includes information indicating that the subject's biometric index is being measured. Not limited.

Also, the third information can be displayed at the third time point (7380).

At this time, the third point in time (7380) may be, but is not limited to, the point at which the bioindex measurement for the subject is completed, and at least one measurement is completed in the case of real-time measurement. It is time.

Also, the third information may include a measured biometric index. For example, as shown in FIG. 54, the third information displayed at the third time point (7380) includes, but is not limited to, measured heart rate, oxygen saturation and blood pressure.

Further, as shown in FIG. 54, the first information and the second information are simultaneously displayed at the second time point (7370), and the first, second and third information are displayed at the third time point (7380). can be displayed simultaneously.

9.1.6. Various Embodiments of Smart Mirror Device Including Opening/Closing Device FIG. 55 is a diagram for explaining a smart mirror device including an opening/closing device according to one embodiment.

Referring to FIG. 55, a smart mirror device (7400) according to one embodiment may include an image sensor (7410), a mirror display (7420) and an opening/closing device (7430).

At this time, the image sensor 7410 and the mirror display 7420 are the same as described above, so a repeated description will be omitted.

In addition, the open state of the opening/closing device (7430) described below may mean a state in which a window for the image sensor (7410) to acquire an image is secured. A closed state may mean a state in which a window for the image sensor 7410 to acquire an image is not secured.

In addition, the image sensor 7410 is used to sense the open state and the closed state. For example, when the degree of furniture sensed by the image sensor 7410 is below a reference value, the closed state is sensed. , The open state is detected when the furniture is greater than or equal to a reference value, but is not limited thereto, and the open state and the closed state are detected using the image sensor 7410 in various ways. be able to.

In addition, an external sensor is used to detect the open state and the closed state. When it is detected as a closed state and when it is opened below a reference value, it can be detected as a closed state, but is not limited thereto, and the open state and the closed state are detected using the external sensor in various ways. A state can be sensed.

According to one embodiment, the opening/closing device (7430) can perform opening/closing with respect to the image sensor (7410). For example, the opening/closing device 7430 opens or closes the front of the image sensor 7410 housed in the internal housing space to adjust the image acquired by the image sensor 7410, or to control the image from the outside. Sensors (7410) can be made unobservable.

More specifically, at the first time point 7440 when the opening/closing device 7430 is in the closed state, the image sensor 7410 may not be observed outside by the opening/closing device 7430. .

Also, the image sensor 7410 may not acquire an image at the first time point 7440 . For example, power to the image sensor 7410 may be cut off by closing the opening/closing device 7430, but the embodiment is not limited thereto.

Also, at the second time point 7450 when the opening/closing device 7430 is in the open state, the image sensor 7410 can be observed outside.

Also, the image sensor 7410 can acquire an image at the second time point 7450 . For example, power is supplied to the image sensor 7410 through the opening operation of the opening/closing device 7430, but the embodiment is not limited thereto.

In addition, the surface of the floor cover opening/closing device 7430 is made of a mirror, and is made of the same material as the outer surface of the mirror display 7420, but is not limited thereto.

In addition, the opening/closing device 7430 may protect the image sensor 7410 from external dust. For example, when the opening/closing device 7430 is in a closed state, external dust is prevented from contacting the image sensor 7410, but the present invention is not limited thereto.

9.1.7. Various Embodiments of Smart Mirror Device Placed in a Shoe Box A smart mirror device placed at the doorway of a building can measure the biometrics of a person entering or leaving the house. Generally, people wear clothes in the crate of a house or the doorway of a building, so there is little privacy problem due to the image sensor, and it is easy to install the image sensor for measuring the biometric index.

In the following, a description will be given of a smart mirror device arranged at the entrance of a building, which is represented by a clog box, and the smart mirror device arranged in the clog box will be described for convenience of explanation.

FIG. 56 is a diagram for explaining a smart mirror device placed in a crate according to one embodiment.

Referring to FIG. 56, a smart mirror device (7460) according to one embodiment may include an image sensor (7470) and a mirror display (7480).

At this time, the image sensor 7470 and the mirror display 7480 are applied to the above-described contents, so the repeated description will be omitted.

The image sensor (7470) according to one embodiment can acquire an image of the subject (7490). For example, the image sensor (7470) can acquire images of the subject (7490) entering and exiting the crate where the smart mirror device (7460) is located.

Also, the smart mirror device 7460 can acquire a biometric index based on the image frames acquired from the image sensor 7470 .

Also, the mirror display (7480) can reflect and provide the figure of the subject (7490). For example, the subject (7490) can observe himself/herself through the mirror display.

Also, the mirror display (7480) can display the biometric index of the subject (7490). For example, the mirror display 7480 can display biometrics obtained based on image frames obtained from the image sensor 7470, but is not limited thereto.

Also, the mirror display (7480) can display biological information of the subject (7490). For example, the mirror display 7480 can display biometric information obtained based on the obtained biometric index, but is not limited thereto.

Also, the mirror display (7480) can be formed in at least a part of the smart mirror device (7460). For example, as shown in FIG. 56, the mirror display 7480 can be arranged to occupy a predetermined area in the center of the smart mirror device 7460. FIG.

Also, although not shown in FIG. 56, it is obvious that the mirror display (7480) can display basic information, personal information, and the like.

9.1.7.1. Various Embodiments of Smart Mirror Device Using Trigger Signal When the smart mirror device for biometric measurement continuously operates, power consumption for acquiring and analyzing images is high. can be sustained. Therefore, the trigger signal can be utilized to save energy and provide a smarter smart mirror device.

FIG. 57 is a flow chart illustrating a method of operating a smart mirror device according to one embodiment.

Referring to FIG. 57, a method of operating a smart mirror device (7500) according to an embodiment includes acquiring an on-trigger (S7510) and acquiring a biometric index of a subject (S7520). , acquiring an off-trigger (S7530), and stopping at least one operation of the smart mirror device (S7540), but is not limited thereto.

More specifically, the method of operating a smart mirror device (7500) according to one embodiment may include obtaining an on-trigger (S7510).

At this time, the on-trigger is a trigger signal for starting at least one operation of the smart mirror device. For example, the on-trigger may be, but is not limited to, a trigger signal for an image sensor to perform an operation of acquiring an image, and turns on the power of the smart mirror device. a trigger signal for initiating at least one operation of the smart mirror device, such as to

Also, the on-trigger can be provided in various ways.

For example, if the smart mirror device includes a motion sensor, the on-trigger can be obtained from the motion sensor. More specifically, when a predetermined motion is sensed from the motion sensing sensor, the smart mirror device can obtain a corresponding trigger signal, and such a trigger signal can become the on-trigger. It can, but is not limited to.

Also, at this time, the motion sensor is disposed inside the smart mirror device, but is not limited thereto, and may be disposed outside the smart mirror device.

Also, for example, when the smart mirror device includes an input device such as a touch panel, the on-trigger can be obtained from the input device. More specifically, a predetermined input can be obtained from the input device, the smart mirror device can obtain a corresponding trigger signal therefor, and such trigger signal can be the on-trigger, It is not limited to this.

Also, for example, if the smart mirror device includes an image sensor, the on-trigger can be obtained from the image sensor. More specifically, a lighting device located in a crate can detect a predetermined movement and emit light, the image sensor can detect the light emitted from the lighting device, and the smart mirror device can detect the light emitted from the lighting device. A corresponding trigger signal can be obtained, and such trigger signal can be the on-trigger, but is not limited thereto.

Also, for example, when an entrance/exit door is included in the angle of view of the image sensor, the on-trigger can be acquired based on the change in the entrance/exit. More specifically, the image sensor can determine that the doorway is opened, and the smart mirror device can obtain a corresponding trigger signal accordingly, and such a trigger signal can be the on-trigger, It is not limited to this.

Also, the method of operating a smart mirror device (7500) according to an embodiment may include obtaining a biometric index of the subject (S7520).

At this time, the biomedical signal for the person to be measured is obtained based on the image frames obtained through the image sensor, and detailed description thereof will be omitted as described above.

Also, the method of operating a smart mirror device (7500) according to an embodiment may include obtaining an off-trigger (S7530).

At this time, the off-trigger is a trigger signal for stopping at least one operation of the smart mirror device. For example, the off-trigger may be, but is not limited to, a trigger signal for stopping the operation of the image sensor to acquire an image. is a trigger signal for stopping at least one operation of

Also, the off-trigger can be provided in various ways.

For example, if the smart mirror device includes a motion sensor, the off-trigger can be obtained from the motion sensor. More specifically, if the motion detection sensor does not sense a predetermined motion for a certain period of time, the smart mirror device can obtain a corresponding trigger signal, and such a trigger signal is sent to the off-trigger. can be, but is not limited to.

Also, for example, when the smart mirror device includes an input device such as a touch panel, the off-trigger can be obtained from the input device. More specifically, a predetermined input can be obtained from the input device, for which the smart mirror device can obtain a corresponding trigger signal, such trigger signal can be the off-trigger, It is not limited to this.

Also, for example, if the smart mirror device includes an image sensor, the off-trigger can be obtained from the image sensor. More specifically, the lighting device located in the crate can detect that there is no predetermined movement for a certain period of time and turn off, the image sensor can detect the intensity of the turned off light, and the smart mirror device. can obtain a corresponding trigger signal therefor, such trigger signal can be the off-trigger, but is not limited thereto.

Also for example, the off-trigger can be obtained based on biometric acquisition of the smart mirror device. More specifically, if the measurement target area does not exist within the angle of view of the image sensor, biometrics cannot be acquired, and if biometrics are not acquired for a certain period of time, the smart mirror device acquires a corresponding trigger signal. The trigger signal can be the off-trigger, but is not limited thereto.

Also, for example, when a doorway is included in the angle of view of the image sensor, the off-trigger can be acquired based on a change in the doorway. More specifically, the image sensor can determine that the doorway is opened, and the smart mirror device can obtain a corresponding trigger signal accordingly, and such trigger signal can be the off-trigger. but not limited thereto.

58 and 59 are diagrams for explaining the operation of a smart mirror device using a trigger signal according to one embodiment.

58 and 59, a smart mirror device (7550) according to one embodiment may include an image sensor (7560) and a mirror display (7570).

At this time, the image sensor 7560 and the mirror display 7570 are applied to the above-described contents, so the repeated description will be omitted.

Referring to FIG. 58 according to one embodiment, at least one operation of the smart mirror device can be initiated when the subject (7580) is positioned in a predetermined on-trigger generation area.

At this time, when the smart mirror device includes a motion sensor, the predetermined on-trigger generation area may be the sensing range of the motion sensor, or the sensing range of the motion sensor of the lighting device installed in the crate. may be, but is not limited to.

Also, referring to FIG. 59 according to one embodiment, when the smart mirror device 7550 receives a predetermined input from the subject 7580, at least one operation of the smart mirror device can be initiated. .

At this time, the predetermined input is obtained through an input device such as a touch panel, but is not limited thereto.

9.1.7.2. Various Embodiments of Smart Mirror Device Measuring Biometrics According to Priority When there are multiple people in the biometric measurement area of the smart mirror device, biometrics can be measured according to priority.

FIG. 60 is a diagram for explaining the operation method of the smart mirror device according to one embodiment.

Referring to FIG. 60, a smart mirror device operating method (7600) according to an embodiment includes acquiring an image including a plurality of people (S7610), prioritizing biometric measurements (S7620), prioritizing The method includes at least one step of obtaining the biometric index according to the rank (S7630) and outputting the obtained biometric index (S7640), but is not limited thereto.

More specifically, the method of operating a smart mirror device (7600) according to an embodiment may include acquiring an image including a plurality of people (S7610), where the content of acquiring the image described above is Since it is applied, the repeated explanation will be omitted.

In addition, the method of operating the smart mirror device (7600) according to an embodiment may include determining priorities for biometric measurements (S7620) and obtaining biometrics according to the priorities (S7630).

At this time, ease of biometric measurement may be considered to determine the priority. For example, a biometric index for a person occupying a larger area from the images acquired by the plurality of persons is measured, but the embodiment is not limited thereto.

Also, for example, a biometric index of a person recognized as a person is measured from the images captured by the plurality of persons, but the present invention is not limited thereto.

Also, for example, a biometric index for a person occupying an area closer to the center from the images acquired by the plurality of persons is measured, but is not limited thereto.

In addition, for example, the biometric index of a person positioned closer to the image sensor is measured in the images captured by the plurality of people, but the present invention is not limited thereto.

In addition, pre-stored information may be used to determine the priority. For example, a biometric index of a person previously stored as a person to be measured is measured from the plurality of persons, but the present invention is not limited thereto.

In addition, for example, when the biometric index is obtained according to a preset priority order, and more specifically, when the priority order for the child among the family members is set high, the child and the father are located in the biometric index measurement area at the same time. When doing so, the child's biometric index is measured, but is not limited thereto.

In addition, biometric measurement order information may be used to determine the priority. For example, if an image for a first subject is acquired first and an image for the second subject is acquired later, the biometric index for the first subject is measured, but is not limited thereto. .

Also, the method of operating a smart mirror device (7600) according to one embodiment may include outputting the acquired biometric index (S7640).

At this time, the biometric index is the biometric index for the person determined according to the priority.

Also, the biometric index can be output together with an indicator for the measured person. For example, a guide area for a person who is determined according to the priority order and whose biometric index is to be measured is output, but the present invention is not limited thereto.

FIG. 61 is a diagram for explaining the operation of the smart mirror device according to one embodiment.

Referring to FIG. 61, a smart mirror device (7650) according to one embodiment may include an image sensor (7660) and a mirror display (7670).

At this time, since the above-described contents are applied to the image sensor 7660 and the mirror display 7670, the repeated description will be omitted.

Referring to FIG. 61 according to one embodiment, a first person (7680) and a second person (7690) can be located within the measurement range of the image sensor (7660).

Also, the first person (7680) can be a target of biometric measurement of the smart mirror device (7650).

For example, as shown in FIG. 61, when the first person (7680) is located closer to the image sensor (7660) than the second person (7690), the first person (7680) It can be a bioindex measurement target, but is not limited thereto.

Further, for example, in the image acquired by the image sensor (7660), if the area occupied by the image of the first person (7680) is larger than the area occupied by the image of the second person (7690), the first person (7690) A person (7680) can be a biometric subject, but is not limited to this.

Also, for example, if only the first person (7680) is recognized as a person in the smart mirror device (7650), the first person (7680) can be a biometric measurement target. is not limited to

Also, for example, in the image acquired by the image sensor (7660), the area occupied by the image of the first person (7680) is closer to the center than the area occupied by the image of the second person (7690). , the first person (7680) can be a biometric measurement target, but is not limited thereto.

Also, for example, if the priority for the first person (7680) is stored higher than the priority for the second person (7690), the first person (7680) can be a biometric measurement target. but not limited thereto.

In addition, for example, when the first person (7680) is stored as a person to be measured, the first person (7680) can be a bioindex measurement target, but is not limited thereto.

Also, when the biometric index for the first person (7680) is measured, the biometric index for the first person (7680) may be output. For example, as shown in FIG. 61, heart rate, oxygen saturation, and blood pressure for the first person (7680) are output, but are not limited thereto.

Also, when the biometric index for the first person (7680) is measured, an indicator for the first person (7680) may be output. For example, as shown in FIG. 61, the guide area for the first person (7680) is output, but is not limited thereto.

9.1.7.3.Various Embodiments of Smart Mirror Device for Measuring Biometrics of Multiple Subjects When there are multiple people within the biometric measurement area of the smart mirror device, biometrics are measured according to priority. can be done.

FIG. 62 is a diagram for explaining the operation method of the smart mirror device according to one embodiment.

Referring to FIG. 62, a method of operating a smart mirror device (7700) according to an embodiment includes acquiring an image including a plurality of subjects (S7710) and acquiring biometrics for the plurality of subjects. (S7720) and at least one step of outputting biometrics for a plurality of subjects (S7730), but is not limited thereto.

More specifically, the method of operating a smart mirror device (7700) according to an embodiment may include acquiring an image including a plurality of subjects (S7610), for which the above-described image is acquired. Since the content applies, the repeated explanation will be omitted.

Also, the method of operating a smart mirror device (7700) according to an embodiment may include acquiring biometrics for a plurality of subjects (S7720), to which the biometrics acquisition method described above is applied. Therefore, repeated explanations will be omitted.

According to one embodiment, a region can be set for each of a plurality of subjects to obtain biometrics for the plurality of subjects. For example, if a first subject and a second subject are located within the biometric measurement area, the area for the first subject and the area for the second subject can be set.

Also, multiple biometric indices can be obtained for multiple subjects. For example, the heart rate, oxygen saturation, and blood pressure of a plurality of subjects are obtained, but not limited thereto.

Also, the types of biometric indices acquired for each of the plurality of subjects can be different from each other. For example, the heart rate of the first subject is obtained, and the oxygen saturation and blood pressure of the second subject are obtained, but are not limited thereto.

In addition, a detailed area may be set for each of a plurality of subjects to obtain a plurality of biometrics of the plurality of subjects. For example, areas are set for a first subject, and a first detailed area for measuring heart rate, a second detailed area for measuring oxygen saturation, and blood pressure for the first subject are set. A third detail area is set for measuring heart rate, an area is set for a second subject, a fourth detail area is set for measuring heart rate for said second subject, and a fourth detail area is set for measuring oxygen saturation. and a sixth detailed area for measuring blood pressure, but not limited thereto.

Also, the first to sixth detail areas may be different from each other and at least partially overlap each other.

Also, the method of operating a smart mirror device (7700) according to an embodiment may include outputting biometrics for a plurality of subjects (S7730).

At this time, the biometric index may be at least one biometric index for each of a plurality of subjects.

Also, while outputting the biometric index, an indicator for a plurality of subjects can be output together. For example, when the biometrics of a first subject and a second subject are acquired, the guide regions for the first subject and the second subject are output, but the present invention is not limited thereto.

In addition, while outputting the biometric index, an indicator indicating what kind of subject the output biometric index is can be output together. For example, when a biometric index for a first subject is output, an indicator capable of identifying the first subject may be output together, and when a biometric index for a second subject is output, a second An indicator that can identify the subject can be output together.

FIG. 63 is a diagram for explaining the operation of the smart mirror device according to one embodiment.

Referring to FIG. 63, a smart mirror device (7750) according to one embodiment may include an image sensor (7760) and a mirror display (7770).

At this time, the image sensor 7760 and the mirror display 7770 are applied to the above-described contents, so the repeated description will be omitted.

Referring to FIG. 63 according to one embodiment, a first subject (7780) and a second subject (7790) can be located within the measurement range of the image sensor (7760).

Further, a region for each of the plurality of subjects is set within the image frame acquired to acquire the biometrics for the first subject (7780) and the second subject (7790). can be done.

Also, a biometric index for the first subject (7780) and the second subject (7790) can be obtained. For example, the heart rate for the first subject (7780) and the second subject (7790) is obtained, but is not limited thereto.

Also, a plurality of biometric indices for each of the first subject 7780 and the second subject 7790 can be obtained. For example, as shown in FIG. 63, the heart rate, oxygen saturation and blood pressure for the first subject (7780) are obtained, and the heart rate, oxygen saturation and blood pressure for the second subject (7790) are obtained. is obtained, but is not limited thereto.

Also, for example, although not shown in FIG. 63, the heart rate of the first subject (7780) is obtained, and the oxygen saturation and blood pressure of the second subject (7790) are obtained, It is not limited to this.

9.1.7.4. Various embodiments of smart mirror device that outputs other information depending on the situation The information needed for

Therefore, the smart mirror device can output different information so as to provide different information to the user depending on the situation.

In addition, the smart mirror device can output different biometrics to provide different biometrics to the user according to the situation.

FIG. 64 is a diagram for explaining the operation method of the smart mirror device according to one embodiment.

Referring to FIG. 64, a method of operating a smart mirror device (7800) according to an embodiment includes determining information provision status (S7810), outputting first information according to the first status (S7820), and The step of outputting the second information according to the second situation (S7830) includes at least one step, but is not limited thereto.

More specifically, the method of operating a smart mirror device (7800) according to one embodiment may include determining an information provision status (S7810).

At this time, the information provision status may mean the status of the smart mirror device user. For example, the information provision situation includes, but is not limited to, a situation in which the user of the smart mirror device enters or exits a building.

Also, the information provision status may mean a situation determined simply to provide information. For example, the information provision status includes, but is not limited to, a first status or a second status.

Also, the information provision status may mean a status related to the presence or absence of biometric measurement. For example, the information providing situation includes, but is not limited to, a situation in which it is not necessary to measure biometrics or a situation in which biometrics are required to be measured.

Also, the information providing status may mean a time-related status. For example, the information provision status includes, but is not limited to, morning status or afternoon status.

Also, the information provision status may mean a status according to an external environment. For example, the information provision status includes, but is not limited to, external environment conditions such as external epidemic conditions.

Also, the smart mirror device may include additional sensors to determine the information provision status. For example, a motion detection sensor may be added to determine the user's entry/exit status, and if the user's motion is first detected by the first motion detection sensor, the first status is determined, and the second status is determined. If the user's motion is first detected by the motion detection sensor, it is determined as a second situation, but is not limited thereto.

Also, an image sensor included in the smart mirror device may be used to determine the information provision status. For example, using the image frames acquired through the image sensor to determine the entry/exit status of the user, and more specifically, the user in a plurality of image frames acquired in time series. If the region is closer to the central portion from the first side, it is determined as the first situation, and if it is closer to the central portion from the second side, it is determined as the second situation, but is not limited thereto.

Also, an image sensor included in the smart mirror device may be used to determine the information provision status. For example, in order to determine whether the situation requires biometric measurement and information provision, the image frames acquired through the image sensor are used, and more specifically, a plurality of time-series acquired In the image frame, if the movement speed of the area corresponding to the user exceeds the reference value, it is determined as the first situation, and if the movement speed is equal to or less than the reference value, it is determined as the second situation. Not limited.

Also, time information can be used to determine the information provision status. For example, if it is before the reference time, it is determined as the first situation, and if it is after the reference time, it is determined as the second situation, but the present invention is not limited thereto.

Also, recognition information can be used to determine the information provision status. For example, if the face recognition information of a person is used to determine whether the situation requires biometric measurement and information provision, and more specifically, if the user is registered in the face recognition result, If the user is determined as the first situation and is not registered, it is determined as the second situation, but is not limited thereto.

In addition, moving direction information of the person to be measured can be used to determine the information provision status. For example, information on the direction of movement of the subject sensed or sensed using an external sensor is used to determine the information provision situation, and more specifically, the subject moves in the first direction. If the subject moves in a second direction, it is determined as a second situation, but is not limited thereto.

In addition, the method of operating a smart mirror device (7800) according to an embodiment includes outputting first information according to a first situation (S7820) and outputting second information according to a second situation (S7830). , but not limited to.

At this time, the first information and the second information may differ from each other when the first situation includes a situation of leaving the building and the second situation includes a situation of entering the building.

For example, the first information includes outside weather information, today's schedule information, product information according to outside weather, etc., but is not limited thereto, and may be information according to the first situation.

Also, the second information includes, but is not limited to, internal temperature information, internal humidity information, security information, internal air quality information, etc., and may be information according to the second situation.

Also, at least one of the first information and the second information includes biometric information, but is not limited thereto.

Also, the biometric index included in the first information and the biometric index included in the second information may be different from each other. For example, providing a biometric index of a first subject can output heart rate, and providing a biometric index of a second subject can output blood pressure, but is limited to this. not.

65 and 66 are diagrams for explaining the operation of the smart mirror according to one embodiment.

65 and 66, a smart mirror device (7850) according to one embodiment may include an image sensor (7860) and a mirror display (7870).

At this time, the image sensor 7860 and the mirror display 7870 are applied to the above-described contents, so the repeated description will be omitted.

According to one embodiment, FIG. 65 is a diagram illustrating operation of the smart mirror device (7850) in a first situation, and FIG. 66 is a diagram illustrating operation of the smart mirror device (7850) in a second situation. is.

In addition, the information output in the first situation and the second situation are different from each other, and the above-mentioned contents are applied to this, so a repeated explanation will be omitted.

Also, an external sensor (7861) is used to determine the first and second situations, but is not limited thereto, and the above examples for determining the first and second situations are applied. can

Also, although not shown in FIGS. 65 and 66, the information output in the first situation and the second situation may overlap each other at least partially. For example, the information output in the first situation and the second situation includes all biometrics, but is not limited thereto.

9.2. Various Embodiments of the Bioindex Measuring Device Arranged in a Vehicle The bioindex acquisition method and drowsiness detection method described above can be used in a variety of applications. For example, the bioindex acquisition method or drowsiness detection method can be used in a bioindex measuring device placed in a vehicle.

Also, the bioindex measuring device can perform an operation corresponding to the bioindex or biometric information obtained by the bioindex obtaining method.

Also, the biometric measuring device may perform an operation corresponding to the drowsiness information obtained by the drowsiness sensing method.

In addition, although the operation of the biometric measuring device will be described below, the described contents can be executed by other processors such as an ECU installed in a vehicle, and can also be executed by a processor included in a server.

In addition, the device described as the biometric measuring device may mean an image acquisition device, and in this case, the operation according to the described content may be executed by another processor such as an ECU or a processor included in a server. can.

FIG. 67 is a diagram for explaining a bioindex measuring device according to an embodiment.

According to FIG. 67, a biometric measuring device (8010) according to one embodiment may be placed in a vehicle (8000) to measure biometrics of a passenger (8020) on board said vehicle (8000). be able to.

At this time, the biometric measuring device (8010) can be arranged at various positions of the vehicle (8000).

For example, the biometric measuring device (8010) is arranged in an overhead console, sun visor, room mirror, dashboard, instrument panel, steering wheel, Saint Pecia, console box, etc. of the vehicle (8000). Not limited.

Also, the passenger (8020) may be a driver, but is not limited to this, and may be a person such as a passenger on board the vehicle.

In addition, when there are multiple passengers (8020), the biometric device (8010) can measure the biometrics of multiple people.

In addition, the biometric measuring device 8010 may include an image sensor and may acquire a biometric based on an image frame acquired from the image sensor, but is not limited thereto, such as a contact sensor. can be used.

In addition, when the biometric measuring device (8010) includes an image sensor, the image sensor may be a visible camera for obtaining a visible light image, an infrared camera for obtaining an infrared image, and an infrared camera for obtaining an infrared image. ), etc., but is not limited thereto, and can also be provided by a hybrid type camera for capturing visible and infrared images.

In addition, the biometric measuring device (8010) can operate by changing modes depending on the amount of external light. For example, it can operate in the first mode during the day when there is a lot of external light, and it can operate in the second mode during the night when there is little external light, but the present invention is not limited thereto.

In addition, for example, the biometric measuring device (8010) can operate in a first mode for obtaining a biometric based on an RGB image when the external light is above the reference value, and when the external light is below the reference value. In some cases, it can operate in a second mode of obtaining biometrics based on IR images, but is not limited thereto.

FIG. 68 is a diagram for explaining a bioindex measuring device according to an embodiment.

Referring to FIG. 68, a biometric measuring device according to one embodiment can be placed at various locations on a vehicle.

For example, as shown in FIG. 68, a first biometric device (8011) may be located on the driver's side dashboard and a second biometric device (8012) may be located on the passenger side dashboard. The third biometric measuring device (8013) may be placed in the rearview mirror, but is not limited thereto.

Also, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) may be installed inside the vehicle.

In addition, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can obtain the driver's bioindex.

For example, at least one of the first through third bioindex measuring devices (8011, 8012, 8013) measures the driver's bioindex such as heart rate, blood pressure, oxygen saturation, and body temperature. can be, but is not limited to.

In addition, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can perform an operation based on the obtained biometrics of the driver.

For example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) guides a nearby hospital when the acquired bioindex of the driver is abnormal. Such operations as, but not limited to, can be performed.

Further, for example, if at least one of the first to third bioindex measuring devices (8011, 8012, 8013) has an abnormality in the acquired biometric index of the driver, Information for this can be transmitted by the device, but is not limited thereto.

Further, for example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) may, if there is an abnormality in the acquired biometric index of the driver, A hardware alarm such as is operated, but is not limited to this.

Further, for example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) autonomously controls the vehicle when the acquired biometric index of the driver is abnormal. An operation for changing to the driving mode can be performed, but is not limited thereto.

In addition, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can acquire biometric information of the driver.

For example, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) acquires the driver's biometric information such as drowsiness information and condition information of the driver. can be, but is not limited to.

In addition, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can perform an operation based on the acquired biometric information of the driver.

For example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) may detect a hardware factor such as voice vibration when the acquired driver drowsiness information is abnormal. Activates a wear-like alarm, but is not limited to this.

In addition, for example, at least one of the first to third biometric index measuring devices (8011, 8012, 8013) is configured to, if the acquired drowsiness information of the driver is abnormal, to take a sleepy rest soon. , but is not limited thereto.

Further, for example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) detects that the acquired drowsiness information of the driver is abnormal, Information for this can be transmitted by the device, but is not limited thereto.

Further, for example, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) automatically controls the vehicle when the acquired drowsiness information of the driver is abnormal. An operation for changing to the driving mode can be performed, but is not limited thereto.

Further, for example, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) selects appropriate music based on the acquired condition information of the driver. Actions can be performed, but are not limited thereto.

In addition, for example, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) guides appropriate content based on the acquired condition information of the driver. Actions can be performed, but are not so limited.

In addition, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) can obtain a passenger's biometric index.

For example, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) measures the passenger's bioindex such as heart rate, blood pressure, oxygen saturation, and body temperature. can be, but is not limited to.

In addition, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can perform an operation based on the acquired biometrics of the passenger.

For example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) gives an alarm to the driver when the acquired biometric index of the passenger is abnormal. Such operations as, but not limited to, can be performed.

In addition, for example, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) guides the passenger to a nearby hospital when there is an abnormality in the acquired biometric index of the passenger. Operations such as, but not limited to, can be performed.

Also, for example, if at least one of the first to third bioindex measuring devices (8011, 8012, 8013) has an abnormality in the acquired bioindex of the passenger, other mobile device Information for this can be transmitted by, but is not limited to.

Further, for example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) causes the vehicle to autonomously travel when there is an abnormality in the acquired biometric index of the passenger. An action can be performed to cause the mode to change, but is not so limited.

Also, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can acquire biometric information of the passenger.

For example, at least one bioindex measuring device among the first to third bioindex measuring devices (8011, 8012, 8013) can acquire the passenger's biometric information such as drowsiness information and condition information of the passenger. but not limited thereto.

In addition, at least one of the first to third bioindex measuring devices (8011, 8012, 8013) can perform an operation based on the acquired passenger's biometric information.

For example, if at least one of the first to third biometric devices (8011, 8012, 8013) is abnormal in the acquired biometric information of the passenger, Actions such as announcing information may be performed, but are not so limited.

Further, for example, if at least one of the first to third biometric measuring devices (8011, 8012, 8013) has an abnormality in the acquired biometric information of the passenger, other mobile device can transmit information thereon, but is not limited thereto.

FIG. 69 is a diagram for explaining a biometric measuring device arranged in an autonomous vehicle according to one embodiment.

Referring to FIG. 69, a biometric index measuring device according to one embodiment can be placed at various locations on a vehicle.

For example, as shown in FIG. 69, a first biometric device (8014) may be placed on one side of the table and a second biometric device (8015) may be placed on the other side of the table. 3 The biometric measuring device (8016) is placed on the ceiling of the autonomous vehicle, but is not limited thereto.

Also, at least one of the first to third bioindex measuring devices (8014, 8015, 8016) may be installed inside the vehicle.

In addition, at least one of the first to third bioindex measuring devices (8014, 8015, 8016) can obtain a passenger's bioindex.

For example, at least one of the first to third bioindex measuring devices (8014, 8015, 8016) measures the passenger's bioindex such as heart rate, blood pressure, oxygen saturation, and body temperature. can be, but is not limited to.

Also, at least one of the first to third biometric measuring devices (8014, 8015, 8016) can perform an operation based on the acquired biometrics of the passenger.

For example, at least one of the first through third biometric measuring devices (8014, 8015, 8016) guides a nearby hospital to a nearby hospital when the acquired biometric index of the passenger is abnormal. Such operations as, but not limited to, can be performed.

In addition, for example, at least one of the first to third bioindex measuring devices (8014, 8015, 8016) detects an abnormality in the acquired biometric index of the passenger, and if there is an abnormality, the passenger is taken to a nearby hospital. Actions such as running can be performed, but are not limited to this.

Further, for example, if at least one of the first to third bioindex measuring devices (8014, 8015, 8016) has an abnormality in the acquired biometric index of the passenger, another mobile Information for this can be transmitted by the device, but is not limited thereto.

Further, for example, at least one of the first to third biometrics measuring devices (8014, 8015, 8016) may emit voice, vibrate, or , etc., but is not limited to this.

In addition, for example, if at least one of the first to third biometric measuring devices (8014, 8015, 8016) has an abnormality in the acquired biometric index of the passenger, information on this can be displayed, but is not limited to this.

In addition, at least one of the first to third biometric measuring devices (8014, 8015, 8016) can perform an operation according to the recognition result of the passenger.

For example, at least one of the first to third biometric measuring devices (8014, 8015, 8016) may recognize the face of the passenger, and if the face is recognized, the recognition information for the passenger identified, but is not limited to this.

Further, for example, at least one of the first to third biometric measuring devices (8014, 8015, 8016) is operated only when the passenger's face is recognized and a biometric is acquired. Information can be displayed for recognized passengers, but is not so limited.

Also, for example, at least one of the first to third biometric measuring devices (8014, 8015, 8016) can acquire the passenger's biometric information.

For example, at least one of the first to third bioindex measuring devices (8014, 8015, 8016) acquires the passenger's biometric information such as drowsiness information and condition information of the passenger. can be, but is not limited to.

In addition, at least one of the first to third biometric measuring devices (8014, 8015, 8016) can perform an operation based on the acquired biometric information of the passenger.

For example, at least one bioindex measuring device among the first to third bioindex measuring devices (8014, 8015, 8016) detects an abnormality in the acquired biometric information of the passenger, and the hardware such as voice, vibration, etc. A wear alarm can be activated, but is not limited to this.

Further, for example, at least one of the first to third biometric measuring devices (8014, 8015, 8016) detects an abnormality in the acquired biometric information of the passenger, Information for this can be transmitted by the device, but is not limited thereto.

In addition, although not shown in FIGS. 68 and 69, the bioindex measurement device can be installed in an ambulance or the like and used to measure the patient's bioindex to obtain biometric information.

9.2.1.Various Embodiments of Bio-Index Measuring Apparatus Performing Operation Based on Drowsiness Information FIG. 70 is a flow chart illustrating a method of operating a bio-index measuring apparatus according to one embodiment.

Referring to FIG. 70, a method of operating a biometric measuring device (8100) according to an embodiment may include acquiring a biometric index (S8110) and determining a drowsiness stage (S8120). An alarm can be output according to the drowsiness stage. For example, as shown in FIG. 70, a first alarm (S8130), a second alarm (S8140) and a third alarm (S8150) can be output, but not limited thereto.

At this time, the step of obtaining the biometric index (S8110) is applied to the above-described method of obtaining the biometric index, etc., so the repeated description will be omitted.

In addition, since the sleepiness sensing method described above is applied to the step of determining the sleepiness level (S8120), repeated description will be omitted.

Also, the first alarm (S8130) is an alarm output at the weakest drowsiness stage.

For example, the first alarm (S8130) includes, but is not limited to, a visual alarm.

Also, the second alarm (S8140) is an alarm output in an intermediate stage.

For example, the second alarm (S8140) may include, but is not limited to, an audible alarm with a low intensity or a long repetition period.

Also, the third alarm (S8150) is an alarm output at the highest drowsiness stage.

For example, the third alarm (S8150) may include, but is not limited to, an audible alarm with a high intensity or a short repetition period.

However, the first to third alarms (S8130, S8140, S8150) described above are not limited to the above examples, and it is obvious that examples of outputting different alarms depending on the drowsiness stage are all applicable. .

Also, the method of operating the biometric device (8100) according to one embodiment may include transmitting drowsiness information (S8160).

Also, the step of transmitting the drowsiness information (S8160) may be performed only at least part of the drowsiness step. For example, in the case of the first drowsiness level, the step of transmitting the drowsiness information (S8160) may not be performed, and in the case of the second drowsiness level and the third drowsiness level, the step of transmitting the drowsiness information (S8160) may be performed. but not limited thereto.

Also, transmitting the drowsiness information (S8160) may include transmitting the drowsiness information to an administrator. For example, information on the level of drowsiness of the vehicle driver may be transmitted to a manager who dispatches vehicles so that the information is reflected in the dispatch of vehicles.

Also, transmitting the drowsiness information (S8160) may include transmitting the drowsiness information to the mobile terminal of the driver. For example, drowsiness information and information about the travel section may be transmitted to the mobile terminal of the driver to remind the driver in the next trip.

Also, transmitting the drowsiness information (S8160) may include transmitting the drowsiness information to the manager again. For example, information on the drowsiness stage of the driver and information on the driving section may be transmitted and used to create drowsiness guidance.

9.2.2. Various Embodiments of Driving Scheduling Method Using Biometric Index and Biometric Information FIG. 71 is a flow chart illustrating a method of operating a biometric measuring device according to an embodiment.

Referring to FIG. 71, the operation method (8200) of the biometrics measuring device according to one embodiment includes acquiring biometrics (S8210), acquiring biometrics (S8220), and calculating a driving scheduling index. (S8230), but not limited thereto.

At this time, the step of obtaining the biometric index (S8210) is applied to the above-described method of obtaining the biometric index, etc., so the repeated description will be omitted.

In addition, since the biometric information acquisition method described above is applied to the step of acquiring biometric information (S8220), repeated description will be omitted.

Also, the driving scheduling index is an index for assisting driving scheduling.

At this time, the driving scheduling index can be obtained based on the biometric index and biometric information. For example, the driving scheduling index is obtained based on the driver's heart rate information, drowsiness information, etc., but is not limited thereto.

Also, the driving scheduling index may be obtained based on vehicle operation information. For example, the driving scheduling index is obtained based on the driving distance, driving time, driving frequency, etc. of the vehicle, but is not limited thereto.

Also, the driving scheduling index can be obtained based on a driver's biometric index, biometric information, and vehicle operation information. For example, the driving scheduling index may be obtained based on driver's drowsiness information, vehicle driving distance, driving frequency, etc., but is not limited thereto.

FIG. 72 is a diagram for explaining an operation method of a driving scheduling assistance device using a biometric device according to one embodiment.

Referring to FIG. 72, the driving scheduling auxiliary device can acquire information from a first biometric measuring device installed in a first vehicle and a second biometric measuring device installed in a second vehicle. can.

More specifically, the first biometric measuring device can acquire biometrics and biometric information for a first driver riding in the first vehicle. For example, the first biometric measuring device may acquire heart rate and drowsiness information for the first driver, but is not limited thereto.

Also, the second biometric index measuring device can acquire the biometric index and biometric information of a second driver riding in the second vehicle. For example, the second biometric measuring device may acquire heart rate and drowsiness information of the second driver, but is not limited thereto.

Further, as shown in FIG. 72, the driving scheduling assist device can acquire the biometrics and biometric information of the first driver acquired from the first biometric measuring device. 2. The biometric information of the second driver obtained from the biometric measuring device and the biometric information can be obtained, but is not limited thereto.

Also, as shown in FIG. 72, the driving scheduling auxiliary device can acquire first operation information for the first operation of the first vehicle and second operation information for the second operation of the second vehicle. .

At this time, the first and second operation information includes, but is not limited to, operation distance, operation frequency, operation time, and the like.

In addition, the driving scheduling assist device provides the biometric index and biometric information for the first driver, the biometric index and biometric information for the second driver, the first operation information for the first vehicle, and the biometric information for the second vehicle. 2 It is possible to calculate the driving schedule ring index based on the operation information.

For example, a first driving scheduling index may be calculated based on the first drowsiness information for the first driver and the first operation information for the first vehicle, and the second drowsiness information for the second driver and the The second driving scheduling index may be calculated based on the second operation information for the second vehicle, but is not limited thereto.

Further, for example, when the first travel distance included in the first travel information and the second travel distance included in the second travel information are the same, among the first driver and the second driver, drowsiness According to the information, the driving scheduling index can be calculated so that the driver with the lower drowsiness index performs the third service.

Further, for example, the first driving scheduling index may be calculated by adding different weights to the first drowsiness information and the first operation information for calculating the first driving scheduling index. can.

More specifically, more weights are added to the first operation information to calculate the first driving scheduling index, but the invention is not limited thereto.

Further, for example, the second driving scheduling index is calculated by further adding different weighting values to the second drowsiness information and the second operation information for calculating the second driving scheduling index. can be done.

More specifically, more weights are added to the second operation information to calculate the second driving scheduling index, but the invention is not limited thereto.

In addition, the driving scheduling auxiliary device may perform driving scheduling for the third service using the calculated driving scheduling index, but is not limited thereto.

Also, the driving scheduling assist device may display the calculated driving scheduling index, but is not limited thereto.

9.2.3.Various Embodiments of Bio-Index Measuring Apparatus Used to Release Sleeping Money of Vehicle FIG. 73 is a flow chart for explaining a method of operating a bio-index measuring apparatus according to one embodiment.

Referring to FIG. 73, a method of operating a biometric measuring device (8300) according to an embodiment includes recognizing a face of a subject (S8310), obtaining a biometric of the subject (S8320), and at least part of the step of unlocking at least part of the operation of the vehicle (S8330), but is not limited thereto.

At this time, the step of recognizing the face of the person to be measured (S8310) applies not only the above contents but also the usual face recognition method, so the repeated description will be omitted.

In addition, the step of obtaining the biometric index of the person to be measured (S8320) applies the above-described biometric index obtaining method, etc., and thus repeated description will be omitted.

Also, the step of unlocking at least a part of the operation of the vehicle (S8330) may be performed when the subject's face is recognized and the biometric index is obtained.

For example, if the subject's face is recognized but no biometric index is obtained, the step of unlocking at least a portion of the operation of the vehicle (S8330) may not be performed, although there are limitations to this. not.

Also, in this case, it is possible to prevent unlocking by stealing a photograph or the like, thereby further enhancing the security of the vehicle.

Also, in the step of unlocking at least a part of the operation of the vehicle (S8330), if the recognized face of the subject matches an ID stored in a server and a biometric index is obtained, can be performed.

For example, if the subject's face is recognized and the biometric index is obtained, but the recognized subject's face does not match the ID stored in the server, at least part of the vehicle's operation (S8330) may not be performed, but is not limited to this.

Also, the step of unlocking at least a portion of the vehicle during operation (S8330) is performed when the subject's face is recognized and the biometric index matches an ID stored in a server. can be done.

For example, when the subject's face is recognized and a biometric index is obtained, but the obtained biometric index of the subject does not match the ID stored in the server, at least The step of partially unlocking (S8330) may not be performed, but is not limited thereto.

Also, the locking of at least a part of the operation of the vehicle may be a locking of the starting of the vehicle, but is not limited thereto.

9.2.4. Various Embodiments of Operating Method of Driving Index Calculating Apparatus Using Biometric Measuring Device FIG. 74 is a flow chart illustrating an operating method of a driving index calculating apparatus according to one embodiment.

Referring to FIG. 74, the operation method (8350) of the driving index calculation device according to one embodiment includes obtaining driving information (S8360), obtaining drowsiness information for the subject (S8370), and At least part of the step of calculating a driving index for a person (S8380) is included, but not limited thereto.

At this time, the driving index may be a comprehensive evaluation index of the driving of the driver, or may be a numerical value calculated by comprehensively considering various information, but is not limited thereto.

Also, at this time, the operation information includes, but is not limited to, operation distance information, operation frequency information, operation time information, operation section information, road information, etc. for the vehicle in operation.

In addition, since the sleepiness sensing method described above is applied to the step of obtaining sleepiness information for the subject (S8370), repeated description will be omitted.

Also, calculating the driving index for the subject (S8380) may include calculating the driving index based on the operation information and the drowsiness information.

For example, the driving index may be calculated by comprehensively considering the number of times sleepiness occurs during the operation time, the degree of sleepiness, and countermeasures against the sleepiness, but is not limited thereto.

In addition, compensation can be provided to the driver based on the calculated driving index. For example, compensation is given to the driver based on the driving index calculated based on the driving time information of the driver who has driven for a long time and the drowsiness information that does not occur, but is not limited thereto.

In addition to the above examples, the driving index for the person to be measured can be calculated by various methods based on the driving time and sleepiness information.

9.3. Various Embodiments of Bioindex Measuring Devices Installed in Infant Monitoring Devices Monitoring infants using image sensors such as cameras is becoming essential for the health and safety of infants. there is

In addition, monitoring the health and safety of infants by measuring their biometric index is becoming a necessity.

However, installing a separate sensor or a separate contact-type sensor to monitor the biometrics of the infant causes inconvenience to the infant and does not serve as a resting place for comfort and comfort. can be restricted.

Therefore, using image sensors such as cameras to monitor infants at the same time as monitoring infant biometrics by using image sensors such as cameras will protect the health and safety of infants and at the same time protect the health of infants. It can provide a peaceful and comfortable resting place.

FIG. 75 is a diagram illustrating an infant monitoring device according to one embodiment.

Referring to FIG. 75, an infant monitoring device (8400) according to one embodiment can contactlessly acquire a biometric index of an infant (8410).

More specifically, the infant monitoring device (8400) may include an image sensor, and may obtain biometrics based on image frames obtained from the image sensor, but is not limited thereto.

In addition, the infant monitoring device 8400 acquires the biometric index based on the image frame, since the above-mentioned contents are applied, so the repeated explanation will be omitted.

In addition, the infant monitoring device 8400 can store and transmit images acquired through the image sensor.

For example, the infant monitoring device (8400) can acquire an image including at least part of the infant (8410) to be measured, and store or transmit the acquired image to the user's mobile. can be done.

The infant monitoring device (8400) can also sense movement of the infant (8410). For example, the infant monitoring device (8400) can sense, but is not limited to, turning over of the infant (8410).

Also, the infant monitoring device (8400) can sense voices emitted from the infant (8410). For example, the infant monitoring device (8400) can sense the infant's (8410) crying, but is not limited thereto.

Also, the infant monitoring device (8400) can generate a voice. For example, when there is a voice input from an input device, the infant monitoring device (8400) may output a voice corresponding to the input, more specifically, the voice of the parents of the infant (8410) is input. If so, it can be output, but is not limited to this.

The infant monitoring device (8400) can also output lighting. For example, if there is an input for lighting from an input device, the infant monitoring device 8400 may output lighting corresponding to the input, but is not limited thereto.

The infant monitoring device (8400) can also control the movement of the cradle in which the infant (8410) resides. For example, if there is an input to the cradle from an input device, the infant monitoring device 8400 can perform a cradle action corresponding to the input, but is not limited thereto.

Also, the infant monitoring device (8400) can output an alarm. For example, when an event occurs in the infant (8410), an audible alarm can be output, but not limited to, a visual and tactile alarm can be output, and the user's mobile terminal You can also transmit information to

In addition, the infant monitoring device 8400 can perform various operations for monitoring the infant 8410 in addition to the above examples.

9.3.1. Various Embodiments of Infant Monitoring Apparatus Performing Action upon Occurrence of an Event FIG. 76 is a diagram for explaining the occurrence of an event for an infant.

Referring to FIG. 76, it can be seen that a variety of events can occur for infants.

Referring to FIG. 76(a), it can be seen that there is a possibility that an infant will turn over.

Also, when the infant rolls over, the passage for breathing is blocked, which poses a great danger to the infant.

Further, referring to (b) of FIG. 76, it can be seen that there is a possibility that an event may occur in which the infant moves out of the monitoring area.

In this case, infants were sometimes moved by nurses at postnatal nursing homes, but there were also cases where infants were unintentionally moved by suspicious persons.

Also, although not shown in FIG. 76, a variety of events can occur that endanger the health and safety of the infant.

Therefore, there may be a need for an infant monitoring device that can monitor and track such events.

FIG. 77 is a flow chart illustrating a method of operating an infant monitoring device according to one embodiment.

Referring to FIG. 77, a method of operating an infant monitoring device (8500) according to one embodiment includes obtaining a biometric index (S8510), determining occurrence of an event (S8520), and performing an action corresponding to the event. (S8530), but not limited thereto.

At this time, the step of obtaining the biometric index (S8510) applies the above-described method of obtaining biometric index, so a repeated description will be omitted.

According to one embodiment, the step of determining occurrence of the event (S8520) may be determined based on the obtained biometric index.

For example, if the biometric index is not obtained through the step of obtaining the biometric index (S8510), it may be determined that an event has occurred. As a more specific example, if the infant subject to biometric acquisition turns over, the biometric is not acquired, and this can be determined as an event occurrence, but is not limited thereto.

In addition, for example, if there is an abnormality in the biometrics obtained through the step of obtaining the biometrics (S8510), it may be determined that an event has occurred. As a more specific example, if there is an abnormality in the heart rate of an infant whose biometric index is to be acquired, it can be determined as an event occurrence, but the present invention is not limited to this.

Also, according to one embodiment, the step of determining occurrence of the event (S8520) may be determined based on a plurality of biometric indices.

For example, if at least one bioindex is not obtained through the step of obtaining bioindex (S8510), it may be determined that a first event has occurred, and if all bioindexes are not obtained, it may be determined that a second event has occurred. but not limited thereto.

In addition, for example, if at least one bioindex obtained through the step of obtaining bioindex (S8510) is abnormal, it may be determined that the first event has occurred. It can be determined as the occurrence of the second event, but is not limited thereto.

Also, according to one embodiment, the step of determining occurrence of the event (S8520) may be determined based on a change in biometric index.

For example, when a sudden change is detected in the biometrics obtained through the step of obtaining the biometrics (S8510), it may be determined that an event has occurred. When an infant changes from a sleeping state to an awake state, the infant's heart rate may increase, which can be determined as an event occurrence, but is not limited thereto.

Also, according to one embodiment, performing an action corresponding to the event (S8530) may include performing an action corresponding to the determined event information.

At this time, the action corresponding to the event may include hardware alarms such as visual, auditory, and tactile alarms. For example, but not by way of limitation, the infant monitoring device may output an audible voice alarm if it is determined that an event has occurred.

Also, the action corresponding to the event may include an action to record an image. For example, the infant monitoring device can operate to record an image when it is determined that an event has occurred, but is not so limited.

Also, the action corresponding to the event may include an action of transmitting information about the event. For example, if the infant monitoring device determines that an event has occurred, the infant monitoring device can transmit information about the event to the user's mobile terminal, but is not limited thereto.

Also, the operation corresponding to the event may include an operation of storing information about the time when the event occurred. For example, the infant monitoring device can store, but is not limited to, information about the time of the event when it is determined that the event has occurred.

In addition, the action corresponding to the event may include various actions corresponding to the generated event in addition to the above examples.

FIG. 78 is a flow chart illustrating a method of operating an infant monitoring device according to one embodiment.

Referring to FIG. 78, a method of operating an infant monitoring device (8550) according to one embodiment includes obtaining a biometric index (S8560), determining the occurrence of an event (S8570), Including, but not limited to, at least some of performing an action (S8580) and performing a second action corresponding to a second event (S8590).

At this time, the step of obtaining the biometric index (S8560) applies the above-described method of obtaining biometric index, so a repeated description will be omitted.

According to one embodiment, the step of determining occurrence of the event (S8570) may be determined based on the obtained biometric index.

For example, if the biometric index is not obtained through the step of obtaining the biometric index (S8560), it may be determined that an event has occurred. As a more specific example, if the infant subject to biometric acquisition turns over, the biometric is not acquired, and this can be determined as an event occurrence, but is not limited thereto.

Also, for example, if there is an abnormality in the biometrics obtained through the step of obtaining the biometrics (S8560), it may be determined that an event has occurred. As a more specific example, if there is an abnormality in the heart rate of an infant whose biometric index is to be acquired, it can be determined as an event occurrence, but the present invention is not limited to this.

Also, according to one embodiment, the step of determining the event occurrence (S8570) may be determined based on a plurality of biometric indices.

For example, if at least one bioindex is not obtained through the step of obtaining the bioindex (S8560), it can be determined that a first event has occurred, and if none of the bioindexes are obtained, it can be determined that a second event has occurred. is not limited to

In addition, for example, when at least one bioindex obtained through the step of obtaining bioindex (S8560) is abnormal, it can be determined that the first event occurs, and when all bioindexes are abnormal, the second event can be determined. It can be determined as an occurrence, but is not limited thereto.

Also, according to one embodiment, the step of determining occurrence of the event (S8570) may be determined based on a change in biometric index.

For example, when a sudden change is detected in the biometrics acquired through the step of acquiring biometrics (S8560), it can be determined that an event has occurred. When the sleeping state changes to the weather state, the infant's heart rate increases, which can be determined as an event occurrence, but is not limited thereto.

Also, according to one embodiment, performing the first action corresponding to the first event (S8580) and performing the second action corresponding to the second event (S8590) are performed using the determined event information. may include performing an operation corresponding to .

At this time, the first operation and the second operation may include hardware alarms such as visual, auditory, and tactile alarms. For example, but not by way of limitation, the infant monitoring device may output an audible voice alarm if it is determined that an event has occurred.

Also, the first operation and the second operation may include an operation for recording an image. For example, the infant monitoring device can operate to record an image when it is determined that an event has occurred, but is not so limited.

Also, the first action and the second action may include an action of transmitting information about an event. For example, if the infant monitoring device determines that an event has occurred, the infant monitoring device can transmit information about the event to the user's mobile terminal, but is not limited thereto.

Also, the first operation and the second operation may include an operation of storing information on event occurrence time. For example, the infant monitoring device can store, but is not limited to, information about the time of the event when it is determined that the event has occurred.

Also, the first operation and the second operation may be different from each other, but are not limited thereto, and may be the same as each other, or may perform at least part of the same operation as each other.

Also, the first action and the second action may include various actions corresponding to the first event and the second event, other than the examples described above.

9.3.2. Various Embodiments of Infant Monitoring Systems FIG. 79 is a diagram illustrating a mobile application for implementing an infant monitoring system according to one embodiment.

Referring to FIG. 79, a mobile application (8600) according to one embodiment includes an image display area (8610), a biometric index display area (8620), an input button display area (8630), an image time display area (8640), and an event display area (8640). Including, but not limited to, at least a portion of the time-recorded video display area (8650).

At this time, the image displayed in the image display area 8610 may be an image or an image acquired from the infant monitoring device, but is not limited thereto.

Also, the biometrics displayed in the biometrics display area (8620) includes at least one biometric. For example, as shown in FIG. 79, the biometric index includes, but is not limited to, heart rate, oxygen saturation, blood pressure, and the like.

Also, the biometric index displayed in the biometric index display area 8620 may be the biometric index obtained from the infant monitoring device, but is not limited thereto.

Also, the biometrics displayed in the biometrics display area 8620 may be biometrics obtained from at least one sensor, but is not limited thereto.

Also, the biometrics displayed in the biometrics display area (8620) may be biometrics obtained from different sensors. For example, the heart rate displayed in the biometric index display area (8620) may be a biometric index obtained from an infant monitoring device, and the blood pressure may be a biometric index obtained from an external blood pressure sensor. , but not limited to.

Also, the input buttons displayed in the input button display area 8630 may include at least one or more input buttons.

Also, the input button includes at least some of various input buttons such as a video recording button, a lighting button, a talking button, a cradle shaking button, and an alarm button, but is not limited thereto.

Also, the video time displayed in the video time display area (8640) is time information for the recorded video.

Also, the event occurrence time can be displayed in the video time displayed in the video time display area 8640 . For example, as shown in FIG. 79, the video time displays the first event occurrence time point and the second event occurrence time point, but is not limited thereto.

In addition, thumbnails of videos recorded at the event time can be displayed in the video display area 8650 .

In addition, the information of the video recorded at the event time can be displayed in the video display area 8650 . For example, as shown in FIG. 79, time information of the recorded video is displayed, but the present invention is not limited thereto.

However, the above examples and drawings only show one embodiment and are not limited thereto, and the application for implementing the infant monitoring system can be provided in various forms.

9.4. Various Embodiments of Bioindex Measurement Devices Installed in the Library Measuring bioindex and biometric information in a space for personal work and study, such as a library, is a continuous monitoring of personal health. , can monitor a person's concentration to enable efficient work or study.

Also, such concentration can be obtained based on heart rate. For example, concentration information is acquired based on changes in heart rate, magnitude, change pattern, etc., but is not limited thereto.

FIG. 80 is a diagram for explaining a bioindex measuring device arranged in a library according to one embodiment.

Referring to FIG. 80, a biometric measuring device (8700) according to one embodiment can acquire biometrics and biometric information of a subject (8710).

At this time, the biometric index includes, but is not limited to, biometric indices such as heart rate, oxygen saturation, blood pressure, and body temperature.

Also, the biometric information includes, but is not limited to, biometric information such as drowsiness information, condition information, and concentration information.

In addition, since the above-described method is applied to the method for obtaining the biometric index and the biometric information, repeated explanation will be omitted.

FIG. 81 is a flow chart for explaining the operation method of the biometric measuring device according to one embodiment.

Referring to FIG. 81, the operating method (8720) of the biometric measuring device according to one embodiment includes obtaining bioindex (S8730), obtaining biometric information (S8740), and operations corresponding to the biometric information. (S8750).

At this time, the step of obtaining the biometric index (S8730) and the step of obtaining the biometric information (S8740) are the same as described above, so repeated description will be omitted.

Also, the step of performing an action corresponding to the biometric information (S8750) may include performing various actions corresponding to various biometric information.

For example, if the biometric information is drowsiness information, the biometric measuring device may output an alarm according to the drowsiness stage, but the embodiment is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the biometric measuring device may perform an operation of transmitting information to an administrator according to the drowsiness stage, but the embodiment is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the biometric measuring device may perform an operation of transmitting information to a user's terminal according to the drowsiness stage, but the embodiment is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the biometric device may perform an operation of transmitting information to the scheduling device according to the drowsiness stage, but the embodiment is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the biometric measuring device may output a rest recommendation message according to the drowsiness stage, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric measuring device can execute an operation of outputting an alarm depending on the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric measuring apparatus may perform an operation of transmitting information to the administrator according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric measuring apparatus may perform an operation of transmitting information to the user's terminal according to the condition, but is not limited thereto.

Also, for example, if the biometric information is condition information, the biometric device may perform an operation of transmitting information to the scheduling device according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric measuring device can execute an operation of outputting a rest recommendation message depending on the condition, but is not limited thereto.

Also, for example, when the biometric information is concentration level information, the biometric measuring apparatus may perform an operation of outputting an alarm according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration level information, the biometric measuring apparatus may perform an operation of transmitting information to an administrator according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration level information, the biometric measuring apparatus may perform an operation of transmitting information to a user's terminal according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration information, the biometric device may perform an operation of transmitting information to the scheduling device according to the concentration, but is not limited thereto.

In addition, the biometric measuring device can perform operations corresponding to biometric information and biometrics.

In addition, the biometric measuring apparatus can perform various operations corresponding to biometric information and biometrics in addition to the above examples.

FIG. 82 is a flow chart for explaining the operation method of the biometric measuring device according to one embodiment.

Referring to FIG. 82, an operating method (8760) of a biometric measuring device according to an embodiment includes obtaining a biometric index (S8770), obtaining biometric information (S8780), and performing a study or work scheduling index. At least part of the step of calculating (S8790) can be included.

At this time, the step of obtaining the biometric index (S8770) and the step of obtaining the biometric information (S8780) are the same as described above, so repeated descriptions will be omitted.

Also, the step of calculating the study or work scheduling index (S8790) may be performed based on the acquired biometric index and/or biometric information.

For example, a study or work scheduling index is calculated based on the acquired heart rate and drowsiness information, but is not limited thereto.

In addition, the step of calculating a study or work scheduling index (S8790) may be performed based on the acquired biometric index, biometric information and/or study-related information.

For example, a study or work scheduling index is calculated based on the acquired sleepiness information and study time information, but the invention is not limited thereto.

In addition, by using the study or work scheduling index, the study or work can be scheduled to be performed in a time period with high efficiency.

In addition, scheduling can be performed so that a break can be taken in a time period of low efficiency using the study or work scheduling index.

Therefore, when utilizing the study or work scheduling index, it is possible to perform the study or work with maximum efficiency within a limited time.

9.5. Various Examples of Bioindex Measuring Device Used for Cognitive Rehabilitation Treatment In the case of patients undergoing cognitive rehabilitation treatment, such as dementia patients, if highly intensive treatment is performed during cognitive rehabilitation treatment, the therapeutic effect can be improved. can be maximized.

However, the supply and demand of rehabilitation therapists to monitor the concentration of each patient does not meet the demand, and it is difficult to improve the concentration of each patient and perform high-quality treatment.

Therefore, in order to supplement this, if the bioindex and biometric information are obtained using a bioindex measuring device, it is possible to monitor the degree of concentration of each patient, and use this to perform high-quality treatment. can be done.

FIG. 83 is a diagram for explaining a bioindex measuring device used for cognitive rehabilitation treatment according to one embodiment.

Referring to FIG. 83, the biometric device according to one embodiment includes a first biometric device (8800), a second biometric device (8801), a third biometric device (8802), a fourth biometric device At least one bioindex measuring device can be included among the index measuring device (8803) and the fifth biometric measuring device (8804).

At least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814). At least one of them is able to measure a biometric index.

At this time, since the above-described contents are applied to the biometric index measurement method, repeated description will be omitted.

At least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814). Biometric information of at least one of them can be obtained. For example, at least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814) Concentration information of at least one of them can be obtained, but is not limited thereto.

At least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814). If there is an abnormality in the biometric information of at least one of them, an alarm can be output so that the rehabilitation therapist (8810) can recognize it. For example, at least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814) If there is an abnormality in the biometric information of at least one of them, an audible alarm can be output, and information related to the biometric information can be transmitted to the mobile terminal of the rehabilitation therapist (8810). is not limited to

At least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814). If there is an abnormality in the biometric information of at least one of them, an alarm can be output so that the patient with the abnormal biometric information can recognize this. For example, at least one bioindex measuring device among the first to fifth bioindex measuring devices (8800, 8801, 8802, 8803, 8804) is used for the first to fourth patients (8811, 8812, 8813, 8814) If there is an abnormality in the biometric information of at least one of them, an audible alarm can be output, and information related to the biometric information can be displayed through the corresponding patient's display.

Also, as described above, real-time monitoring of the patient's biometrics and biometrics to help maintain concentration during treatment is a way to maximize the effectiveness of cognitive rehabilitation treatment.

FIG. 84 is a flow chart for explaining the operation method of the biometric measuring device according to one embodiment.

Referring to FIG. 84, an operation method (8850) of a biometric measuring device according to an embodiment includes acquiring a biometric index of a patient (S8860), acquiring biometric information of the patient (S8870), and At least part of the step of performing an operation corresponding to the biometric information (S8880) can be included.

At this time, the step of obtaining the patient's biometric index (S8860) and the step of obtaining the patient's biometric information (S8870) are the same as described above, so repeated descriptions will be omitted.

In addition, the step of performing an operation corresponding to biometric information of the patient (S8880) may include performing various operations corresponding to various biometric information.

For example, if the biometric information is concentration information, the biometric device may output an alarm according to the concentration, but is not limited thereto.

Also, for example, when the biometric information is concentration level information, the biometric measuring apparatus may perform an operation of transmitting information displayed on the patient's display according to the concentration level. not.

In addition, for example, when the biometric information is concentration level information, the biometric measuring device may perform an operation of transmitting information to a rehabilitation therapist's mobile terminal according to the concentration level, but is not limited thereto. .

Also, for example, when the biometric information is concentration level information, the biometric measuring device can perform an operation of outputting a display for increasing the concentration level according to the concentration level, but is not limited thereto.

In addition, the biometric measuring device can perform operations corresponding to biometric information and biometrics.

In addition, the biometric measuring apparatus can perform various operations corresponding to biometric information and biometrics in addition to the above examples.

9.6.Various Examples of Bioindex Measuring Devices Used for Immigration Control Epidemic control at airports where people from various countries and ethnic groups come and go can play an important role in the national quarantine system. .

However, there is a shortage of management personnel compared to the number of people going in and out of the airport, and it is difficult to closely manage each person.

Therefore, if the biometric index is easily measured at a place where each person undergoes immigration inspection, there is a possibility that close management for each person will become easier.

FIG. 85 is a diagram for explaining a biometric measuring device used for immigration inspection according to one embodiment.

Referring to FIG. 85, a biometric measuring device (8901, 8902) according to one embodiment can be placed in an immigration kiosk (8900) to measure biometrics of immigrants (8911, 8912).

At this time, the biometric measuring device (8901, 8902) can perform face recognition used for immigration, but is not limited thereto.

Also, the biometric index includes at least one of biometric indices such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, since the above-mentioned contents are applied to the method of measuring the bioindex by the bioindex measuring device (8901, 8902), the repeated explanation will be omitted.

Also, the bioindex measuring devices (8901, 8902) can perform operations corresponding to the measured bioindex. For example, as shown in FIG. 85, if the measured body temperature of the first immigration person (8911) is within the normal range, the first biometric measuring device (8901) displays first information (8921) indicating immigration enhancement. and if the measured body temperature of the second immigrant (8912) is outside the normal range, the second biometric device (8902) requests a more accurate body temperature measurement. The second information (8922) can be displayed, but is not limited to this.

In addition, as mentioned above, monitoring the biometrics of immigrants at the time of immigration inspection and strengthening quarantine is a way to maximize the effectiveness of national quarantine.

FIG. 86 is a flow chart for explaining the operation method of the biometric measuring device according to one embodiment.

Referring to FIG. 86, an operating method (8950) of a biometric measuring device according to an embodiment includes steps of obtaining a biometric of an immigrant (S8960) and performing an operation corresponding to the biometric (S8970). can include at least a portion of

At this time, since the above-described contents are applied to the step of obtaining the biometric index of the immigrant (S8960), repeated description will be omitted.

Also, the step of performing operations corresponding to the biometrics (S8970) may include performing various operations corresponding to various biometrics.

For example, if the biometric index is the heart rate and the heart rate is within a normal range, the biometric index measuring device may perform a corresponding operation, but is not limited thereto.

In addition, for example, when the biometric index is the heart rate and the heart rate is out of the normal range, the bioindex measuring device can perform a corresponding operation, but is not limited thereto. .

Also, for example, when the bioindex is the oxygen saturation and the oxygen saturation is within a normal range, the bioindex measuring device may perform a corresponding operation, but is not limited thereto. .

Further, for example, when the bioindex is the oxygen saturation and the oxygen saturation is outside the normal range, the bioindex measuring device can perform a corresponding operation. Not limited.

Also, for example, when the bioindex is blood pressure and the blood pressure is within a normal range, the bioindex measuring device may perform a corresponding operation, but is not limited thereto.

Also, for example, when the bioindex is blood pressure and the blood pressure is out of the normal range, the bioindex measuring device may perform a corresponding operation, but is not limited thereto.

Also, for example, if the biometric index is body temperature and the body temperature is within a normal range, the biometric device may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, if the bioindex is body temperature and the body temperature is out of the normal range, the biometric measuring device may perform a corresponding operation, but is not limited thereto.

Also, for example, when the biometric index is within the normal range, the corresponding operation includes, but is not limited to, displaying information permitting entry and exit.

In addition, for example, when the biometric index is out of the normal range, the corresponding operation includes, but is not limited to, disallowing entry and exit or displaying information indicating the detailed examination subject. .

In addition, the biometric measuring apparatus can perform various operations corresponding to biometrics in addition to the above examples.

9.7. Various Embodiments of Biometric Measuring Devices Used in Security Devices Security devices through biometric recognition, such as face recognition, iris recognition, and fingerprint recognition, are becoming active.

However, there is an increasing number of cases where security devices are incapacitated through photographs, fingerprint copies, and the like.

Therefore, when a security device utilizing biometrics and biometrics is provided, it is possible to prevent the security device from being disabled, thereby making the security device stronger.

FIG. 87 is a diagram for explaining a biometric measuring device used in a security device according to one embodiment.

Referring to FIG. 87, a biometric measuring device (9010) according to one embodiment can be installed in a security device (9000) to measure a biometric of a subject (9020).

More specifically, the security device 9000 can perform biometric recognition such as face recognition, fingerprint recognition, and iris recognition of the person to be measured 9020, but is not limited thereto.

In addition, the biometric measuring device (9010) can acquire the biometric of the object to be measured (9020), but is not limited thereto.

At this time, the biometric index includes at least one of biometric indices such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, since the above-mentioned contents are applied to the method of measuring the biometrics by the biometrics measuring device (9010), the repeated description will be omitted.

In addition, the security device 9000 can release security based on biometric recognition and measured biometrics. For example, after first confirming the existence of the designated user through iris recognition, it is possible to release the security by secondarily determining whether the user is a person through the acquisition of the biometric index. , but not limited to.

In addition, the security device 9000 can release security based on multiple biometrics and multiple biometrics. For example, after first confirming the existence of the designated user through iris recognition and fingerprint recognition, it secondarily determines whether it is a person through the acquisition of heart rate and oxygen saturation. Security can be turned off, but is not so limited.

Also, the security device (9000) can release security based on biometrics. For example, if the acquired heartbeat-related signal matches the heartbeat-related signal of the designated user, it can be determined as the designated user and the security can be released. Not limited.

In addition, as described above, strengthening security by additionally using a biometric index measuring device as a security device is a method of maximizing the effect of strengthening security.

FIG. 88 is a flow chart illustrating a method of operation of a security device according to one embodiment.

Referring to FIG. 88, a method of operating a security device (9050) according to an embodiment includes the step of determining whether biometric recognition matches (S9060), the step of determining whether biometric index acquisition is possible (S9070), and the operation according to the result. (S9080).

At this time, the biometric recognition may include at least one of biometric recognition such as fingerprint recognition, face recognition, and iris recognition.

Also, the biometric index may include at least one biometric index such as heart rate, oxygen saturation, blood pressure, and body temperature.

In addition, the step of determining whether the biometrics match (S9060) may be performed using a conventional biometrics recognition method, but is not limited thereto.

In addition, since the method for obtaining the biometric index for determining whether or not to obtain the biometric index applies to the method described above, repeated description will be omitted.

Also, the step of determining whether to obtain the biometric index (S9070) may include determining whether to obtain the biometric index. For example, the step of determining whether to acquire the biometric index (S9070) includes, but is not limited to, determining whether the heart rate is being acquired.

In addition, the step of determining whether to obtain the biometric index (S9070) may include determining whether biometric signals for obtaining the biometric index match. For example, the step of determining whether to obtain the biometric index (S9070) includes determining whether a heartbeat signal for obtaining the heartbeat rate matches a heartbeat signal of a specified user. is not limited to

In addition, the step of determining whether to obtain the biometric index (S9070) may include determining whether or not the biometric index matches. For example, the step of determining whether the biometric index can be acquired (S9070) includes determining whether the heart rate of the subject measured in advance and the currently measured heart rate are within a reference range based on the heart rate. including, but not limited to, determining the

Further, the step of performing the operation according to the result (S9080) includes the step of releasing the security if all the security release conditions are satisfied, and the step of maintaining the security if the security release conditions are not satisfied. is not limited to

In addition, the security device can perform various operations for strengthening security using biometrics, in addition to the above examples.

9.8. Various Embodiments of Bioindex Measuring Device Used in Kiosk When a bioindex measuring device that can easily measure bioindex is placed in a kiosk for displaying information, it can be easily used in daily life. can be monitored to more sustainably monitor the health of the individual.

FIG. 89 is a diagram for explaining a biometric measuring device used in a kiosk according to one embodiment.

Referring to FIG. 89, a biometric measuring device (9110) according to one embodiment can be placed in a kiosk (9100) to measure a biometric of a subject (9120).

At this time, the kiosk 9100 may include at least one display for displaying information. For example, as shown in FIG. 89, the kiosk 9100 can display date information and weather information, but is not limited thereto.

In addition, the biometric measuring device (9110) can obtain at least one biometric. For example, the biometric measuring device 9110 may acquire heart rate, but is not limited thereto, and acquires at least one biometric among biometric indices such as heart rate, oxygen saturation, blood pressure, and body temperature. can do.

In addition, the biometric measuring device (9110) can acquire at least one biometric information. For example, the biometric measuring device 9110 may acquire today's condition information, but is not limited thereto, and at least one of biometric information such as drowsiness information, condition information, concentration information, and health information. Biological information can be acquired.

Also, the kiosk 9100 can display the acquired biometric index. For example, as shown in FIG. 89, the kiosk (9100) can display the acquired heart rate, oxygen saturation and blood pressure, but is not so limited.

Also, the kiosk (9100) can display the acquired biometric information. For example, as shown in FIG. 89, the kiosk (9100) can display the obtained today's condition information, but is not limited thereto.

In addition, the kiosk 9100 can transmit the acquired biometric index and biometric information. For example, the kiosk 9100 can transmit the acquired biometric index and biometric information to the mobile terminal of the subject 9120, but is not limited thereto.

In addition, the kiosk 9100 can transmit the acquired biometric index and biometric information. For example, the kiosk 9100 may transmit the acquired biometric index and biometric information to a terminal where an administrator can check, but is not limited thereto.

In addition, in this case, it is possible to detect patients with bioindex abnormalities in the building where the kiosk 9100 is installed, and to strengthen the epidemic prevention and security of the building.

In addition, as mentioned above, adding a bioindex measuring device to the kiosk can continuously monitor the health of each person and strengthen the epidemic prevention and security of the building where the kiosk is installed. be able to.

In addition, the above-mentioned kiosk device is used for personnel management, for example, it is used as a kiosk for entering and exiting construction site workers, and is used to give measures to construction site workers who have abnormal bioindex. but not limited thereto.

FIG. 90 is a flowchart for explaining a method of operating a biometric measuring device according to one embodiment.

Referring to FIG. 90, an operation method (9150) of a biometric measuring device according to an embodiment includes acquiring a biometric index (S9160), acquiring biometric information (S9170), and measuring the biometric index and biometric information. At least part of the step of outputting (S9180) may be included.

At this time, the step of obtaining the biometric index (S9160) applies the above-described method of obtaining biometric index, so a repeated description will be omitted.

In addition, since the biometric information acquisition method described above is applied to the step of acquiring the biometric information (S9170), repeated description will be omitted.

Also, the step of outputting the biometric index and biometric information (S9180) may include displaying the biometric index and biometric information. For example, outputting the biometric index and biometric information (S9180) may include, but is not limited to, displaying heart rate, oxygen saturation, blood pressure, and today's condition information.

Also, the step of outputting the biometric index and biometric information (S9180) may include transmitting the biometric index and biometric information. For example, outputting the biometric index and biometric information (S9180) may include, but is not limited to, transmitting heart rate, oxygen saturation, blood pressure, and today's condition information to other mobile terminals.

Also, the step of outputting the biometric index and biometric information (S9180) may include printing the biometric index and biometric information. For example, the step of outputting the biometric index and biometric information (S9180) includes, but is not limited to, printing heart rate, oxygen saturation, blood pressure and today's condition information.

In addition, the biometric acquisition device can perform various operations of acquiring and outputting biometrics and biometric information in addition to the above-described examples.

A method according to an embodiment can be embodied in a program instruction form that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable media may include program instructions, data files, data structures, etc. singly or in combination. The program instructions recorded on the medium may be those specifically designed and constructed for the embodiment, or they may be well known and available to those skilled in the computer software arts. Examples of computer readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs, DVDs, floptical disks, etc. Specially configured hardware to store and execute program instructions such as magneto-optical media such as disk, and ROM, RAM, flash memory, etc. includes hardware devices. Examples of program instructions include high-level language code that can be executed by a computer using an interpreter, etc., as well as machine language code, such as produced by a compiler. The hardware devices described above are configured to act as one or more software modules, and vice versa, to perform the operations of the embodiments.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the techniques described may be performed in a different order than in the manner described and/or components such as systems, structures, devices, circuits, etc. described may be combined or combined in a manner different than in the manner described. , other components or equivalents may be substituted or substituted to achieve suitable results.

Therefore, other implementations, other embodiments, and equivalents of the claims are covered by the claims.
<Mode for carrying out the invention>

The foregoing describes relevant matter in the best mode for carrying out the invention.

Claims (25)

A non-contact method of measuring a Physiological parameter for a subject, executed by one or more processors, comprising:
acquiring a plurality of image frames of the subject from an imaging device ;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
calculating a first difference value and a second difference value based on the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; and wherein the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is a difference value between the first color channel value and the second color channel value for the same image frame. indicating a difference value of the three color channel values;
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; obtaining a first characteristic value based on the average value of the difference values;
obtaining a second characteristic value based on the second difference value for at least one image frame included in the first image frame group and an average value of the second difference value for the first image frame group; and determining the biometric index for the subject based on the first characteristic value and the second characteristic value,
The first color channel value is the average pixel value of the first color channel for one image frame, the second color channel value is the average pixel value of the second color channel for one image frame, and the third color channel value is the average pixel value of the second color channel for one image frame. A biometric method in which the channel value indicates the average pixel value of the third color channel for one image frame. The method of claim 1, wherein the biometric index includes at least one of heart rate and blood pressure. 2. The biometric method of claim 1, wherein the first, second and third color channels are color channels according to the RGB color space. To reduce noise, considering the absorbance of hemoglobin and oxyhemoglobin,
4. The bioindex measurement method of claim 3, wherein the first color channel is set to the Green channel, the second color channel is set to the Red channel, and the third color channel is set to the Blue channel. the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group;
the second characteristic value is obtained based on a second deviation value of the second difference value for at least one image frame included in the first image frame group;
the first deviation value is calculated based on the first difference value for the at least one image frame and an average value of the first difference values for the first group of image frames;
The biometric index of claim 1, wherein the second deviation value is calculated based on the second difference value for the at least one image frame and an average value of the second difference values for the first group of image frames. Measuring method. 6. The bioindex measuring method according to claim 5, wherein said first characteristic value and said second characteristic value are normalized values. The first characteristic value is a value normalized by a first standard deviation value, the second characteristic value is a value normalized by a second standard deviation value,
The first standard deviation value is the standard deviation value of the first difference value for the first image frame group, and the second standard deviation value is the standard deviation value of the second difference value for the first image frame group. 7. The bioindex measurement method according to claim 6. 2. The biometric index measuring method according to claim 1, wherein said biometric index for said subject is determined based on a third characteristic value obtained by summing said first characteristic value and said second characteristic value. further comprising outputting the biometric index;
the determined biometric index includes a first biometric index and a second biometric index;
2. The bioindex measuring method according to claim 1, wherein the output bioindex is determined based on the first bioindex and the second bioindex. the first biometric index is determined based on a second group of image frames, the second biometric index is determined based on a third group of image frames;
the number of image frames included in the first image frame group is smaller than the number of image frames included in the second and third image frame groups;
10. The method of claim 9, wherein the first image frame group is included in the second image frame group. The method of claim 10, wherein the number of image frames included in the second image frame group is the same as the number of image frames included in the third image frame group. further comprising outputting the biometric index based on the determined biometric index;
the determined biometric index comprises at least four preliminary biometric indices;
2. The biometric index measuring method according to claim 1, wherein said biometric index to be output is determined based on said four preliminary biometric indices. further comprising outputting the biometric index based on the determined biometric index;
the output biometric index includes a first biometric index and a second biometric index;
the second biometric index is the same type of biometric index as the first biometric index, and the second biometric index is output after the first biometric index is output;
The method of claim 1, wherein if the difference between the second bioindex and the first bioindex exceeds a reference value, the second bioindex is corrected and output. A non-contact method of measuring a Physiological parameter for a subject , executed by one or more processors, comprising :
obtaining a plurality of image frames of the subject from an infrared camera;
obtaining a first region value, a second region value and a third region value for at least one image frame included in the plurality of image frames;
calculating a first difference value and a second difference value based on the first region value, the second region value and the third region value for at least one image frame included in the plurality of image frames; ,
said first difference value for at least one image frame included in said first image frame group for a first group of image frames acquired at a first preset time, and said first difference value for said first group of image frames; obtaining a first characteristic value based on the average value of the difference values;
obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference values for the first image frame group; determining the biometric index for the subject based on the first characteristic value and the second characteristic value;
The first region value is an average pixel value for a first region of interest in one image frame, the second region value is an average pixel value for a second region of interest in one image frame, and the third region value is is the average pixel value for the third region of interest in one image frame,
The first difference value is the difference between the first area value and the second area value for the same image frame, and the second difference value is the difference between the first area value and the third area value for the same image frame. A bioindex measurement method that is a value. 15. The biometric index measuring method of claim 14, wherein the biometric index includes at least one of heart rate and blood pressure. the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group;
the second characteristic value is obtained based on a second deviation value of the second difference value for at least one image frame included in the first image frame group;
the first deviation value is calculated based on the first difference value for the at least one image frame and an average value of the first difference values for the first group of image frames;
The biometric index of claim 14, wherein the second deviation value is calculated based on the second difference value for the at least one image frame and an average value of the second difference values for the first group of image frames. Measuring method. A bioindex measuring device for measuring a bioindex (physiological parameter) for a subject in a non-contact manner,
an image acquisition unit for acquiring an image frame of the person to be measured; and a control unit for acquiring the biometric index using the image frame,
The control unit
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of obtained image frames;
calculating a first difference value and a second difference value based on the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; death,
for a first group of image frames acquired at a first preset time, the first difference value for at least one image frame included in the first group of image frames, and the first difference value for the first group of image frames; Obtain the first characteristic value based on the average value of one difference value,
obtaining a second characteristic value based on the average value of the second difference value for at least one image frame included in the first image frame group and the second difference value for the first image frame group;
determining the biometric index for the subject based on the first characteristic value and the second characteristic value;
The first color channel value is the average pixel value for the first color channel in one image frame, the second color channel value is the average pixel value for the second color channel in one image frame, and the third color channel value is the average pixel value for the second color channel in one image frame. channel value is the average pixel value for the third color channel in one image frame,
The first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value and the third color channel value for the same image frame. A biometric measuring device that is the difference value of the channel values. the first, second and third color channels are color channels according to the RGB color space;
To reduce noise, considering the absorbance of hemoglobin and oxyhemoglobin,
18. The biometric measuring device of claim 17, wherein the first color channel is set to the Green channel, the second color channel is set to the Red channel, and the third color channel is set to the Blue channel. the first characteristic value is obtained based on a first deviation value of the first difference value for at least one image frame included in the first image frame group;
the second characteristic value is obtained based on a second deviation value of the second difference value for at least one image frame included in the first image frame group;
the first deviation value is calculated based on the first difference value for the at least one image frame and an average value of the first difference values for the first group of image frames;
The biometric index of claim 17, wherein the second deviation value is calculated based on the second difference value for the at least one image frame and an average value of the second difference values for the first group of image frames. measuring device. The control unit acquires biometric information based on the determined biometric index,
18. The biometric index measuring device according to claim 17, wherein the biometric information includes at least one of emotion information, drowsiness information, stress information, and excitement level information. A non-contact method of measuring a Physiological parameter for a subject , executed by one or more processors, comprising :
acquiring a plurality of image frames of the subject from an imaging device ;
obtaining a first color channel value, a second color channel value and a third color channel value for at least one image frame included in the plurality of image frames;
Obtaining a first characteristic value based on the first, second and third color channel values for at least one image frame included in a first group of image frames acquired during a first preset time. stages and
Obtaining a second characteristic value based on the first, second and third color channel values for at least one image frame included in a second group of image frames acquired during a second preset time. and determining the biometric index based on the first characteristic value and the second characteristic value;
the first group of image frames and the second group of image frames are at least partially overlapped;
To determine the biometric index,
The first characteristic value for a first image frame included in the first image frame group but not included in the second image frame group is used to obtain the first image frame group and the second image frame group. The first characteristic value and the second characteristic value for the second image frame that is included in the second image frame group are used, and the second characteristic value is used for the third image frame that is included in the second image frame group but not included in the first image frame group. A bioindex measurement method in which characteristic values are utilized. calculating a first difference value based on at least a portion of the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; and
calculating a second difference value based on at least one of the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; and
the first characteristic value is obtained based on the first difference value for at least one image frame included in the first image frame group and an average value of the first difference values for the first image frame group;
the second characteristic value is obtained based on an average value of the first difference value for at least one image frame included in the second image frame group and the second difference value for the second image frame group;
The first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value and the third color channel value for the same image frame. 22. The bioindex measurement method of claim 21, which is a difference value of channel values. calculating a first difference value based on at least one of the first color channel value, the second color channel value and the third color channel value for at least one image frame included in the plurality of image frames; and
the first characteristic value is obtained based on a first deviation value of a first difference value for at least one image frame included in the first image frame group;
the second characteristic value is obtained based on a second deviation value of the first difference value for at least one image frame included in the second image frame group;
the first deviation value is calculated based on the first difference value for at least one image frame included in the first image frame group and the average value of the first difference values for the first image frame group;
the second deviation value is calculated based on an average value of the first difference value for at least one image frame included in the second image frame group and the first difference value for the second image frame group; The bioindex measuring method according to claim 21. A bioindex output method for acquiring and outputting a bioindex in a non-contact manner executed by one or more processors ,
obtaining a plurality of image frames from an imaging device , including a first group of image frames and a second group of image frames at least partially overlapping the first group of image frames;
obtaining the biometric index based on the first group of image frames;
outputting the biometric index at a first time point;
obtaining a first biometric index based on the second group of image frames;
and outputting the biometric index at a second time later than the first time,
the biometric index output at the first time point is the biometric index obtained based on the first image frame group;
The biometric index output at the second time point is
if the difference between the first biometric index and the biometric index output at the first time point is equal to or less than a reference value, the first biometric index;
A bioindex output method, wherein if a difference between the first bioindex and the bioindex output at the first time exceeds a reference value, the bioindex is corrected from the bioindex output at the first time. The corrected biometric index is
when the first bioindex is greater than the bioindex output at the first time point, the bioindex is corrected by adding a preset value to the bioindex output at the first time point;
25. When the first biometric index is smaller than the biometric index output at the first time point, the biometric index output at the first time point is corrected by decreasing a preset value. A biometric output method as described.

Images