KR102407956B1 - Smart mirror device - Google Patents

Abstract

The present invention relates to a bio-index measuring method and a bio-index measuring device, and more particularly, to a bio-index measuring method for measuring a bio-index in a non-contact manner, and a bio-index measuring device using the same. A method for measuring a biometric index according to the present invention includes: obtaining a plurality of image frames for a subject; a first color channel value, a second color channel value, and a second color channel value for at least one image frame included in the plurality of image frames obtaining a three-color channel value; for at least one image frame included in the plurality of image frames, a first difference based on the first color channel value, the second color channel value, and the third color channel value value and the second difference value. calculating - the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value for the same image frame and represents a difference value between the third color channel value and the first difference value for at least one image frame included in the first image frame group with respect to the first image frame group acquired for a first preset time. and obtaining a first characteristic based on an average value of the first difference values for the first image frame group, a second difference value for at least one image frame included in the first image frame group, and the obtaining a second characteristic value based on an average value of the second difference values for a first image frame group; and determining a biometric index for the subject based on the first characteristic value and the second characteristic value wherein the first color channel value is an average pixel value of a first color channel for one image frame, the second color channel value is an average pixel value of a second color channel for one image frame, and , the third color channel value may represent an average pixel value of the third color channel for one image frame.

Description

Smart mirror device {Smart mirror device}

Plethysmography is a technique for measuring and analyzing changes in shape and shape when the volume of tissues of the human body, such as organs or blood vessels, changes according to the flow of blood vessels.

The most common technique for measuring photoplethysmography (PPG) using light is to analyze the amount of transmitted light for light irradiated to the human body. This is explained by the Beer-Lambert law. According to this law, the change in transmitted light is a signal proportional to the change in the volume of the transmitted material, so even if the absorbance of the material is not known, the state of the heart can be grasped using the PPG.

Recently, a technology using remote photoplethysmography (rPPG) has emerged, going one step further from the technology using PPG. This is the most popular technology that uses PPG to detect heartbeat signals. A technology that directly contacts a device with a camera and a nearby light, such as a smartphone, to the human body to irradiate the light, and then directly measures the transmitted light to obtain PPG. If there is, a technology related to rPPG (remote photoplethysmography) that detects a change in the volume of a blood vessel from a signal obtained from an image taken with a camera is continuously being researched and developed.

Since the technology using rPPG does not require contact between the object and the measuring device, it can be applied in various ways to devices and places equipped with cameras, such as airport immigration offices and telemedicine.

However, the rPPG technology extracts only the signal related to the volume change of the measurement object from the photographed image because noise generated from ambient light and movement of the object has a large effect on the signal in the process of photographing the object with the camera. It can be seen as a core technology among the technologies for measuring biosignals using this rPPG.
In addition, recently, a smart mirror device capable of displaying has been developed, and technologies for displaying the user's health information using such a smart mirror device are being developed.
(Patent Document 0001) Patent Publication No. 10-2016-0016263 (2016.02.15.)

A problem to be solved according to an embodiment is to obtain a biometric index in a non-contact manner.

A problem to be solved according to another exemplary embodiment is to reduce noise according to movement of a subject in order to obtain a biometric index.

Another problem to be solved according to another embodiment is to reduce noise according to a change in the intensity of external light in order to obtain a biometric index.

Another problem to be solved according to another embodiment is to simultaneously obtain various biometric indices.

Another problem to be solved according to another embodiment is to obtain biometric information based on various biometric indices.

Another problem to be solved according to another embodiment is to simultaneously acquire various biometric indices to be correlated.

Another problem to be solved according to another exemplary embodiment is to detect drowsiness based on a heart rate of an object and LF/HF of a heart rate signal.

A problem to be solved according to another embodiment relates to a smart mirror device for acquiring at least two or more related biometric indices.

Another problem to be solved according to another embodiment relates to a method of operating a smart mirror device for acquiring at least two or more related biometric indices.

A problem to be solved according to another embodiment relates to a smart mirror device including an opening/closing device.

A method of measuring a biometric index in a non-contact manner according to an embodiment includes: obtaining a plurality of image frames for a subject; a first color channel value for at least one image frame included in the plurality of image frames; obtaining a color channel value and a third color channel value; calculating the first color channel value, the second color channel value, and the third color channel value with respect to at least one image frame included in the plurality of image frames Based on the first difference value and the second difference value. calculating - the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value for the same image frame represents a difference value between the value of the third color channel and the value of the third color channel; and obtaining a first characteristic based on an average value of the first difference values for the first image frame group, a second difference value for at least one image frame included in the first image frame group, and the obtaining a second characteristic value based on an average value of the second difference values for a first image frame group; and determining a biometric index for the subject based on the first characteristic value and the second characteristic value wherein the first color channel value is an average pixel value of a first color channel for one image frame, the second color channel value is an average pixel value of a second color channel for one image frame, and , the third color channel value may represent an average pixel value of the third color channel for one image frame.

A method of measuring a biometric index using an infrared camera according to an embodiment includes: obtaining a plurality of image frames for a subject using an infrared camera; at least one image frame included in the plurality of image frames obtaining a first region value, a second region value, and a third region value; the first region value, the second region value, and the third region for at least one image frame included in the plurality of image frames calculating a first difference value and a second difference value based on the values; for at least one image frame included in the first image frame group for a first image frame group acquired for a first preset time obtaining a first characteristic value based on a first difference value and an average value of the first difference values for the first image frame group; obtaining a second characteristic value based on two difference values and an average value of the second difference values for the first image frame group; determining a biometric index, wherein the first region value is an average pixel value for a first region of interest in one image frame, and the second region value is an average value for a second region of interest in one image frame a pixel value, the third region value is an average pixel value for a third region of interest in one image frame, and the first difference value is a difference value between the first region value and the second region value for the same image frame; , the second difference value may be a difference value between the first region value and the third region value for the same image frame.

The apparatus for obtaining a biometric index according to an embodiment includes an image obtaining unit for obtaining an image frame of the subject, and a controller for obtaining a biometric index using the image frame, wherein the control unit includes a plurality of acquired images A first color channel value, a second color channel value, and a third color channel value are obtained with respect to at least one image frame included in the frame, and the first color channel value, the second color channel value, and the third color channel value are obtained for at least one image frame included in the plurality of image frames. A first difference value and a second difference value are calculated based on the color channel value, the second color channel value, and the third color channel value, and for a first image frame group acquired for a first preset time, the obtains a first characteristic value based on an average value of the first difference value for at least one image frame included in one image frame group and the first difference value for the first image frame group, a second characteristic value is obtained based on an average value of a second difference value for at least one image frame included in an image frame group and an average value of the second difference value for the first image frame group, and the first characteristic value and determining a biometric index for the subject based on a second characteristic value, wherein the first color channel value is an average pixel value for a first color channel in one image frame, and the second color channel value is one is an average pixel value for a second color channel in an image frame of A difference value between the first color channel value and the second color channel value, and the second difference value may be a difference value between the first color channel value and the third color channel value for the same image frame.

A non-contact biometric index measurement method according to an embodiment includes: obtaining a plurality of image frames for a subject; a first color channel value and a second color channel value for at least one image frame included in the plurality of image frames and obtaining a third color channel value, based on the first, second and third color channel values for at least one image frame included in the first image frame group obtained for a first preset time. obtaining a first characteristic value, a second characteristic value based on the first, second and third color channel values for at least one image frame included in a second image frame group acquired for a second preset time and determining a biometric index based on the first characteristic value and the second characteristic value, wherein the first image frame group and the second image frame group overlap at least partially, and To determine the index, a first characteristic value for a first image frame included in the first image frame group but not included in the second image frame group is used, and the first image frame group and the second image frame group The first characteristic and the second characteristic value of the second image frame included in the image frame group are used, and the value of the third image frame included in the second image frame group but not included in the first image frame group is used. A second characteristic value may be used.

A non-contact biometric index measurement method according to an embodiment includes: acquiring a plurality of image frames including a first image frame group and a second image frame group that at least partially overlaps the first image frame group, the first image frame obtaining a biometric index based on the group, outputting the biometric index at a first time point, obtaining a first biometric index based on the second image frame group, at a second time point later than the first time point outputting the biometric index, wherein the biometric index output at the first time point is a biometric index obtained based on the first image frame group, and the biometric index output at the second time point is the first biometric index and the first bio-index when the difference between the bio-index output at the first time point is less than or equal to the reference value, and the second bio-index when the difference between the first bio-index and the bio-index output at the first time point exceeds the reference value It may be a biometric index corrected from the biometric index output at one time point.

A method of measuring a biometric index according to an embodiment includes: obtaining a plurality of image frames for a subject; setting a first area and a second area with respect to at least one image frame included in the plurality of image frames; determining the oxygen saturation level of the subject based on a first characteristic obtained based on at least two color channel values among a first color channel value, a second color channel value, and a third color channel value for the first region determining the heart rate of the subject based on a second characteristic obtained based on a first difference value that is a difference value between the first color channel value and the second color channel value for the first region; determining the blood pressure of the subject based on a second difference value that is a difference value between the first color channel value and the second color channel value for a second region and a third characteristic obtained based on the first difference value step, outputting the oxygen saturation, heart rate and blood pressure; wherein the first characteristic is obtained based on a first group of image frames acquired for a first time, the second characteristic is obtained based on a group of second image frames acquired for a second time, and 3 features are acquired based on a third image frame group acquired for a third time, and the first image frame group, the second image frame group, and the third image frame are such that the oxygen saturation, heart rate and blood pressure are obtained in correlation with each other A group may include a plurality of image frames in common.

A method for measuring a biometric index according to another embodiment includes: obtaining a plurality of image frames for a subject; a first region, a second region, and a third region with respect to at least one image frame included in the plurality of image frames setting, based on a first characteristic obtained based on at least two color channel values among a first color channel value, a second color channel value, and a third color channel value for the first region. determining oxygen saturation; determining the heart rate of the subject based on a second characteristic obtained based on a first difference value that is a difference value between the first color channel value and the second color channel value for the first region determining a second difference value that is a difference value between the first color channel value and the second color channel value for the second region and the first color channel value and the second color channel value for the third region determining the blood pressure of the subject based on a third characteristic obtained based on a third difference value that is a difference value of is obtained based on a first group of image frames obtained for a time period, the second characteristic is obtained based on a second group of image frames obtained during a second time, and the third characteristic is obtained based on a third group of images obtained for a third time. The first image frame group, the second image frame group, and the third image frame group may include a plurality of image frames in common so that the oxygen saturation, heart rate, and blood pressure are obtained based on a group of image frames. can

A method for measuring a biometric index according to another embodiment includes obtaining a plurality of image frames for a subject, a first color channel value and a second color channel with respect to at least one image frame included in the plurality of image frames obtaining a value and a third color channel value; based on a first characteristic obtained based on a color channel value of at least two or more of the first color channel value, the second color channel value, and the third color channel value. Determining the oxygen saturation level of the measurer, a first difference value that is a difference value between the first color channel value and the second color channel value, and a second difference value that is a difference value between the first color channel value and the third color channel value determining the heart rate of the subject based on a second characteristic obtained based on and outputting the oxygen saturation, heart rate, and blood pressure, wherein the first characteristic is acquired based on a first image frame group acquired for a first time, and the second characteristic is acquired for a second time the first image is obtained based on a second image frame group, the third characteristic is obtained based on a third image frame group obtained for a third time, and the oxygen saturation, heart rate and blood pressure are obtained in correlation with each other The frame group, the second image frame group, and the third image frame group may include a plurality of image frames in common.

A method for measuring a biometric index according to another embodiment includes: acquiring a plurality of image frames for a subject; setting at least two or more regions with respect to at least one image frame included in the plurality of image frames; obtaining a first color channel value, a second color channel value, a third color channel value, a fourth color channel value, and a fifth color channel value with respect to at least one image frame included in the plurality of image frames; determining the oxygen saturation level of the subject based on a first characteristic obtained based on at least two color channel values among a first color channel value, a second color channel value, and a third color channel value; determining the heart rate of the subject based on a second characteristic obtained based on a color channel value of at least two of a value, a second color channel value, and a third color channel value; the first color channel value; determining the blood pressure of the subject based on a third characteristic obtained based on a color channel value of at least two or more of a color channel value and a third color channel value; a fourth obtained based on the fourth color channel value determining the body temperature of the subject based on a characteristic, and outputting the oxygen saturation, heart rate, body temperature and blood pressure, wherein the first color channel value is a Green channel value, and the second color channel value is The red channel value may be a red channel value, the third color channel value may be a blue channel value, the fourth color channel value may be a saturation channel value, and the fifth color channel value may be a Hue channel value.

A method for measuring a biometric index according to another embodiment includes: acquiring a plurality of image frames for a subject; acquiring N preliminary heart rates based on at least one image frame included in the plurality of image frames; obtaining M preliminary oxygen saturation levels based on at least one image frame included in the plurality of image frames; acquiring K preliminary blood pressures based on at least one image frame included in the plurality of image frames; acquiring a heart rate based on the N preliminary heart rates, acquiring oxygen saturation based on the M preliminary oxygen saturation levels, acquiring blood pressure based on the K preliminary blood pressures, and the heart rate, oxygen saturation and outputting blood pressure, wherein when the image frame acquired when the subject is in the first state is a first image frame, the first image frame includes the N preliminary heart rates, the M preliminary oxygen saturation and It may be commonly included in the image frame for obtaining the K blood pressures.

The method for obtaining biometric information according to an embodiment includes: obtaining a plurality of image frames for a subject; obtaining an index; obtaining a second biometric index based on a second image frame group including at least one image frame included in the plurality of image frames; the first biometric index and the second biometric index Comprising the steps of obtaining biometric information based on at least one of and outputting the first biometric index, the second biometric index, and biometric information, in order to obtain biometric information by reflecting the specific state of the subject The first image frame group and the second image frame group may overlap at least partially.

According to an embodiment, the method for detecting drowsiness based on a heart rate may be performed by at least one processor, comprising: obtaining a heart rate of an object; obtaining a comparison result by comparing the heart rate with a reference heart rate; Comparing the heart rate with the reference heart rate to obtain a comparison result, obtaining a duration in which the heart rate is equal to or less than the reference heart rate based on the comparison result, and determining whether the duration reaches the reference duration The drowsiness detection step of determining the drowsiness state of the subject based on the drowsiness detection step may be included.

A method for detecting drowsiness based on a heart rate and LF/HF according to an embodiment may be performed by at least one processor, acquiring a heart rate with respect to an object, and comparing the heart rate with a reference heart rate to obtain a comparison result acquiring, acquiring a first drowsiness parameter based on the comparison result, acquiring an LF/HF (Low Frequency/High Frequency) value representing a ratio of sympathetic nerve activity and parasympathetic nerve activity of the subject , acquiring a second drowsiness parameter based on the LF/HF value of the subject, and detecting drowsiness of determining the drowsiness state of the subject using at least one of the first drowsiness parameter and the second drowsiness parameter. can do.

A smart mirror device according to an embodiment includes a reflective mirror surface, an image acquisition unit for acquiring a plurality of image frames for a subject, and a display disposed behind the reflective mirror surface and passing through the reflective mirror surface to display visual information and a control unit for controlling the operation of the unit and the image acquisition unit and the display unit and for acquiring the biometric index in a non-contact manner, wherein the control unit is based on a first image frame group included in the plurality of image frames at a first time point controlling the display unit to display the obtained first biometric index, and controlling the display unit to display a second biometric index obtained based on a second image frame group included in the plurality of image frames at a second time point, The first image frame group and the second image frame group include at least one image frame in common, and the first image frame group and the second The at least one image frame commonly included in the image frame group includes an image frame obtained in a first state of the subject observed by the subject through the reflective mirror surface before the first and second time points can do.

A method of operating a smart mirror device according to an embodiment includes acquiring an on-trigger, acquiring a plurality of image frames for a subject, and a first image frame group included in the plurality of image frames obtaining a first biometric index based on , obtaining a second biometric index based on a second image frame group included in the plurality of image frames; Comprising the steps of acquiring an off-trigger and stopping acquisition of a plurality of image frames for the subject, the first image so that a correlation is given to the first biometric index and the second biometric index The frame group and the second image frame group include at least one image frame in common, the on-trigger is obtained from at least one sensor, and the off-trigger is the plurality of It may be obtained from an image sensor for obtaining an image frame.

A smart mirror device according to another embodiment includes a reflective mirror surface, an image acquisition unit for acquiring a plurality of image frames, a display unit disposed behind the reflective mirror surface, and displaying visual information through the reflective mirror surface, the It is disposed in front of the image acquisition unit, and includes an opening/closing unit for opening and closing the field of view of the image acquisition unit, and a control unit for controlling operations of the image acquisition unit and the display unit to obtain a biometric index, wherein the surface of the opening/closing unit is formed of a reflective mirror When the opening/closing unit is in an open state, the control unit controls the display unit so that the biometric index and at least one or more visual information are displayed through the display unit. The display unit may be controlled to be displayed through the display unit.

According to an embodiment of the present invention, a method for obtaining a biometric index in a non-contact manner may be provided.

According to another embodiment of the present invention, a method for reducing noise according to the movement of a subject may be provided.

According to another embodiment of the present invention, a method for reducing noise according to a change in the intensity of external light may be provided.

According to another embodiment of the present invention, a method of simultaneously acquiring various biometric indices may be provided.

According to another embodiment of the present invention, a method for obtaining biometric information based on various biometric indices may be provided.

According to another embodiment of the present invention, there may be provided a method of simultaneously acquiring various bio-indices to be correlated.

According to another embodiment of the present invention, a smart mirror device for acquiring at least two or more related biometric indices may be provided.

According to another embodiment of the present invention, there may be provided a method of operating a smart mirror device for acquiring at least two related biometric indices.

Another object of the present invention is to provide a method and apparatus for detecting drowsiness based on a heart rate of an object and LF/HF of a heart rate signal.

According to another embodiment of the present invention, a smart mirror device including an opening/closing device may be provided.

1 is a diagram of a biometric index and biometric information management system according to an embodiment.
2 is a diagram of a biometric index and biometric information management system according to another exemplary embodiment.
3 is a diagram for explaining a biometric index obtaining apparatus according to an exemplary embodiment.
4 is a flowchart illustrating a method for obtaining a biometric index according to an exemplary embodiment.
5 is a diagram illustrating a method for obtaining a biometric index according to an exemplary embodiment.
6 is a flowchart illustrating a method for acquiring biometric information according to an embodiment.
7 and 8 are diagrams for explaining a method for obtaining a biometric index using a biometric index acquisition model.
9 is a flowchart illustrating a method for measuring a heart rate according to an exemplary embodiment.
10 is a flowchart illustrating a method for measuring oxygen saturation according to an exemplary embodiment.
11 is a flowchart illustrating a method for measuring oxygen saturation according to another exemplary embodiment.
12 is a flowchart illustrating a method for measuring blood pressure according to an exemplary embodiment.
13 is a flowchart illustrating a method for measuring blood pressure according to another exemplary embodiment.
14 is a flowchart illustrating a method for measuring body temperature according to an exemplary embodiment.
15 is a flowchart illustrating a method for acquiring a heart rate according to an exemplary embodiment.
16 is a graph of color channel values according to an exemplary embodiment.
17 is a graph for explaining a noise reduction method according to an embodiment.
18 is a diagram showing the absorbance of hemoglobin and oxyhemoglobin in the visible light band.
19 is a diagram for describing a method of acquiring a characteristic value according to an exemplary embodiment.
20 is a diagram for describing a method of acquiring a characteristic value according to another exemplary embodiment.
21 is a diagram for explaining a method of using a plurality of characteristic values.
22 is a graph in which a frequency component is extracted from a graph for characteristic values.
23 is a diagram for describing a method for acquiring a heart rate according to an exemplary embodiment.
24 is a flowchart illustrating a method of correcting an output heart rate according to an exemplary embodiment.
25 is a diagram for describing a method for extracting a heartbeat signal according to an exemplary embodiment.
26 is a diagram for describing a method of acquiring a heart rate using infrared rays according to an exemplary embodiment.
27 is a diagram for describing a method for acquiring a heart rate using infrared rays according to an exemplary embodiment.
28 is a flowchart illustrating a method for obtaining a biometric index according to an embodiment.
29 is a view for explaining a method for obtaining a plurality of biometric indices and biometric information according to an embodiment.
30 is a diagram for explaining a method for obtaining a plurality of biometrics according to an embodiment.
31 is a diagram for explaining a method of obtaining a plurality of biometrics according to an embodiment.
32 is a diagram for explaining a method for obtaining a plurality of biometrics according to an embodiment.
33 is a view for explaining a method for obtaining a plurality of biometrics according to an embodiment.
34 is a diagram for explaining a method for obtaining a plurality of biometric indices according to an exemplary embodiment.
35 is a diagram for explaining a method for obtaining a plurality of biometric indices according to an embodiment.
36 is a diagram for describing a method of obtaining a plurality of related biometric indices according to an embodiment.
37 is a diagram for describing a method for obtaining a plurality of biometric indices according to an embodiment.
38 is a block diagram of a device for detecting drowsiness based on a heart rate.
39 is a flowchart of a method for detecting drowsiness based on heart rate.
40 is a graph relating to an average heart rate of a subject based on the heart rate of the subject.
41 is a graph of a heart rate of a subject for explaining a situation in which the subject is sensed as drowsy based on the heart rate.
42 is a graph relating to a heart rate including noise.
43 is a graph for explaining a situation in which the subject recovers to a normal state.
44 is a flowchart of a method for detecting drowsiness based on LF/HF.
45 is a graph of LF/HF of a subject for explaining a situation in which the subject is sensed as drowsy based on LF/HF.
46 is a graph relating to LF/HF of a subject for showing a situation in which the subject recovers from a drowsy state based on LF/HF.
47 is a flowchart of a method for detecting drowsiness based on heart rate and LF/HF.
48 is a diagram for explaining a smart mirror device according to an embodiment.
49 is a view for explaining a smart mirror device according to an embodiment.
50 is a diagram for explaining a smart mirror device outputting a guide area according to an exemplary embodiment.
51 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment.
52 is a view for explaining a smart mirror device according to an embodiment.
53 is a diagram for describing a display device for measuring a biometric index in real time according to an exemplary embodiment.
54 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment.
55 is a view for explaining a smart mirror device including an opening and closing device according to an embodiment.
56 is a view for explaining a smart mirror device disposed in a shoe closet according to an embodiment.
57 is a flowchart illustrating a method of operating a smart mirror device according to an exemplary embodiment.
58 and 59 are diagrams for explaining an operation of a smart mirror device using a trigger signal according to an embodiment.
60 is a diagram for explaining a method of operating a smart mirror device according to an embodiment.
61 is a diagram for explaining an operation of a smart mirror device according to an embodiment.
62 is a view for explaining a method of operating a smart mirror device according to an embodiment.
63 is a diagram for explaining an operation of a smart mirror device according to an embodiment.
64 is a diagram for explaining a method of operating a smart mirror device according to an embodiment.
65 and 66 are diagrams for explaining an operation of a smart mirror according to an embodiment.
67 is a view for explaining an apparatus for measuring a biometric index according to an embodiment.
68 is a diagram for describing an apparatus for measuring a biometric index according to an exemplary embodiment.
69 is a diagram for explaining a biometric index measuring device disposed in an autonomous vehicle according to an exemplary embodiment.
70 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
71 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
72 is a diagram for describing a method of operating a driving scheduling assistance device using a biometric index measuring device according to an exemplary embodiment.
73 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
74 is a flowchart for explaining a method of operating an apparatus for calculating a driving index according to an exemplary embodiment.
75 is a diagram for describing an apparatus for monitoring infants and toddlers according to an embodiment.
76 is a view for explaining the occurrence of an event for infants and young children.
77 is a flowchart for explaining a method of operating an apparatus for monitoring infants and toddlers according to an embodiment.
78 is a flowchart for explaining a method of operating an apparatus for monitoring infants and toddlers according to an embodiment.
79 is a diagram illustrating a mobile application for implementing an infant monitoring system according to an embodiment.
80 is a view for explaining an apparatus for measuring a biometric index disposed in a reading room according to an exemplary embodiment.
81 is a flowchart illustrating a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
82 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
83 is a view for explaining an apparatus for measuring a biometric index used for cognitive rehabilitation treatment according to an embodiment.
84 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
85 is a view for explaining an apparatus for measuring a biometric index used for immigration control according to an embodiment.
86 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.
87 is a view for explaining a biometric index measuring device used in a security device according to an embodiment.
88 is a flowchart illustrating a method of operating a security device according to an embodiment.
89 is a view for explaining an apparatus for measuring a biometric index used in a kiosk according to an embodiment.
90 is a flowchart illustrating a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

The embodiments described in the present specification are for clearly explaining the spirit of the present invention to those of ordinary skill in the art distribution to which the present invention belongs, so the present invention is not limited to the embodiments described in this specification, and the The scope should be construed to include modifications or variations without departing from the spirit of the present invention.

The terms used in the present specification have been selected as widely used general terms as possible in consideration of the functions in the present invention, but they may vary depending on the intention, precedent, or emergence of new technology of those of ordinary skill in the art to which the present invention belongs. can However, if a specific term is defined and used in an arbitrary sense, the meaning of the term will be separately described. Therefore, the terms used in this specification should be interpreted based on the actual meaning of the terms and the contents of the entire specification, rather than the names of simple terms.

The drawings attached to this specification are for easily explaining the present invention, and the shapes shown in the drawings may be exaggerated as necessary to help understand the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a known configuration or function related to the present invention may obscure the gist of the present invention, a detailed description thereof will be omitted if necessary.

According to an embodiment, there is provided a method of measuring a biometric index in a non-contact manner, comprising: obtaining a plurality of image frames for a subject; a first color channel value for at least one image frame included in the plurality of image frames; obtaining a second color channel value and a third color channel value; the first color channel value, the second color channel value, and the third color channel value for at least one image frame included in the plurality of image frames a first difference value and a second difference value based on the values. calculating - the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value for the same image frame and represents a difference value between the third color channel value and the first difference value for at least one image frame included in the first image frame group with respect to the first image frame group acquired for a first preset time. and obtaining a first characteristic based on an average value of the first difference values for the first image frame group, a second difference value for at least one image frame included in the first image frame group, and the obtaining a second characteristic value based on an average value of the second difference values for a first image frame group; and determining a biometric index for the subject based on the first characteristic value and the second characteristic value wherein the first color channel value is an average pixel value of a first color channel for one image frame, the second color channel value is an average pixel value of a second color channel for one image frame, and , wherein the third color channel value represents an average pixel value of the third color channel for one image frame.

Here, the biometric index may include at least one of heart rate and blood pressure.

Here, the first, second, and third color channels may be color channels according to the RGB color space.

Here, in order to reduce noise in consideration of the absorbance of hemoglobin and oxyhemoglobin, the first color channel is set as a green channel, the second color channel is set as a red channel, and the third color channel is a blue channel can be set to

Here, the first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group, and the second characteristic value is the first image frame group. It is obtained based on a second deviation value of the second difference value with respect to at least one image frame included in a frame group, wherein the first deviation value includes a first difference value for the at least one image frame and the first difference value. is calculated based on an average value of the first difference value for a group of image frames, wherein the second deviation value is a second difference value for the at least one image frame and the second difference for the first group of image frames It can be calculated based on the average value of the values.

Here, the first characteristic value and the second characteristic value may be normalized values.

Here, the first characteristic value is a value normalized by a first standard deviation value, the second characteristic value is a value normalized by a second standard deviation value, and the first standard deviation value is the first image It may be a standard deviation value of the first difference value for the frame group, and the second standard deviation value may be a standard deviation value of the second difference value for the first image frame group.

Here, the biometric index for the subject may be determined based on a third characteristic value obtained as a sum of the first characteristic value and the second characteristic value.

Here, the method further comprising outputting a biometric index, wherein the determined biometric index includes a first biometric index and a second biometric index, and the output biometric index is the first biometric index and the second biometric index It can be determined based on the index.

Here, the first biometric index is determined based on a second image frame group, the second biometric index is determined based on a third image frame group, and the number of image frames included in the first image frame group is the The number of image frames included in the second and third image frame groups may be smaller than the number of image frames included in the second and third image frame groups, and the first image frame group may be included in the second image frame group.

Here, the number of image frames included in the second image frame group may be the same as the number of image frames included in the third image frame group.

Here, the method further comprising outputting a biometric index based on the determined biometric index, wherein the determined biometric index includes at least four preliminary biometric indices, and the output biometric index is the four preliminary biometric indices. can be determined based on

Here, the method further comprises outputting a biometric index based on the determined biometric index, wherein the output biometric index includes a first biometric index and a second biometric index, and the second biometric index is the second biometric index. 1 bio-index and the same type of bio-index, the second bio-index is output after the first bio-index is output, and the difference between the second bio-index and the first bio-index exceeds a reference value The second biometric index may be corrected and output.

According to another embodiment, there is provided a method of measuring a biometric index in a non-contact manner using an infrared camera, the method comprising: obtaining a plurality of image frames of a subject by using an infrared camera; at least one included in the plurality of image frames obtaining a first region value, a second region value, and a third region value with respect to an image frame of and calculating a first difference value and a second difference value based on the third region value. At least one image included in the first image frame group with respect to the first image frame group acquired for a first preset time. obtaining a first characteristic value based on an average value of the first difference value for the image frame and the first difference value for the first image frame group; obtaining a second characteristic value based on a second difference value for an image frame and an average value of the second difference value for the first image frame group, and based on the first characteristic value and the second characteristic value determining a biometric index for the subject, wherein the first region value is an average pixel value for a first region of interest in one image frame, and the second region value is a second region value in one image frame an average pixel value for a region of interest, the third region value is an average pixel value for a third region of interest in one image frame, and the first difference value is a first region value and a second region value for the same image frame The biometric index measurement method may be provided as a difference value between values, and the second difference value is a difference value between a first region value and a third region value for the same image frame.

Here, the biometric index may include at least one of heart rate and blood pressure.

Here, the first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group, and the second characteristic value is the first image frame group. It is obtained based on a second deviation value of the second difference value with respect to at least one image frame included in a frame group, wherein the first deviation value includes a first difference value for the at least one image frame and the first difference value. is calculated based on an average value of the first difference value for a group of image frames, wherein the second deviation value is a second difference value for the at least one image frame and the second difference for the first group of image frames It can be calculated based on the average value of the values.

According to another embodiment, there is provided a bio-index measuring device for measuring a bio-index in a non-contact manner, an image acquisition unit for obtaining an image frame for the subject, and a controller for obtaining a bio-index using the image frame wherein the control unit obtains a first color channel value, a second color channel value, and a third color channel value with respect to at least one image frame included in the plurality of obtained image frames, Calculate a first difference value and a second difference value based on the first color channel value, the second color channel value, and the third color channel value for at least one image frame included therein, and a first preset time Based on the average value of the first difference value for at least one image frame included in the first image frame group and the first difference value for the first image frame group with respect to the first image frame group acquired during to obtain a first characteristic value, and based on an average value of a second difference value for at least one image frame included in the first image frame group and the second difference value for the first image frame group obtain two characteristic values, and determine a biometric index for the subject based on the first characteristic value and the second characteristic value, wherein the first color channel value is an average pixel for a first color channel in one image frame value, wherein the second color channel value is an average pixel value for a second color channel in one image frame, the third color channel value is an average pixel value for a third color channel in one image frame, and The first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the difference between the first color channel value and the third color channel value for the same image frame. A biometric index measuring device that is a value may be provided.

Here, the first, second, and third color channels are color channels according to the RGB color space, and in order to reduce noise in consideration of the absorbance of hemoglobin and oxyhemoglobin, the first color channel is set as a green channel, , the second color channel may be set as a red channel, and the third color channel may be set as a blue channel.

Here, the first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group, and the second characteristic value is the first image frame group. It is obtained based on a second deviation value of the second difference value with respect to at least one image frame included in a frame group, wherein the first deviation value includes a first difference value for the at least one image frame and the first difference value. is calculated based on an average value of the first difference value for a group of image frames, wherein the second deviation value is a second difference value for the at least one image frame and the second difference for the first group of image frames It can be calculated based on the average value of the values.

Here, the controller obtains biometric information based on the determined biometric index, and the biometric information may include at least one of emotion information, drowsiness information, stress information, and excitability information.

According to another embodiment, there is provided a method for measuring a biometric index in a non-contact manner, comprising: obtaining a plurality of image frames for a subject; a first color channel for at least one image frame included in the plurality of image frames; obtaining a value, a second color channel value, and a third color channel value; the first, second and third for at least one image frame included in the first image frame group acquired for a first preset time obtaining a first characteristic value based on a color channel value; the first, second and third color channel values for at least one image frame included in a second image frame group acquired for a second preset time obtaining a second characteristic value based on At least partially overlapping, in order to determine the biometric index, a first characteristic value for a first image frame included in the first image frame group but not included in the second image frame group is used, The first and second characteristic values for the image frame group and the second image frame included in the second image frame group are used, and are included in the second image frame group, but are not included in the first image frame group. A method for measuring a biometric index may be provided in which a second characteristic value for a third image frame that does not exist is used.

wherein, for at least one image frame included in the plurality of image frames, a first difference value is calculated based on at least a part of the first color channel value, the second color channel value, and the third color channel value. step, wherein the first characteristic value is an average value of the first difference value for at least one image frame included in the first image frame group and the first difference value for the first image frame group is obtained based on, and the second characteristic value is an average value of the first difference value for at least one image frame included in the second image frame group and the second difference value for the second image frame group the first difference value is a difference value between a first color channel value and a second color channel value for the same image frame, and the second difference value is a first color channel value and a second color channel value for the same image frame. It may be a difference value between three color channel values.

Here, the first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group, and the second characteristic value is the second image frame group. It is obtained based on a second deviation value of the first difference value for at least one image frame included in the frame group, and the first deviation value is obtained based on a second deviation value for at least one image frame included in the first image frame group. It is calculated based on a first difference value and an average value of the first difference value with respect to the first image frame group, and the second deviation value for at least one image frame included in the second image frame group It may be calculated based on a first difference value and an average value of the first difference value for the second image frame group.

According to another embodiment, there is provided a biometric index output method for obtaining and outputting a biometric index in a non-contact manner, wherein a plurality of obtaining an image frame, obtaining a biometric index based on the first image frame group, outputting a biometric index at a first time point, obtaining a first biometric index based on the second image frame group step, outputting the biometric index at a second time point later than the first time point, wherein the biometric index output at the first time point is the biometric index obtained based on the first image frame group, and the second The bio-index output at the time point is the first bio-index when the difference between the first bio-index and the bio-index output at the first time point is equal to or less than a reference value, and the first bio-index and the bio-index output at the first time point When the difference between the indices exceeds the reference value, a method for outputting a biometric index that is a biometric index corrected from the biometric index output at the first time point may be provided.

Here, the corrected bio-index is a bio-index corrected by adding a preset value to the bio-index output at the first time point when the first bio-index is greater than the bio-index output at the first time point, When the bio-index is smaller than the bio-index output at the first time point, the bio-index may be a corrected bio-index by subtracting a preset value from the bio-index output at the first time point.

According to one embodiment, there is provided a method for simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, comprising: obtaining a plurality of image frames for a subject; Setting a first region and a second region for at least one image frame, wherein at least two color channel values of a first color channel value, a second color channel value, and a third color channel value for the first region are applied. determining the oxygen saturation level of the subject based on a first characteristic obtained based on the first characteristic; based on a first difference value that is a difference value between the first color channel value and the second color channel value for the first region determining the heart rate of the subject based on the obtained second characteristic; a second difference value that is a difference value between the first color channel value and the second color channel value for the second region; and the first difference value determining the blood pressure of the subject based on a third characteristic obtained based on wherein the first characteristic is obtained based on a first group of image frames acquired for a first time, the second characteristic is obtained based on a group of second image frames acquired for a second time, and 3 features are acquired based on a third image frame group acquired for a third time, and the first image frame group, the second image frame group, and the third image frame are such that the oxygen saturation, heart rate and blood pressure are obtained in correlation with each other A group may be provided with a biometric index measurement method including a plurality of image frames in common.

Here, the first color channel value may be a green channel value, the second color channel value may be a red channel value, and the third color channel value may be a blue channel value.

Here, the first characteristic is that the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the absorbance of oxyhemoglobin for a third color channel in which the absorbance is higher than that of hemoglobin It may be obtained based on three color channel values.

Here, the second characteristic may be obtained based on a third difference value that is a difference value between the first color channel value and the second color channel value for the first region and the first difference value.

Here, the first color channel value may be a green channel value, the second color channel value may be a red channel value, and the third color channel value may be a blue channel value to reduce noise caused by external light.

Here, the second characteristic may include a frequency component value of time series data obtained based on the first difference value.

Here, the third characteristic may include a pulse transit time (PTT) obtained based on the second difference value and the first difference value.

Here, the fourth characteristic is a fourth difference value that is a difference value between the first color channel value and the third color channel value for the second region, the first difference value, the second difference value, and the third difference value may be obtained based on the value.

Here, the first region and the second region may include a face region of the subject, and a center of the second region and a vertical position of the center of the first region may be different from each other.

According to another embodiment, there is provided a method for simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, the method comprising: obtaining a plurality of image frames for a subject; including in the plurality of image frames setting a first region, a second region, and a third region for at least one image frame to be used, at least two of a first color channel value, a second color channel value, and a third color channel value for the first region determining the oxygen saturation level of the subject based on a first characteristic obtained based on the above color channel values; determining the heart rate of the subject based on a second characteristic obtained based on the difference value; a second difference value that is a difference value between the first color channel value and the second color channel value for the second region; determining the blood pressure of the subject based on a third characteristic obtained based on a third difference value that is a difference value between the first color channel value and the second color channel value for the third region, and the oxygen saturation level , outputting the heart rate and blood pressure, wherein the first characteristic is obtained based on a first image frame group acquired for a first time, and the second characteristic is a second image frame group acquired for a second time the first image frame group, the second characteristic is obtained based on a third image frame group obtained for a third time, and the oxygen saturation, heart rate, and blood pressure are obtained in correlation with each other. The image frame group and the third image frame group may be provided with a biometric index measurement method including a plurality of image frames in common.

Here, the first characteristic is that the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the absorbance of oxyhemoglobin for a third color channel in which the absorbance is higher than that of hemoglobin It may be obtained based on three color channel values.

Here, the second characteristic may be obtained based on a third difference value that is a difference value between the first color channel value and the second color channel value for the first region and the first difference value.

Here, the third characteristic may include a pulse transit time (PTT) obtained based on the second difference value and the first difference value.

According to another embodiment, there is provided a method for simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, the method comprising: obtaining a plurality of image frames for a subject; obtaining a first color channel value, a second color channel value, and a third color channel value for at least one image frame included therein, from among the first color channel value, the second color channel value, and the third color channel value determining the oxygen saturation level of the subject based on a first characteristic obtained based on at least two color channel values, a first difference value that is a difference value between the first color channel value and the second color channel value, and the determining the heart rate of the subject based on a second characteristic obtained based on a second difference value that is a difference value between a first color channel value and the third color channel value; the first difference value and the second difference determining the blood pressure of the subject based on a third characteristic obtained based on a value, and outputting the oxygen saturation, heart rate, and blood pressure, wherein the first characteristic is a first image acquired for a first time is obtained based on a group of frames, the second characteristic is obtained based on a second group of image frames obtained for a second time, and the third characteristic is obtained based on a group of third image frames obtained for a third time and the first image frame group, the second image frame group, and the third image frame group include a plurality of image frames in common so that the oxygen saturation, heart rate, and blood pressure are obtained in correlation with each other. can

Here, the first characteristic is that the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the absorbance of oxyhemoglobin for a third color channel in which the absorbance is higher than that of hemoglobin It may be obtained based on three color channel values.

Here, the second characteristic may include a frequency component value of time series data obtained based on the first difference value and the second difference value.

Here, the third characteristic is a slope component value of time series data obtained based on the first difference value and the second difference value, a maximum value, a minimum value, a maximum value, a minimum value, an average value of the maximum value, and an average value of the minimum value , and may include at least one of a difference value and an average value for the average of the maximum and minimum values.

According to another embodiment, there is provided a method for simultaneously measuring various physiological parameters including heart rate, oxygen saturation, blood pressure and body temperature, the method comprising: obtaining a plurality of image frames for a subject; the plurality of images setting at least two regions for at least one image frame included in the frame, a first color channel value, a second color channel value, and a third color for at least one image frame included in the plurality of image frames obtaining a channel value, a fourth color channel value, and a fifth color channel value; a first color channel value obtained based on at least two color channel values of the first color channel value, a second color channel value, and a third color channel value determining the oxygen saturation level of the subject based on a first characteristic; determining the heart rate of the subject based on a third characteristic obtained based on a color channel value of at least two of the first color channel value, the second color channel value, and the third color channel value. Determining the blood pressure of the measurer, determining the body temperature of the subject based on a fourth characteristic obtained based on the fourth color channel value, and outputting the oxygen saturation, heart rate, body temperature, and blood pressure wherein the first color channel value is a green channel value, the second color channel value is a red channel value, the third color channel value is a blue channel value, and the fourth color channel value is a saturation channel value; The fifth color channel value may be a Hue channel value, and a method for measuring a biometric index may be provided.

Here, the first characteristic is that the second color channel value for a second color channel, which is a channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin, and the absorbance of oxyhemoglobin for a third color channel in which the absorbance is higher than that of hemoglobin It may be obtained based on three color channel values.

Here, the second characteristic is based on a first difference value that is a difference value between the first color channel value and the second color channel value and a second difference value that is a difference value between the first color channel value and the third color channel value. can be obtained by

Here, the second characteristic may include a frequency component value of time series data obtained based on the first difference value and the second difference value.

Here, the third characteristic is based on a first difference value that is a difference value between the first color channel value and the second color channel value and a second difference value that is a difference value between the first color channel value and the third color channel value. can be obtained by

Here, the third characteristic is a slope component value of time series data obtained based on the first difference value and the second difference value, a maximum value, a minimum value, a maximum value, a minimum value, an average value of the maximum value, and an average value of the minimum value , at least one of a difference value and an average value for the average of the maximum and minimum values.

Here, the fourth characteristic may be obtained based on the fourth color channel value and the fifth color channel value.

Here, the fourth characteristic may include the skin temperature of the subject.

According to another embodiment, there is provided a method for simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, the method comprising: obtaining a plurality of image frames for a subject; acquiring N preliminary heart rates based on at least one image frame included therein, acquiring M preliminary oxygen saturation values based on at least one image frame included in the plurality of image frames, and the plurality of image frames obtaining K preliminary blood pressures based on at least one image frame included in Acquiring blood pressure based on K preliminary blood pressures and outputting the heart rate, oxygen saturation, and blood pressure, wherein when the image frame obtained when the subject is in a first state is a first image frame, The first image frame may be provided with a method of measuring a biometric index commonly included in an image frame for obtaining the N preliminary heart rates, the M preliminary oxygen saturation values, and the K blood pressures.

According to another embodiment, there is provided a method of acquiring biometric information using various physiological parameters, the method comprising: acquiring a plurality of image frames for a subject; at least one selected from the plurality of image frames; obtaining a first biometric index based on a first image frame group including an image frame; a second biometric index based on a second image frame group including at least one image frame included in the plurality of image frames Comprising the steps of obtaining, obtaining biometric information based on at least one of the first biometric index and the second biometric index, and outputting the first biometric index, the second biometric index, and the biometric information , The first image frame group and the second image frame group at least partially overlap in order to obtain biometric information by reflecting the specific state of the subject.

Here, the method may further include obtaining personal and statistical data on the subject, wherein the biometric information may be obtained based on the first biometric index, the second biometric index, and the obtained personal and statistical data. .

Here, the first and second biometric indexes may include at least one of heart rate, oxygen saturation, blood pressure, and body temperature, and the biometric information may include at least one of drowsiness information, stress information, excitability information, and emotion information. .

According to an embodiment, the method for detecting drowsiness based on a heart rate may be performed by at least one processor, comprising: obtaining a heart rate of an object; obtaining a comparison result by comparing the heart rate with a reference heart rate; Comparing the heart rate with the reference heart rate to obtain a comparison result, obtaining a duration in which the heart rate is equal to or less than the reference heart rate based on the comparison result, and determining whether the duration reaches the reference duration and a drowsiness detection step of determining the drowsiness state of the subject based on the reference duration. The reference duration includes a first reference duration, a second reference duration, and a third reference duration, and the second reference duration is is longer than the first reference duration, the third reference duration is longer than the second reference duration, and the drowsiness state of the subject includes a normal state, a first drowsiness state, a second drowsiness state, and a third drowsiness state and, the first drowsiness state indicates a state in which the subject is more likely to enter the sleep state than the normal state, and the second drowsiness state indicates a higher probability that the subject will enter the sleep state than the first drowsiness state. state, and the third drowsiness state may indicate a state in which the subject is more likely to enter the sleep state than the second drowsiness state.

Here, the step of detecting drowsiness may include determining that the drowsiness state of the subject is the first drowsiness state when the duration is longer than the first reference duration and shorter than the second reference duration, the duration determining that the drowsiness state of the subject is the second drowsiness state when the time is longer than the second reference duration and shorter than the third reference duration; if the duration is longer than the third reference duration, the drowsiness determining that the state is the second drowsy state.

Here, the drowsiness detection step is a step of determining the drowsiness state of the subject as the normal state when the duration for which the heart rate of the object is maintained at or below the reference heart rate is shorter than the first reference duration. may include more.

Here, the first drowsiness state may indicate a state in which the subject is not aware of drowsiness but is physically likely to enter the sleep state.

Here, the heart rate of the object may represent an average value of a plurality of heart rates in a predetermined time interval including a time point at which the heart rate is acquired.

Here, when the drowsiness state of the subject is any one of the first drowsiness state to the third drowsiness state, acquiring a recovery duration in which the heart rate is greater than or equal to the reference heart rate, the recovery duration is The method may further include a recovery detection step of determining the drowsiness state of the subject based on whether a recovery reference duration is reached.

Here, the recovery reference duration includes a first recovery reference duration, a second recovery reference duration, and a third recovery reference duration, and the recovery detection step includes: In this case, if the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the drowsiness state of the subject is determined as the second drowsiness state, and the recovery duration is the second recovery duration If it is longer than a reference duration and shorter than the third recovery reference duration, the drowsiness state of the subject is determined as the first drowsiness state, and if the recovery duration is longer than the third recovery reference duration, the drowsiness of the subject If the state is determined as the normal state, and when the drowsiness state of the subject is the second drowsiness state, the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration When the drowsiness state is determined as the first drowsiness state, and the recovery duration is longer than the second recovery reference duration and shorter than the third recovery reference duration, the drowsiness status of the subject is determined as the normal state, and the When the drowsiness state of the subject is the first drowsiness state, if the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the drowsiness status of the subject may be determined as the normal state .

wherein one of the first reference duration, the second reference duration, and the third reference duration is at least one of the first recovery reference duration, the second recovery reference duration, and the third recovery reference duration can be the same as

Here, the first reference duration, the second reference duration, and the third reference duration may all be different from the first recovery reference duration, the second recovery reference duration, and the third recovery reference duration. have.

According to an embodiment, a method for detecting drowsiness based on a heart rate and LF/HF performed by at least one processor includes obtaining a heart rate with respect to an object, comparing the heart rate with a reference heart rate to obtain a comparison result step, obtaining a duration for which the heart rate is changed based on the comparison result, obtaining a first drowsiness parameter (drowsiness step determined based on the heart rate) based on the duration, the step; obtaining a LF/HF (Low Frequency/High Frequency) value of the object representing the ratio of the sympathetic nerve activity and the parasympathetic nerve activity of the object; obtaining a second drowsiness parameter based on the LF/HF value of the object and determining a sleepiness state of the object using at least one of the first sleepiness parameter and the second sleepiness parameter.

Here, the acquiring of the duration may include acquiring the duration based on a length of a time period in which the heart rate is equal to or less than the reference heart rate based on the comparison result.

Here, the drowsiness state includes a first drowsiness state and a second drowsiness state, and the second drowsiness state is a state in which the subject is more likely to enter the sleep state than the first drowsiness state, and the drowsiness state is the A first drowsiness parameter obtained based on a comparison result between the duration and the first reference duration and the second reference duration, and LF/HF of the subject, obtained based on a comparison result between the first reference value and the second reference value is determined according to a second drowsiness parameter, wherein the drowsiness state is determined as the first drowsiness state when the duration is greater than the first reference duration and LF/HF of the subject is less than the first reference value, the drowsiness The state may be determined as the second drowsy state when the duration is greater than the second reference interval or the LF/HF of the object is less than the second reference value.

Here, the drowsiness state further includes a third drowsiness state, and the first drowsiness state is a state in which the subject is not aware of drowsiness but is physically likely to enter a sleep state, and the second drowsiness state and the second drowsiness state 3 The drowsiness state may be a state in which the person is aware of drowsiness and has a possibility of entering the sleep state physically, and the third drowsiness state may be a state in which the possibility of entering the sleep state is higher than that of the second drowsiness state.

Here, the drowsiness state includes a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state, and the first stage drowsiness state is a possibility that the subject enters a sleep state rather than the normal state this high state, the second stage drowsiness state represents a state in which the subject is more likely to enter the sleep state than the first stage drowsiness state, and the third stage drowsiness state is higher than the second stage drowsiness state indicates a state in which the subject is highly likely to enter the sleep state, and as the values of the first sleepiness parameter and the second sleepiness parameter are larger, the state in which the object is more likely to enter the sleep state is indicated, When the value of the drowsiness parameter and the value of the second drowsiness parameter match, a matching value is obtained, and based on the coincident value, the drowsiness state is set to a normal state, a first stage drowsiness state, a second stage drowsiness state, and a second stage drowsiness state and a second stage drowsiness state. It can be determined as one of three levels of drowsiness.

Here, the drowsiness state includes a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state, and the first stage drowsiness state is a possibility that the subject enters a sleep state rather than the normal state this high state, the second stage drowsiness state represents a state in which the subject is more likely to enter the sleep state than the first stage drowsiness state, and the third stage drowsiness state is higher than the second stage drowsiness state indicates a state in which the subject is highly likely to enter the sleep state, and as the values of the first sleepiness parameter and the second sleepiness parameter are larger, the state in which the object is more likely to enter the sleep state is indicated, When the value of the drowsiness parameter does not match the value of the second drowsiness parameter, a larger value of the value of the first drowsiness parameter and the value of the second drowsiness parameter is obtained, and based on the larger value, the drowsiness is obtained. The state may be determined as one of a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state.

Here, the drowsiness state includes a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state, and the first stage drowsiness state is a possibility that the subject enters a sleep state rather than the normal state this high state, the second stage drowsiness state represents a state in which the subject is more likely to enter the sleep state than the first stage drowsiness state, and the third stage drowsiness state is higher than the second stage drowsiness state indicates a state in which the subject is highly likely to enter the sleep state, and as the values of the first sleepiness parameter and the second sleepiness parameter are larger, the state in which the object is more likely to enter the sleep state is indicated, When the value of the drowsiness parameter does not match the value of the second drowsiness parameter, a smaller value of the value of the first drowsiness parameter and the value of the second drowsiness parameter is obtained, and based on the smaller value, the drowsiness is obtained. The state may be determined as one of a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state.

Here, the drowsiness state includes a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state, and the first stage drowsiness state is a possibility that the subject enters a sleep state rather than the normal state this high state, the second stage drowsiness state represents a state in which the subject is more likely to enter the sleep state than the first stage drowsiness state, and the third stage drowsiness state is higher than the second stage drowsiness state indicates a state in which the subject is highly likely to enter the sleep state, and as the values of the first sleepiness parameter and the second sleepiness parameter are larger, the state in which the object is more likely to enter the sleep state is indicated, When the value of the drowsiness parameter does not match the value of the second drowsiness parameter, an average value of the value of the first drowsiness parameter and the value of the second drowsiness parameter is obtained, and based on the average value, the drowsiness state is set to a steady state , one of the first stage drowsiness state, the second stage drowsiness state, and the third stage drowsiness state.

Here, the method may be performed through a recording medium in which a program for performing the method is recorded.

According to another embodiment, as a smart mirror device that simultaneously displays at least two biometric indexes among heart rate, oxygen saturation, blood pressure, and body temperature, a reflective mirror surface and a plurality of image frames for the subject are acquired an image acquisition unit for performing an image acquisition unit, disposed behind the reflective mirror surface, a display unit for displaying visual information through the reflective mirror surface, and a control unit for controlling the operation of the image acquisition unit and the display unit to obtain a biometric index in a non-contact manner including, wherein the controller controls the display unit to display a first biometric index obtained based on a first image frame group included in the plurality of image frames at a first time point, and at a second time point, the plurality of images The display unit is controlled to display a second biometric index obtained based on a second image frame group included in a frame, and the first image frame group and the second biometric index are correlated with each other; The second image frame group includes at least one image frame in common, and the at least one image frame included in the first image frame group and the second image frame group in common before the first and second time points There may be provided a smart mirror device including an image frame obtained in the first state of the subject to be observed through the reflective mirror surface to the subject.

Here, the first and second biometric indexes may include at least one of heart rate, oxygen saturation, blood pressure, and body temperature.

Here, the first image frame group and the second image frame group may be identical to each other.

Here, the first image frame group and the second image frame group may be different from each other.

Here, the first time point and the second time point may be the same time point.

Here, the first time point is a time point preceding the second time point, and the at least one image frame commonly included in the first image frame group and the second image frame group is the reflection mirror surface before the first time point may include an image frame obtained in the second state of the subject observed by the subject through .

Here, the controller controls the display unit to display a third biometric index obtained based on a third image frame group included in the plurality of image frames at a third time point, and the first, second, and third biometrics The first, second, and third image frame groups commonly include at least one image frame, and at least one of the first, second, and third image frame groups commonly included in the index so that the index is correlated. The image frame may include an image frame obtained in a first state of the subject to be observed through the reflective mirror surface before the first, second, and third time points.

Here, the controller acquires the first and second biometric indexes based on at least some of the image frames among the plurality of image frames for a first time period, and at least one image frame among the plurality of image frames for a second time period recognizes the subject based on , the second time is shorter than the first time, the second time is included in the first time, Information about the subject can be displayed.

Here, the information on the subject may be the first and second biometric indexes of the subject that have been previously measured and stored.

Here, the control unit controls the display unit to display first information at a fourth time point, and controls the display to display second information at a fifth time point, wherein the first information includes weather information and time information, , the second information may include information on the subject, the fourth time point may be a time point prior to the fifth time point, and the fifth time point may be a time point preceding the first and second time points.

Here, the second information may include at least one of schedule information, medication information, recognition information, messenger information, and interest information of the subject.

Here, the controller controls the display unit to display the first information at the fourth time point, controls the display unit to display the first information and the second information at the fifth time point, and controls the display unit at the second time point The display unit may be controlled to display the first information, the second information, the first biometric index, and the second biometric index.

Here, the first biometric index and the second biometric index are obtained based on at least three color channel values for the plurality of image frames, and the first image frame included in the first image frame group is obtained. Both the first biometric index and the second biometric index may be displayed within 10 seconds from a point in time.

Here, when an image of a plurality of persons is included in the image frame obtained through the image obtaining unit, the control unit may include the first and second images of one of the plurality of persons selected according to priority. The display unit may be controlled such that a biometric index is obtained, the obtained first and second biometric indexes are displayed, and information on a selected one of the plurality of persons is displayed.

In this case, when an image frame including the first subject and the second subject is obtained after the image frame including the first subject is obtained, the control unit may be configured to obtain the second subject based on the image of the first subject The display unit may be controlled to obtain first and second biometric indexes, and to display information on the first subject.

Here, the controller may determine the priority of the plurality of people based on the obtained image frame.

Here, when the image of the first subject and the second subject are included in the image frame acquired through the image acquisition unit, the controller is configured to control the first subject based on a fourth image frame group included in the plurality of image frames. obtaining the first biometric index for the subject, and obtaining the second biometric index for the first subject based on a fifth image frame group included in the plurality of image frames, wherein the plurality of image frames obtaining the first biometric index for the second subject based on a sixth image frame group included in and obtain the second biometric index, wherein the fourth and fifth image frame groups include at least one image frame in common, and the sixth and seventh image frame groups have at least one image frame in common. may include

Here, the control unit acquires biometric information based on the first biometric index, and controls the display unit to display the biometric information at a third time point, and displays the first biometric index and the biometric information in real time. The biometric information displayed at the third time point may be obtained based on the first biometric index displayed at the first time point, and the first time point may be a time prior to the third time point.

Here, the controller acquires the biometric information based on the first and second biometric indexes, and the biometric information displayed at the third time point is displayed at the first time point in order to improve the accuracy of the biometric information It is obtained based on the first biometric index and the second biometric index displayed at the second time point, and the third time point may be later than the first and second time points.

Here, the biometric information may include at least one or more biometric information of condition information, concentration information, drowsiness information, and emotion information.

Here, the control unit obtains biometric information based on the first biometric index, and controls the display unit to display the biometric information at a third time point, but the display unit is displayed at the third time point to display accurate biometric information The biometric information is obtained based on the first biometric index displayed up to the third time point, and the third time point may be later than the first time point.

Here, the biometric information displayed at the third time point may be obtained based on an average value of the first biometric index displayed up to the third time point.

Here, the controller obtains the biometric information based on the first and second biometric indices, and the biometric information displayed at the third time point includes the first biometric index and the third biometric information displayed up to the third time point. It is obtained based on the second biometric index displayed up to a time point, and the third time point may be later than the first and second time points.

Here, the period in which the first bio-index is updated and the period in which the second bio-index is updated may be different from each other.

Here, the control unit determines the information provision situation, controls the display unit to display first information in case of a first situation, and controls the display unit to display second information in case of a second situation, The information may include the first and second biometric indexes, but the second information may not include the first and second biometric indexes.

Here, the control unit may determine the information provision situation based on the moving direction of the subject.

Here, the smart mirror device may further include a motion detection sensor for detecting the moving direction of the subject.

Here, the controller may determine the information provision situation based on the plurality of obtained image frames.

Here, the first information includes at least one of external weather information, schedule information of the subject, and time information, and the second information includes internal temperature information, internal humidity information, internal air information, security information, and activity It may include at least one piece of time information.

Here, the first information may include at least two or more biometric indices, and the second information may include at least one of weather information, time information, news information, schedule information, and medication information.

Here, the controller obtains at least three color channel values for at least one image frame included in the plurality of image frames, and calculates a first difference value and a second difference value based on the at least three color channel values. and obtain the first biometric index and the second biometric index based on the first difference value and the second difference value.

According to another embodiment, there is provided an operating method of a smart mirror device that simultaneously displays at least two biometric indexes among heart rate, oxygen saturation, blood pressure, and body temperature, the method comprising: acquiring an on-trigger , obtaining a plurality of image frames for the subject, obtaining a first biometric index based on a first image frame group included in the plurality of image frames, a second image included in the plurality of image frames obtaining a second biometric index based on a frame group, displaying the first and second biometric indexes, obtaining an off-trigger, and obtaining a plurality of image frames for the subject stopping, wherein the first image frame group and the second image frame group include at least one image frame in common so that the first biometric index and the second biometric index are associated with each other, An on-trigger is acquired from at least one sensor, and the off-trigger is acquired from an image sensor for acquiring the plurality of image frames.

Here, the on-trigger may be obtained from at least one of a motion detection sensor, a touch sensor, a mouse, and a keyboard.

According to another embodiment, as a smart mirror device for displaying at least one of a biometric index of heart rate, oxygen saturation, blood pressure, and body temperature, an image acquisition unit for acquiring a reflective mirror surface and a plurality of image frames , a display unit disposed behind the reflective mirror surface and displaying visual information passing through the reflective mirror surface, disposed in front of the image acquisition unit, an opening/closing unit for opening and closing a field of view of the image acquisition unit, and the image acquisition unit and the display a control unit for controlling negative operation and obtaining a biometric index; Including, wherein the surface of the opening/closing unit is formed of a reflective mirror, and when the opening/closing unit is in an open state, the control unit controls the display unit so that the biometric index and at least one or more visual information are displayed through the display unit, and the opening/closing unit is closed In this case, a smart mirror device may be provided in which the controller controls the display unit so that at least one piece of visual information is displayed through the display unit.

Here, the control unit controls the image acquisition unit to acquire an image frame from the image acquisition unit when the opening and closing unit changes from the closed state to the open state, and acquires the image when the opening and closing unit changes from the open state to the closed state The image acquisition unit may be controlled so that the unit does not acquire an image frame.

0. Definition of terms

As used herein, the term 'measurement' may be understood as a concept including all of measuring, determining by guessing, and measuring the size of other quantities based on a certain amount.

As used herein, 'heart rate' may be understood as a heart rate, which is a result of the heart rate and the concept of 'heart rate', which may mean the number of beats measured near the heart as a result of the heart rate. It can be understood as a concept including all concepts of 'pulse', which may mean that the generated vibration is propagated to peripheral blood vessels.

As used herein, 'blood pressure' may be understood as the pressure generated in a blood vessel when the heart pushes blood, which is a value that can be understood as a normal 'blood pressure' regardless of the measurement site (eg, For example, it can be understood as a value that can be inferred as a value measured in an artery of the upper arm).

As used herein, 'oxygen saturation' may be understood as a degree of saturation of oxygen in blood, and more specifically, may be understood as a fraction of oxygen hemoglobin to total hemoglobin in blood.

As used herein, 'Core Temperature' may be understood as the body temperature of a person or animal, and may be understood differently from 'Skin Temperature' that may be measured from the skin.

As used herein, 'skin temperature' may be understood as the measured surface temperature of the skin.

As used herein, an 'image' may be understood as a concept including one image or a plurality of images included in an image.

As used herein, a 'color channel' may be understood as each axis constituting a color space, for example, a red channel, a green channel, and a blue channel constituting an RGB color space. may mean a red axis, a green axis, and a blue axis, and these color channels may be configured in 2D, 3D, or 4D.

As used herein, the 'image of the subject' may be understood as an image including the position of the subject to be measured. For example, when the position of measurement is the face of the subject, the face region of the subject may be It can be understood as an image containing

As used herein, 'personal and statistical data' may mean collectible personal and statistical data of the subject, such as age, gender, height, weight, etc. It may mean data, and statistical data calculated for a group that includes or is related to the subject (e.g., average blood pressure in their 20s, average skin color of yellow people, average height of men in their 30s, average weight of men in Korea) etc.) can also mean personal and statistical data that can be quantified.

As used herein, 'time series data' may mean data listed along a time axis, but is not limited thereto, and may mean data listed along an image frame axis that may correspond to time, and may mean data listed along a time axis or an image frame axis It may mean data that can be sorted according to

1. Biometric index and biometric information management system

"Physical parameter" may mean a result according to the physiological activity of the human body that can be measured or estimated, and may include, for example, heart rate, oxygen saturation, blood pressure, body temperature, blood flow, etc. It is not limited, and may mean other results according to physiological activities of the human body that can be measured or estimated.

"Physical information" may refer to information that can be calculated in consideration of at least some of results according to physiological activities of the human body, such as biometric indexes, and personal and statistical data such as facial expressions, postures, and age, for example, , drowsiness information, stress information, excitement information, emotion information, and the like, but is not limited thereto.

A biometric index and biometric information management system may refer to a management system that can store, share, or analyze biometric information that can be measured or estimated and calculated biometric information to comprehensively manage personal health, etc. have.

1 and 2 are diagrams of a biometric index and biometric information management system according to an embodiment, respectively.

Referring to FIG. 1 , a biometric index and biometric information management system 100 according to an embodiment may include a biometric index acquisition device 10 and a server 20 .

In this case, the biometric index acquisition device 10 may measure a physiological parameter of the subject. More specifically, the biometric index acquisition device 10 may invasively measure the biometric index of the subject, may be non-invasively, may be measured in a contact method, or may be measured in a non-contact manner have.

For example, the biometric index acquisition device 10 may measure a biometric index such as a heart rate, oxygen saturation, and blood pressure of the subject by analyzing the image or image of the subject, but is not limited thereto.

Also, the biometric index acquisition device 10 may calculate physiological information based on the measured biometric index. More specifically, the biometric index acquisition device 10 may calculate biometric information based on the measured biometric index, and comprehensively considers the measured biometric index and personal and statistical data such as facial expression, posture, age, etc. can also be calculated.

For example, the biometric index acquisition device 10 may calculate biometric information, such as emotional information and drowsiness information, of the subject based on the measured biometric indices such as heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

Also, the biometric index acquisition device 10 may store the measured biometric index and the calculated biometric information. For example, the biometric index acquisition device 10 may store the measured biometric index and calculated biometric information of the subject in an internal memory, but is not limited thereto.

Also, the biometric index acquisition device 10 may display the measured biometric index and the calculated biometric information. For example, the biometric index acquisition device 10 may further include a display, and may display the measured biometric index and calculated biometric information for the subject using the display, but is not limited thereto. , the corresponding information may be transmitted to be displayed on an external display.

In addition, the biometric index may be displayed once through the display, and the biometric index changing in real time may be continuously displayed.

Also, the biometric index acquisition device 10 may transmit the biometric index and the biometric information for the subject to the server 20 .

In this case, the biometric index and the biometric information transmitted toward the server 20 may be stored in the server 20 as personal data. For example, the biometric index measured from the subject A and the calculated biometric information may be stored in the server 20 as data for the subject A, and the biometric index measured from the subject B and calculated biometric information may be stored in the server 20 as data for the subject B.

In addition, the server 20 may transmit the biometric index and biometric information for the subject by communicating with an external terminal when necessary. For example, when a person named A, whose biometric index and biometric information data is stored in the server 20, visits a hospital to receive medical treatment, a doctor to treat the subject A is a biometric index and Biometric information may be required. At this time, when the server 20 receives a request for transmission of the biometric index and biometric information data for the subject A from the external terminal disposed in the hospital, the server 20 communicates with the external terminal to obtain the biometric index and the biometric data for the subject A Information data can be transmitted.

As such, the biometric index and biometric information management system 100 can serve as a basis for providing a continuous and comprehensive management service for personal health, and the biometric index and biometric information management system 100 as well as the above-described examples ) is a self-evident fact that can serve as a basis for providing various comprehensive management services using biometric indexes and biometric information that are continuously measured, stored and managed.

Also, referring to FIG. 2 , the biometric index and biometric information management system 100 according to an embodiment may include an image acquisition device 30 and a server 20 .

In this case, the image acquisition device 30 may acquire an image or an image of the subject.

Also, the server 20 may acquire an image or an image of the subject from the image acquisition device 30 .

Also, the server 20 may obtain personal and statistical data about the subject from an input device, an external terminal, or the like.

Also, the server 20 may measure the biometric index of the subject based on the acquired image or image. For example, the server 20 may analyze the acquired image or image to measure a biometric index, such as a heart rate, oxygen saturation, blood pressure, etc. of the subject, but is not limited thereto.

Also, the server 20 may calculate biometric information based on the measured biometric index. More specifically, the server 20 may calculate biometric information based on the measured biometric index, and calculate biometric information by comprehensively considering the measured biometric index and personal and statistical data such as facial expression, posture, and age. can do.

For example, the server 20 may calculate biometric information, such as emotional information and drowsiness information, of the subject based on the measured biometric indices such as heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

Also, the server 20 may store the measured biometric index and the calculated biometric information.

In addition, it is clear that the biometric index and biometric information management system 100 including the image acquisition device 30 and the server 20 can perform the functions of the biometric index and biometric information management system described above in FIG. 1 . Therefore, duplicate descriptions will be omitted.

2. Various embodiments of biometric index acquisition device

3 is a diagram for explaining a biometric index obtaining apparatus according to an exemplary embodiment.

Referring to FIG. 3 , the apparatus 1000 for obtaining a biometric index according to an embodiment may include an image obtaining unit 1010 , a control unit 1020 , a storage unit 1030 , and a communication unit 1040 . However, the biometric index obtaining apparatus 1000 may include only at least a portion of the image obtaining unit 1010 , the control unit 1020 , the storage unit 1030 , and the communication unit 1040 . For example, the biometric index acquisition apparatus 1000 may include only the image acquisition unit 1010 and the control unit 1020 , but is not limited thereto and may be implemented in various ways.

Also, the image acquisition unit 1010 may acquire an image or an image of the subject. More specifically, the image acquisition unit 1010 may include a photographing device, and may acquire an image or an image of the subject by using the photographing device, or may be disposed outside the biometric index acquisition apparatus 1000 . An image or an image of the subject may be acquired from a photographing device, but the present invention is not limited thereto.

In addition, when the image acquisition unit 1010 acquires an image or an image of the subject from the imaging device, the imaging device includes a visible light camera for acquiring a visible light image, and an infrared camera for acquiring an infrared image. (IR camera) or the like may be provided, but is not limited thereto, and a hybrid type camera for acquiring a visible light image and an infrared image may be provided.

Also, when the photographing apparatus obtains a visible light image, the obtained visible light image may be obtained with at least one or more color channel values. For example, the obtained visible light image may be obtained as color channel values of an RGB color space expressed as red, green, and blue, and may include Hue, Saturation, It may be obtained as a color channel value of the HSV color space expressed as value, but is not limited thereto, and may be obtained as a color channel value of various color spaces such as YCrCb and YiQ.

Also, when the photographing device acquires an infrared image, the photographing device may acquire an infrared image through an infrared light disposed inside or outside the photographing apparatus. In this case, the infrared illumination may irradiate infrared rays of a near infrared region, which is a wavelength band of 750 nm to 3000 nm, but is not limited thereto, and a middle infrared region, a far infrared region and an extreme infrared ray region. Infrared rays in the extreme infrared can also be irradiated.

Also, the controller 1020 may acquire a biometric index by using the image of the subject acquired from the image acquisition unit 1010 .

For example, the control unit 1020 may analyze the image of the subject obtained from the image acquisition unit 1010 to obtain the bio-indexes such as heart rate, oxygen saturation, blood pressure, and body temperature of the subject, but this It is not limited, and various biometric indices can be obtained.

Also, the controller 1020 may calculate biometric information based on the obtained biometric index.

For example, the control unit 1020 may calculate biometric information such as emotional information and drowsiness information of the subject based on the obtained biometric indices such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto. information can be calculated.

Also, the controller 1020 may control operations of at least some of the image acquisition unit 1010 , the storage unit 1030 , and the communication unit 1040 .

Also, the storage unit 1030 may store the biometric index and biometric information obtained by the controller 1020 . More specifically, the storage unit 1030 may store the biometric index and biometric information for one subject, and may store the biometric index and biometric information for several subjects, respectively.

Also, the communication unit 1040 may transmit the biometric index and biometric information obtained by the controller 1020 . More specifically, the communication unit 1040 may transmit the biometric index and biometric information obtained by the controller 1020 to the management server or to the user's terminal.

3. Various embodiments of biometric index and biometric information acquisition method

3.1 How to obtain a biometric index

4 is a flowchart illustrating a method for obtaining a biometric index according to an exemplary embodiment.

Referring to FIG. 4 , the method 1100 for obtaining a biometric index according to an embodiment may include obtaining an image of the subject for the subject ( S1110 ).

At this time, since it has been described above that an image can be obtained by using various cameras such as a visible light camera and an infrared camera to obtain an image of the subject, or images can be obtained from various cameras, a detailed description will be omitted. .

Also, the method 1100 for obtaining a biometric index according to an embodiment may include detecting a skin region ( S1120 ).

In this case, the skin region may mean a region that can be estimated as the skin region of the subject in the image of the subject.

In addition, the skin region may reflect changes in blood vessels due to heartbeat, and detecting the skin region in this way may improve the accuracy of obtaining the biometric index.

In addition, according to an embodiment, in order to detect the skin region, an area other than the skin such as eyes and hair of the person to be measured may be removed except for a skin region capable of reflecting a color change due to the expansion of blood vessels. For example, a color value of an area other than the skin, such as the eyes and hair of the subject, may be substituted with a meaningless value such as black, but is not limited thereto.

Also, according to an embodiment, a specific color space may be used to detect the skin region. For example, in the step of detecting the skin region (S1120), the obtained image of the subject is replaced with a value of the YCrCb color space, and the skin region is detected based on the image expressed in the YCrCb color space. However, the present invention is not limited thereto.

In addition, it is apparent that the skin region can be detected using various conventionally known techniques for detecting the skin region.

Also, the method 1110 for obtaining a biometric index according to an embodiment may include setting a region of interest ( S1130 ).

In this case, the region of interest may mean a region of interest for data processing among the acquired images of the subject, and may mean a region that can be used for data processing to obtain a biometric index, not limited

Also, according to an embodiment, in order to set the ROI, a face region of the subject may be set. For example, the face region of the subject may be set in order to set a region of interest included in the face of the subject.

More specifically, a region having a predetermined ratio based on the set center of the face region of the subject may be set as the ROI.

For example, 80% vertical and 60% horizontal cropping may be performed based on the center of the face region of the subject, but is not limited thereto.

Also, according to an embodiment, a feature point may be used to set the region of interest. For example, from the acquired image of the subject, a nose region of the subject may be extracted as a feature point, and a region of interest may be set based on the extracted feature point, but is not limited thereto.

More specifically, a region having a predetermined size based on a feature point extracted with respect to the set face region may be set as the ROI, but is not limited thereto.

Also, according to an embodiment, a plurality of specific points may be used to set the ROI. For example, from the acquired image of the subject, the eye and nose regions of the subject may be extracted as feature points, and a region of interest may be set based on the extracted feature points.

In addition, when a region of interest is set for each of a plurality of consecutively acquired images, the region of interest for each of the plurality of images may be independently set and may be set to be related.

For example, in order to set the ROI for each of the plurality of images to be related, a face region of the subject for each of the plurality of images may be set, and the center of the face region set in the first image frame and the first image If the difference between the centers of the face regions set in the second image frame acquired after the frame does not exceed the threshold, the face region of the second image frame may be set to be the same as the face region set in the first image frame. not limited

In addition, when the difference between the center of the face region set in the first image frame and the center of the face region set in the second image frame exceeds a threshold, the face region of the second image frame is separated from the face region set in the first image frame. It may be set to be different, but is not limited thereto.

In addition, the region of interest according to an embodiment may be set to include a part of a body part or a part of a face according to a desired biometric index, and at least one region of interest may be set.

For example, the region of interest may be set to include at least a part of the cheek region in order to acquire a heart rate among the biometric indexes. More specifically, the region of interest may be set to include a cheek region in which a heart rate can be easily obtained because the degree of expansion of blood vessels can be well reflected according to blood flow, but is not limited thereto.

In addition, for example, at least two regions of interest may be set to obtain blood pressure among the biometric indexes, and more specifically, the region of interest may include an upper region of the face to reflect blood flow. It may be set as a region of interest including a region and a lower region of the face, but is not limited thereto.

In addition, for example, at least two regions of interest may be set to obtain blood pressure among the biometric indexes, and more specifically, two or more regions of interest may be set as two or more regions of interest having different distances from the heart. . For example, the region of interest may be set as a region of interest including a hand region and a region of interest including a face region, but is not limited thereto.

Also, the method 1110 for obtaining a biometric index according to an exemplary embodiment may include processing data on an ROI ( S1140 ).

In addition, a color channel value for the ROI may be extracted to process the data for the ROI. In this case, the color channel value may be an average value of color channel pixel values of pixels included in the ROI, and may also be referred to as an average pixel value.

For example, when a color channel value according to an RGB color space is extracted, a red channel pixel value, a green channel pixel value, and a blue channel pixel value of each pixel included in the ROI may be extracted, and the ROI may be A red channel value that is an average of included red channel pixel values, a blue channel value that is an average of blue channel pixel values, and a green channel value that is an average of green channel pixel values may be extracted, but is not limited thereto, and the HSV, YCrCb color space is not limited thereto. Color channel values according to various color spaces, such as, may be extracted.

Also, a color channel value extracted according to a specific color space may be converted into another color space. For example, a color channel value extracted according to an RGB color space may be converted into a color channel value according to various color spaces such as HSV and YCrCb color space.

In addition, the color channel value extracted to process the data for the ROI may be a color channel value combined by applying a weight to at least some of the color channel values extracted according to various color spaces.

In addition, the color channel value may be extracted for each of a plurality of continuously acquired image frames, or may be extracted for at least some of the image frames.

In addition, the color channel values extracted from one image frame may be processed through calculation or the like. More specifically, a plurality of channel values obtained from one image frame can be processed through operations such as adding or subtracting from each other. may be processed through, but is not limited thereto, and various channel values may be processed through various operations.

In addition, color channel values or processed values extracted from each of the plurality of image frames may be processed through calculation or the like. More specifically, the color channel values extracted from each of the plurality of image frames or the processed values of the color channel values may be processed through calculations such as obtaining an average of a predetermined interval or a deviation, but is not limited thereto, and is not limited thereto, and is not limited to a predetermined interval. It is also possible to obtain the difference between the maximum and minimum values during the period, and can be processed through various calculations.

In addition, color channel values extracted from each of the plurality of image frames and processed values of the color channel values may be processed to obtain at least one time series data.

Also, a feature value for obtaining a biometric index may be extracted based on at least some of color channel values, processing values, and time series data. For example, a frequency component for obtaining a heart rate may be extracted based on a frequency component of the time series data, but is not limited thereto.

Also, the method 1110 for obtaining a biometric index according to an embodiment may include obtaining a biometric index ( S1150 ).

In addition, a feature value extracted from the region of interest may be used to obtain the biometric index. For example, the heart rate may be acquired based on a frequency component of time series data extracted from the ROI, but is not limited thereto.

In addition, since the step of processing the data on the region of interest ( S1140 ) and the step of obtaining the biometric index ( S1150 ) may be different for each biometric index, more detailed information will be described in detail in the corresponding part.

5 is a diagram illustrating a method for obtaining a biometric index according to an exemplary embodiment.

Referring to FIG. 5 , an image 1161 of a subject may include a face region, detect a skin region ( 1162 ), and detect a region of interest ( 1163 ). A detailed description thereof will be omitted, as has been described above, and overlapping descriptions will be omitted.

Also, referring to FIG. 5 , a color channel value for a region of interest may be extracted ( 1164 ), and the extracted color channel value may be processed ( 1165 ), and a biometric index may be obtained based on this ( 1166 ).

However, this will be described in detail in the relevant section below.

3.2 How to obtain biometric information

6 is a flowchart illustrating a method for acquiring biometric information according to an embodiment.

Referring to FIG. 6 , the method for obtaining biometric information according to an embodiment may include obtaining a biometric index ( S1210 ).

In this case, the biometric index may be obtained by the above-described method for obtaining the biometric index, but is not limited thereto, and may be obtained by an external sensor such as an ECG sensor.

In addition, the redundant description of the bio-index will be omitted.

Also, the method for obtaining biometric information according to an embodiment may include obtaining personal and statistical data ( S1220 ).

In this case, the personal and statistical data may mean collectible personal and statistical data of the subject, such as age and gender, and may mean observable personal and statistical data such as facial expressions and wrinkles of the subject, but is not limited thereto. It may be a variety of personal and statistical data except for biometrics for obtaining biometric information without it.

Also, the method for obtaining biometric information according to an embodiment may include obtaining biometric information ( S1230 ).

In this case, the biometric information may be drowsiness information or emotional information of the subject, but is not limited thereto.

In addition, the biometric index may be used to obtain the biometric information. For example, a heart rate among the biometric index may be used to obtain the drowsiness information. More specifically, when the heart rate of the person to be measured falls below the reference heart rate, the measured person may be considered to be in a state of drowsiness, and the degree of drowsiness of the person may be obtained according to the time at which the target heart rate is lower than the reference heart rate.

Also, for example, heart rate and blood pressure among the biometric indexes may be used to obtain the emotional information. More specifically, when the heart rate and blood pressure of the subject are equal to or greater than the reference heart rate and blood pressure, the subject may be considered to be in an excited state.

In addition, the biometric index and the personal and statistical data may be used to obtain the biometric information. For example, personal and statistical data such as a heart rate and an age and gender of a person to be measured may be used to obtain the drowsiness information.

Also, for example, personal and statistical data such as a heart rate, blood pressure, and a person's expression, age, and gender may be used to obtain the emotional information.

In addition, weights may be assigned to the biometric index and the personal and statistical data to obtain the biometric information. For example, in order to obtain the biometric information, different weights may be given to the biometric index and the personal and statistical data, and different weights may be assigned to each subject.

3.3 Method of obtaining biometric index using biometric index acquisition model

7 and 8 are diagrams for explaining a method for obtaining a biometric index using a biometric index acquisition model.

7A illustrates a method for obtaining a biometric index 1300 using the biometric index obtaining model 1302 according to an embodiment.

In this case, the biometric index acquisition model 1302 according to an embodiment may be implemented by a machine learning method. For example, the biometric index acquisition model 1302 may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, or the like.

In addition, the biometric index acquisition model 1302 according to an embodiment may be implemented as an artificial neural network (ANN). For example, the biometric index acquisition model 1302 may be implemented as a feedforward neural network, a radial basis function network, or a kohonen self-organizing network, etc. not limited

In addition, the biometric index acquisition model 1302 according to an embodiment may be implemented as a deep neural network (DNN). For example, the biometric index acquisition model 1302 is a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM), or Gated Recurrent Units (GRUs). and the like, but is not limited thereto.

Also, the image 1301 input to the biometric index acquisition model 1302 may be acquired image data itself.

Also, the image 1302 input to the biometric index acquisition model 1302 may be pre-processed image data. For example, the image 1302 may be subjected to eulerian video magnification, but is not limited thereto, and may be subjected to various pre-processing such as obtaining an average value of the obtained RGB values.

In addition, the obtained biometric index 1303 may be a heart rate, oxygen saturation, blood pressure, body temperature, or the like.

In addition, the obtained biometric index 1303 may be one, and a plurality of biometric indexes may be simultaneously obtained. For example, a heart rate may be obtained as a result of the biometric index acquisition model 1302 , but the present invention is not limited thereto, and a heart rate and blood pressure may be simultaneously obtained as a result of the biometric index acquisition model 1302 .

Also, FIG. 7B illustrates a method for obtaining a biometric index 1350 using a biometric index obtaining model 1354 according to another exemplary embodiment.

In this case, the biometric index acquisition model 1354 may acquire the features 1352 and personal and statistical data 1353 extracted from the image 1351 as input values. For example, a feature called time series data for color channel values may be extracted from the image 1351 , and the biometric index acquisition model 1354 receives time series data and personal and statistical data for the color channel values as input values. It can be obtained as a result of the biometric index 1355.

In addition, the personal and statistical data may mean collectible personal and statistical data of the subject, such as age, gender, height, weight, etc. It may also mean personal and statistical data that can be quantified, such as average blood pressure, average color, average height, and average weight.

In addition, since the contents of the above-described biometric index acquisition model 1302 can be applied to the biometric index acquisition model 1354 , the overlapping description will be omitted.

Also, FIG. 8 shows a method for obtaining a biometric index 1400 using a biometric index obtaining model 1405 according to another embodiment.

In this case, the biometric index acquisition method 1400 may include a feature extraction model 1402 , and the feature extraction model 1402 according to an embodiment may be implemented as a machine learning method. For example, the feature extraction model 1402 may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, reinforcement learning, or the like.

Also, the feature extraction model 1402 according to an embodiment may be implemented as an artificial neural network (ANN). For example, the biometric index acquisition model 1302 may be implemented as a feedforward neural network, a radial basis function network, or a kohonen self-organizing network, etc. not limited

Also, the feature extraction model 1402 according to an embodiment may be implemented as a deep neural network (DNN). For example, the biometric index acquisition model 1302 is a convolutional neural network (CNN), a recurrent neural network (RNN), a Long Short Term Memory Network (LSTM), or Gated Recurrent Units (GRUs). and the like, but is not limited thereto.

In addition, the biometric index acquisition model 1405 may acquire the features 1403 and personal and statistical data 1404 extracted from the feature extraction model 1402 as input values, and based on this, obtain the biometric index 1406 ) can be calculated as a result.

In addition, since the above-described biometric index acquisition model 1302 can be applied to the biometric index acquisition model 1405, redundant descriptions will be omitted.

As in the example described with reference to FIGS. 7 and 8 , machine learning, artificial neural network, or deep neural network model may be used to obtain a biometric index.

4. Various Examples of Obtaining a Biometric Index

4.1 Various embodiments of heart rate measurement method

When the heart beats in the body of a living organism, blood can be transported throughout the body by the beating of the heart. At this time, blood flows through the blood vessel, and the volume of the blood vessel may change over time, and the amount of blood contained in the blood vessel may change.

Accordingly, when measuring a change in the volume of a blood vessel or a change in the amount of blood, the heart rate may be acquired. For example, when the amount of blood included in the blood vessel changes, the amount of hemoglobin and oxyhemoglobin included in the blood may change, and accordingly, the amount of light reflected by the blood may change. Accordingly, when the change in the amount of light reflected by the blood is measured as described above, the heart rate may be obtained.

In addition, it is apparent that various principles for measuring the heart rate with light may be applied in addition to the above-described exemplary principles.

9 is a flowchart illustrating a method for measuring a heart rate according to an exemplary embodiment.

Referring to FIG. 9 , the heart rate measuring method 1500 according to an exemplary embodiment includes acquiring an image for at least one image frame among a plurality of acquired image frames ( S1510 ), and detecting a skin region ( S1520 ). ), detecting the ROI ( S1530 ) and processing the data on the ROI ( S1540 ), but is not limited thereto.

In addition, since the steps of acquiring the image ( S1510 ), detecting the skin region ( S1520 ), and detecting the region of interest ( S1530 ) have been described above, overlapping descriptions will be omitted.

In addition, the processing of the data on the ROI ( S1540 ) may be performed on at least one image frame among a plurality of acquired image frames.

Also, a color channel value for the region of interest may be extracted from at least one image frame among a plurality of image frames obtained to process data on the region of interest. In this case, the color channel value may be an average value of color channel values of pixels included in the ROI, and may be referred to as an average pixel value.

In addition, since the detailed content of the step of processing the data on the ROI ( S1540 ) has been described above, the overlapping description will be omitted.

In addition, the heart rate measuring method 1500 according to an exemplary embodiment includes at least some of extracting time series data for at least a part of an image frame group among a plurality of acquired image frames ( S1550 ) and acquiring a heart rate ( S1560 ). may include, but is not limited thereto.

In addition, the step of extracting the time series data ( S1550 ) may be performed on at least some image frame groups among the plurality of obtained image frames.

In this case, the image frame group may mean a plurality of continuous or discontinuous image frame groups. For example, the image frame group may mean a group of continuous image frames from the first image frame to the 180th image frame, but is not limited thereto, and an image of at least a portion of the first image frame to the 180th image frame. It may mean a group of frames.

Also, the step of acquiring the heart rate ( S1560 ) may be performed on at least some of the acquired image frame groups among the plurality of image frames.

In this case, a frequency component of the acquired time series data may be extracted to acquire the heart rate. For example, in order to obtain the heart rate, the time series data may be transformed according to a Fourier transform (FT) and frequency components may be extracted, but the present invention is not limited thereto, and the time series data is subjected to a Fast Fourier transform (Fast Fourier transform). , FFT), discrete Fourier transform (DFT), short time fourier transform (STFT), etc. may be transformed, but the present invention is not limited thereto, and various processes for extracting frequency components may be used.

Also, the heart rate may be obtained based on one heart rate, or may be obtained based on at least two or more heart rates. For example, one heart rate may be obtained based on one time series data, another heart rate may be obtained based on another time series data, and a final heart rate may be obtained based on at least two or more obtained heart rates. A heart rate may be obtained, but is not limited thereto.

4.2 Various Examples of Oxygen Saturation Measurement Method

Oxygen saturation (SPO2) refers to the amount of oxygen bound to hemoglobin, and can be expressed as a fraction of oxyhemoglobin to total hemoglobin in blood.

In addition, hemoglobin and oxyhemoglobin may have the same or different absorption rates for light having one wavelength. For example, hemoglobin and oxyhemoglobin may have different absorbances for light in the 700 nm band, different absorbances to light in the 1000 nm band, and similar absorbance to light in the 800 nm band. .

In addition, when the amount of blood included in the blood vessel changes, the amount of hemoglobin and oxyhemoglobin included in the blood may change.

Therefore, the extinction coefficient of hemoglobin with respect to the light of the first wavelength band, the extinction coefficient of oxyhemoglobin, the extinction coefficient of hemoglobin with respect to the light of the second wavelength band, the extinction coefficient of oxyhemoglobin, and the first wavelength according to the change in the amount of blood Oxygen saturation can be obtained by using the degree of change of light in the band and the degree of change in light in the second wavelength band according to the change in the amount of blood.

, 제1 파장 대역의 빛에 대한 헤모글로빈의 흡광도를

, 제2 파장 대역의 빛에 대한 산소헤모글로빈의 흡광도를

, 제2 파장 대역의 빛에 대한 헤모글로빈의 흡광도를

, 산소헤모글로빈의 비율을 S 라고 할 때 다음과 같은 수학식 1이 성립할 수 있으며,For example, the absorbance of oxyhemoglobin with respect to light in the first wavelength band

, the absorbance of hemoglobin for light in the first wavelength band

, the absorbance of oxyhemoglobin to light in the second wavelength band

, the absorbance of hemoglobin for light in the second wavelength band

, when the ratio of oxyhemoglobin is S, the following Equation 1 can be established,

<Equation 1>

Oxygen saturation may be expressed as S*100 (%), but is not limited thereto.

In addition, it is obvious that various principles for measuring oxygen saturation with light may be applied in addition to the above-described exemplary principles.

10 is a flowchart illustrating a method for measuring oxygen saturation according to an exemplary embodiment.

Referring to FIG. 10 , the oxygen saturation measurement method 1600 according to an embodiment includes the steps of acquiring an image (S1610), detecting a skin region with respect to at least one image frame among a plurality of acquired image frames (S1610) At least some of S1620), detecting the ROI (S1630), and processing at least two color channel values for the ROI (S1640) may be included, but is not limited thereto.

In addition, since the steps of acquiring the image ( S1610 ), detecting the skin region ( S1620 ), and detecting the region of interest ( S1630 ) have been described above, overlapping descriptions will be omitted.

In addition, since the operation of processing at least two color channel values for the ROI ( S1640 ) may perform the above-described operations of processing data for the ROI, redundant description will be omitted.

In addition, the two color channels may be selected in consideration of the absorbance of hemoglobin and the absorbance of oxygen hemoglobin. For example, when the RGB color space is used, a red channel in which the absorbance of hemoglobin is higher than that of oxyhemoglobin and a blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin may be selected, but the present invention is not limited thereto.

In addition, the oxygen saturation measuring method 1600 according to an embodiment includes extracting time-series data for at least two color channel values with respect to at least some image frame groups among a plurality of acquired image frames ( S1650 ), and oxygen saturation It may include at least a part of the step of obtaining ( S1660 ), but is not limited thereto.

Also, the step of extracting time series data for the at least two color channel values ( S1650 ) may be performed for at least some image frame groups among the plurality of obtained image frames.

In this case, the image frame group may mean a plurality of continuous or discontinuous image frame groups. For example, the image frame group may mean a group of continuous image frames from the first image frame to the 180th image frame, but is not limited thereto, and an image of at least a portion of the first image frame to the 180th image frame. It may mean a group of frames.

In addition, the obtaining of the oxygen saturation ( S1660 ) may be performed on at least some of the obtained image frame groups among the plurality of image frames.

In this case, an AC component and a DC component of the obtained at least two time series data may be used to obtain the oxygen saturation level. In this case, the AC component may mean the difference between the maximum and minimum values of time series data, and may mean the difference between the average of the maximum values and the average of the minimum values, but is not limited thereto, and may be understood as a typical AC component. can Also, the DC component may be understood as an average value of time series data, but is not limited thereto, and may be understood as a normal DC component.

Also, to obtain the oxygen saturation, an equation may be used.

, Red 채널에 대한 헤모글로빈의 흡광도를

, Blue 채널에 대한 산소헤모글로빈의 흡광도를

, Blue 채널에 대한 헤모글로빈의 흡광도를

, 산소헤모글로빈의 비율을 S 라고 할 때 다음과 같은 수학식 2가 성립할 수 있다.For example, the absorbance of oxyhemoglobin for the Red channel

, the absorbance of hemoglobin for the Red channel

, the absorbance of oxyhemoglobin for the Blue channel.

, the absorbance of hemoglobin for the Blue channel.

, when the ratio of oxyhemoglobin is S, the following Equation 2 may be established.

<Equation 2>

At this time, the oxygen saturation may be expressed as S*100 (%), but is not limited thereto.

In addition, it is apparent that, in addition to the above-described exemplary equations, various equations for calculating oxygen saturation using at least two color channel values may be used.

In addition, the oxygen saturation level may be obtained based on one oxygen saturation level, or may be obtained based on at least two or more oxygen saturation levels. For example, one oxygen saturation may be obtained based on at least two time series data, and another oxygen saturation level may be obtained based on another at least two time series data, and the obtained at least two or more oxygen levels may be obtained. The final oxygen saturation may be obtained based on the saturation, but is not limited thereto.

In addition, the finally obtained oxygen saturation and an oxymeter may be used to obtain more accurate oxygen saturation.

11 is a flowchart illustrating a method for measuring oxygen saturation according to another exemplary embodiment.

Referring to FIG. 11 , the method 1700 for measuring oxygen saturation according to an embodiment includes the steps of acquiring an image for at least one image frame among a plurality of acquired image frames ( S1710 ), and detecting a skin region ( S1710 ). At least some of S1720), detecting the ROI (S1730), and processing at least one color channel value for the ROI (S1740) may be included, but is not limited thereto.

In addition, the oxygen saturation measuring method 1700 according to an embodiment may acquire a plurality of IR (Infrared) image frames, and acquire an IR image for at least one IR image frame among the plurality of acquired IR image frames. It may include at least some of the step of performing (S1711), the step of detecting the skin region (S1721), the step of detecting the region of interest (S1731), and the step of processing IR data for the region of interest (S1741), but limited thereto doesn't happen

In addition, the steps of acquiring the image (S1710, S1711), detecting the skin region (S1720, S1721), and detecting the region of interest (S1730, S1731) are described above, so the overlapping description will be omitted. do.

In addition, the steps of processing at least one color channel value for the region of interest ( S1740 ) and processing the IR data for the region of interest ( S1741 ) are the operations of the processing of the data for the region of interest described above. Since it can be performed, redundant descriptions will be omitted.

In addition, the at least one color channel and the wavelength band of the IR may be selected in consideration of absorbance of hemoglobin and absorbance of oxyhemoglobin. For example, when using the RGB color space, a red channel in which the absorbance of hemoglobin is higher than that of oxyhemoglobin may be selected, and an IR wavelength band in the 880 nm band in which the absorbance of oxyhemoglobin is higher than that of hemoglobin may be selected. , but is not limited thereto.

In addition, the oxygen saturation measurement method 1700 according to an embodiment includes extracting time series data for color channel values with respect to at least some image frame groups among a plurality of obtained image frames ( S1750 ), and time series for IR data It may include at least a part of extracting data (S1751) and acquiring oxygen saturation (S1760), but is not limited thereto.

In addition, since the steps of extracting the time series data for the color channel value ( S1750 ) and the time series data for the IR data ( S1751 ) may be performed by the above-described steps of extracting the time series data, overlapping The description will be omitted.

In addition, the obtaining of oxygen saturation ( S1760 ) may be performed on at least some image frame groups among the plurality of obtained image frames and at least some IR image frame groups among the plurality of obtained IR image frames.

In this case, the image frame group of at least some of the plurality of obtained image frames and the group of at least some of the obtained IR image frames may be the same as or different from each other.

For example, when the image frame and the IR image frame are acquired in different sequences, the image frame group and the IR image frame group may be different from each other, and the image frame and the IR image frame are acquired in the same sequence. In this case, the image frame group and the IR image frame group may be identical to each other, but the present invention is not limited thereto.

Also, to obtain the oxygen saturation, an equation may be used.

, Red 채널에 대한 헤모글로빈의 흡광도를

, 880nm에 대한 산소헤모글로빈의 흡광도를

, 880nm 대한 헤모글로빈의 흡광도를

, 산소헤모글로빈의 비율을 S 라고 할 때 다음과 같은 수학식 3이 성립할 수 있다.For example, the absorbance of oxyhemoglobin for the Red channel

, the absorbance of hemoglobin for the Red channel

, the absorbance of oxyhemoglobin at 880 nm

, the absorbance of hemoglobin for 880 nm

, when the ratio of oxyhemoglobin is S, the following Equation 3 can be established.

<Equation 3>

At this time, the oxygen saturation may be expressed as S * 100 (%), but is not limited thereto.

In addition, it is apparent that, in addition to the above-described exemplary equations, various equations for calculating oxygen saturation using at least one color channel value and IR data may be used.

In addition, the oxygen saturation level may be obtained based on one oxygen saturation level, or may be obtained based on at least two or more oxygen saturation levels. For example, one oxygen saturation may be obtained based on at least two time series data, and another oxygen saturation level may be obtained based on another at least two time series data, and the obtained at least two or more oxygen levels may be obtained. The final oxygen saturation may be obtained based on the saturation, but is not limited thereto.

In addition, the finally obtained oxygen saturation and an oxymeter may be used to obtain more accurate oxygen saturation.

4.3 Various embodiments of blood pressure measurement method

The blood pressure may refer to a pressure applied to a wall of a blood vessel by blood flowing through the blood vessel.

Accordingly, blood pressure may be affected by the speed of blood flow, the thickness of the blood vessel wall, waste products accumulated in the blood vessel, and the like.

In addition, when blood flows along the blood vessel, the volume of the blood vessel may change with time, and the amount of blood included in the blood vessel may change.

Accordingly, when the rate of change in the volume of blood vessels and the rate of change in the amount of blood are measured, blood pressure can be obtained.

For example, when measuring changes in blood vessels at two points with different distances from the heart, blood pressure can be obtained based on the difference in changes in blood vessels at two points, and features that can represent changes in blood vessels that change with time Blood pressure may be obtained by extraction, but is not limited thereto.

In addition, it is obvious that various principles for measuring blood pressure with light may be applied in addition to the above-described exemplary principles.

12 is a flowchart illustrating a method for measuring blood pressure according to an exemplary embodiment.

Referring to FIG. 12 , the method 1800 for measuring blood pressure according to an embodiment includes acquiring an image (S1810) and detecting a skin region (S1820) with respect to at least one image frame among a plurality of acquired image frames. ), detecting a first region of interest (S1830), detecting a second region of interest (S1831), processing data on the first region of interest (S1840), and processing data on the second region of interest It may include at least a part of the step (S1841), but is not limited thereto.

In addition, since the step of acquiring the image ( S1810 ) and the step of detecting the skin region ( S1820 ) have been described above, overlapping descriptions will be omitted.

In addition, since the details of the steps of detecting the first region of interest and the second region of interest ( S1830 and S1831 ) have been described above, overlapping descriptions will be omitted.

In this case, the first region of interest and the second region of interest may be set as two regions having different distances from the heart of the subject. For example, the first ROI may be set as an upper region of the subject's face, and the second ROI may be set as a lower region of the subject's face, but the present invention is not limited thereto, and the first ROI is is set as the face region of , and the second region of interest may be set as the back region of the subject's hand.

In addition, the processing of the data on the first region of interest ( S1840 ) and the processing of the data on the second region of interest ( S1841 ) may perform the operations of the processing of the data on the region of interest described above. Therefore, redundant descriptions will be omitted.

In addition, the method 1800 for measuring blood pressure according to an embodiment includes extracting time series data for first and second ROIs with respect to at least some image frame groups among a plurality of acquired image frames ( S1850 ); At least a part of calculating a pulse transit time (PTT) based on time series data ( S1860 ) and acquiring blood pressure ( S1870 ) may be included, but is not limited thereto.

In addition, since the steps of extracting the time series data described above may be performed in the step of extracting the time series data for the first and second regions of interest ( S1850 ), a redundant description will be omitted.

In addition, calculating the PTT based on the acquired time series data ( S1860 ) may be performed for at least some image frame groups among the plurality of acquired image frames.

In this case, the PTT may be calculated based on extreme values of the time series data of the first region of interest and the time series data of the second region of interest. For example, the PTT may be calculated based on a time difference between a local maximum of the time series data for the first region of interest and a local maximum of the time series data for the second region of interest, but is not limited thereto, and the first region of interest is not limited thereto. The PTT may be calculated based on a time difference between the minimum value of the time series data for , and the minimum value of the time series data for the second region of interest.

Also, the PTT may be calculated based on an inflection point of the time series data of the first region of interest and the time series data of the second region of interest. For example, the PTT may be calculated based on a time difference between the inflection point of the time series data for the first ROI and the inflection point of the time series data for the second ROI, but is not limited thereto.

In addition, the PTT may be calculated based on various points of time series data for each ROI in addition to the above-described extreme values and inflection points.

In addition, a time difference between the time series data for the first region of interest and the time series data for the second region of interest may be calculated based on frames acquired at points such as an extreme value and an inflection point. For example, when a local maximum is obtained in a tenth frame from time series data for the first region of interest and a local maximum is obtained from time series data for a second region of interest in a twelfth frame, the first region of interest and the second region of interest The time difference of the time series data for the region may be a time for two frames to be acquired, and a PTT may be calculated based on this.

Also, the step of obtaining the blood pressure ( S1870 ) may be performed on at least some of the obtained image frame groups among the plurality of image frames.

In addition, PTT may be used to obtain the blood pressure. For example, a function for PTT may be used to obtain the blood pressure. More specifically, a function such as Equation 4 may be used, and various functions such as a first-order function, a second-order function, a log function, and an exponential function using PTT as a variable may be used, but is not limited thereto. .

<Equation 4>

In addition, PTT and personal and statistical data may be used to obtain the blood pressure. For example, a function for PTT and a function of personal and statistical data such as age, weight, and height may be used to obtain the blood pressure. More specifically, the following Equation 5 may be used, and various functions such as a primary function, a secondary function, a log function, and an exponential function with PTT, weight, height, and age as variables may be used, However, the present invention is not limited thereto.

<Equation 5>

In addition, a regression analysis method using the above-described function may be used to obtain the blood pressure, but is not limited thereto.

In addition, a machine learning method using the above-described function may be used to obtain the blood pressure, but is not limited thereto.

In addition, it is apparent that, in addition to the above-described exemplary equations, various equations for calculating blood pressure using PTT may be used.

Also, the blood pressure may be obtained based on one blood pressure, or may be obtained based on at least two or more blood pressures. For example, one blood pressure may be acquired based on a first PTT calculated based on time series data for the first and second ROIs, and based on other time series data for the first and second ROIs. Another blood pressure may be obtained based on the calculated second PTT, and a final blood pressure may be obtained based on at least two or more obtained blood pressures, but is not limited thereto.

13 is a flowchart illustrating a method for measuring blood pressure according to another exemplary embodiment.

Referring to FIG. 13 , the method 1900 for measuring blood pressure according to an embodiment includes acquiring an image (S1910) and detecting a skin region (S1920) with respect to at least one image frame among a plurality of acquired image frames. ), detecting the region of interest ( S1930 ) and processing data on the region of interest ( S1940 ), but is not limited thereto.

At this time, for the step of acquiring the image (S1910), the step of detecting the skin region (S1920), the step of detecting the region of interest (S1930), and the step of processing the data for the region of interest (S1940) As the detailed content has been described above, overlapping descriptions will be omitted.

In addition, the method 1900 for measuring blood pressure according to an embodiment includes extracting time series data for at least a part of an image frame group among a plurality of obtained image frames (S1950), extracting features based on the obtained time series data. It may include at least a part of the step (S1960) and the step (S1970) of obtaining the blood pressure, but is not limited thereto.

In addition, in the step of extracting time series data ( S1950 ), since the operations of the step of extracting time series data described above may be performed, redundant description will be omitted.

Also, the step of extracting features based on the obtained time series data ( S1960 ) may be performed on at least some image frame groups among the plurality of obtained image frames.

In this case, the characteristics may refer to mathematical and physical characteristics of the acquired time series data. For example, the characteristic is the local maximum of the obtained time series data, the average of the maximum values, the minimum value, the average of the minimum values, the difference between the maximum and minimum values, the average, the inflection point, the first differential data, the second differential data, the slope at a specific time It may mean mathematical characteristics, such as blood change amount, blood change rate, blood vessel change amount, may mean physical characteristics such as blood vessel change rate, but is not limited thereto.

In addition, it is obvious that the above characteristics may be various characteristics for obtaining blood pressure in addition to the above-described exemplary characteristics.

Also, the step of obtaining the blood pressure ( S1970 ) may be performed on at least some of the obtained image frame groups among the plurality of image frames.

Also, the feature may be used to obtain the blood pressure. For example, a function for the feature may be used to obtain the blood pressure. More specifically, a function such as Equation 6 may be used, and various functions such as a first-order function, a second-order function, a log function, and an exponential function having a characteristic as a variable may be used, but the present invention is not limited thereto.

<Equation 6>

In addition, the characteristic and personal and statistical data may be used to obtain the blood pressure. For example, a function of the characteristic and a function of personal and statistical data such as age, weight, and height may be used to obtain the blood pressure. More specifically, the following Equation 7 may be used, and various functions such as a first-order function, a second-order function, a log function, and an exponential function with features, weight, height, and age as variables may be used. However, the present invention is not limited thereto.

<Equation 7>

In addition, a regression analysis method using the above-described function may be used to obtain the blood pressure, but is not limited thereto.

In addition, a machine learning method using the above-described function may be used to obtain the blood pressure, but is not limited thereto.

In addition, it is obvious that, in addition to the above-described exemplary equations, equations for calculating blood pressure using features may be used.

Also, the blood pressure may be obtained based on one blood pressure, or may be obtained based on at least two or more blood pressures. For example, one blood pressure may be obtained based on a first characteristic calculated based on time series data, and another blood pressure may be obtained based on a second characteristic calculated based on other time series data, The final blood pressure may be obtained based on the obtained at least two or more blood pressures, but is not limited thereto.

4.4 Various embodiments of a method for measuring body temperature

14 is a flowchart illustrating a method for measuring body temperature according to an exemplary embodiment.

Referring to FIG. 14 , the body temperature measuring method 2000 according to an embodiment may include at least a part of acquiring the skin temperature ( S2010 ) and acquiring the body temperature ( S2020 ), but is not limited thereto.

In addition, the step of obtaining the skin temperature (S2010) may be performed in a non-contact manner.

For example, an image sensor such as a camera may be used to obtain the skin temperature. More specifically, at least one color channel value of an image obtained from an image sensor such as a camera may be used to obtain the skin temperature. Illustratively, when the HSV color space is used, the skin temperature may be obtained using a value of S (Saturation), but is not limited thereto.

Also, for example, a sensor such as a thermal imaging camera may be used to obtain the skin temperature, and an image sensor such as an infrared camera may be used, but is not limited thereto.

In addition, the step (S2020) of acquiring the body temperature may be performed in a non-contact manner.

For example, skin temperature may be used to obtain the body temperature. More specifically, the body temperature may be obtained based on the skin temperature using the relationship between the body temperature and the portion where the skin temperature is measured.

In this case, the skin temperature may be obtained from the step of obtaining the skin temperature ( S2010 ), or may be obtained by another external sensor.

Also, for example, an image sensor such as a camera may be used to acquire the body temperature. More specifically, an image obtained from an image sensor such as a camera may be used to obtain the skin temperature. For example, the body temperature may be obtained by using an image obtained from an image sensor such as the camera as data in a body temperature measurement machine learning model, but is not limited thereto.

Hereinafter, a method of acquiring skin temperature and a method of acquiring body temperature will be described in detail.

First, a method of acquiring the skin temperature using an image sensor such as a camera will be described.

According to an embodiment, a brightness value of an acquired image may be used to obtain the skin temperature. For example, a function for the lightness value may be used to obtain the skin temperature. More specifically, the following Equation 8 may be used, and various functions such as a first-order function, a second-order function, a log function, and an exponential function using a brightness value as a variable may be used, but is not limited thereto. .

<Equation 8>

Also, according to an embodiment, a brightness value of an acquired image and personal and statistical data may be used to obtain the skin temperature. For example, a function of a brightness value and a function of personal and statistical data such as age, race, and gender may be used to obtain the skin temperature. More specifically, the following Equation 9 may be used, and various functions such as a primary function, a secondary function, a log function, and an exponential function using brightness value, age, race, and gender as variables may be used. , but is not limited thereto.

<Equation 9>

Also, according to an embodiment, a brightness value and a color value of an acquired image may be used to obtain the skin temperature. For example, a function for a brightness value and a function for a color value may be used to obtain the skin temperature. More specifically, the following Equation 10 may be used, and various functions such as a first-order function, a second-order function, a log function, an exponential function, etc. using a brightness value and a color value as variables may be used, but limited to this it doesn't happen

<Equation 10>

Also, according to an exemplary embodiment, a brightness value, a color value, and personal and statistical data of an acquired image may be used to obtain the skin temperature. For example, a function for a brightness value, a function for a color value, and a function of personal and statistical data such as age, race, and gender may be used to obtain the skin temperature. More specifically, the following Equation 11 may be used, and various functions such as a first-order function, a second-order function, a log function, and an exponential function are used for the function using brightness value, color value, age, race, and gender as variables. may be, but is not limited thereto.

<Equation 11>

Also, according to an embodiment, a brightness value, a color value, and a change value of the brightness value of the acquired image may be used to obtain the skin temperature. For example, a function for a brightness value, a function for a color value, and a function for a change value of the brightness value may be used to obtain the skin temperature. More specifically, the following Equation 12 may be used, and various functions such as a first-order function, a second-order function, a log function, and an exponential function are used for the function using a brightness value, a color value, and a change value of the brightness value as variables. may be, but is not limited thereto. In addition, when the change value is used in this way, measurement noise due to an external environment can be reduced.

<Equation 12>

Also, according to an exemplary embodiment, a brightness value, a color value, a change value of a brightness value, and personal and statistical data of an acquired image may be used to obtain the skin temperature. For example, to obtain the skin temperature, a function for a brightness value, a function for a color value, a function for a change value of the brightness value, and a function of personal and statistical data such as age, race, and gender may be used. More specifically, the following Equation 13 may be used, and the function is a linear function, a quadratic function, a log function, and an exponent using a brightness value, a color value, a change value of a brightness value, age, race, and gender as variables. Various functions such as a function may be used, but the present invention is not limited thereto.

<Equation 13>

Also, according to an embodiment, a brightness value, a color value, a change value of a brightness value, and a change value of a color value of an acquired image may be used to obtain the skin temperature. For example, a function for a brightness value, a function for a color value, a function for a change value of a brightness value, and a function for a change value of a color value may be used to obtain the skin temperature. More specifically, the following Equation 14 may be used, and the functions are a linear function, a quadratic function, a log function, and an exponent in which a brightness value, a color value, a change value of a brightness value, and a change value of a color value are variables. Various functions such as a function may be used, but the present invention is not limited thereto. In addition, when the change value is used in this way, measurement noise due to an external environment can be reduced.

<Equation 14>

Also, according to an embodiment, a brightness value, a color value, a change value of a brightness value, a change value of a color value, and personal and statistical data of an image obtained to obtain the skin temperature may be used. For example, to obtain the skin temperature, a function for a brightness value, a function for a color value, a function for a change value of the brightness value, a function for a change value of a color value, and age, race, gender, etc. A function of data may be used. More specifically, the following Equation 15 may be used, and the function is a primary function, a secondary function using a brightness value, a color value, a change value of a brightness value, a change value of a color value, age, race, and gender as variables. Various functions such as a function, a log function, and an exponential function may be used, but the present invention is not limited thereto.

<Equation 15>

In addition, a regression analysis method using the above-described function may be used to obtain the skin temperature, but is not limited thereto.

In addition, a machine learning method or a deep learning method using the above-described function may be used to obtain the skin temperature, but is not limited thereto.

In addition, it is obvious that various equations may be used in addition to the above-described exemplary equations. For example, an equation using at least a portion of personal and statistical data such as the above-described brightness value, color value, brightness value change value, color value change value, age, race, gender, etc. may be used, and other data may be used. The used mathematical expression may be used.

Next, a method of acquiring body temperature using an image sensor such as a camera will be described.

According to an embodiment, the obtained skin temperature may be used to obtain the body temperature. For example, a function for skin temperature may be used to obtain the body temperature. More specifically, the following Equation 16 may be used, and the function may be various functions such as a first-order function, a second-order function, a log function, and an exponential function using skin temperature as a variable, but is not limited thereto.

<Equation 16>

Also, according to an embodiment, when the body temperature is acquired indoors, the acquired skin temperature and the room temperature may be used to acquire the body temperature. For example, a function for skin temperature and a function for room temperature may be used to obtain the body temperature. More specifically, the following Equation 17 may be used, and the function may be various functions such as a first-order function, a second-order function, a log function, and an exponential function using skin temperature and room temperature as variables, but is not limited thereto. does not

<Equation 17>

Also, according to an embodiment, when the body temperature is acquired indoors, the acquired skin temperature, the room temperature, and the heart rate may be used to acquire the body temperature. For example, a function for skin temperature, a function for room temperature, and a function for heart rate may be used to obtain the body temperature. More specifically, the following Equation 18 may be used, and the function may be various functions such as a first-order function, a second-order function, a log function, and an exponential function using skin temperature, room temperature, and heart rate as variables, but is limited thereto. it doesn't happen

<Equation 18>

Also, according to an embodiment, various skin temperatures may be used simultaneously. For example, when the body temperature is acquired indoors, the obtained first skin temperature for the first body part, the second skin temperature for the second body part, the room temperature, and the heart rate may be used to acquire the body temperature. For example, a function for a first skin temperature, a function for a second skin temperature, a function for room temperature, and a function for heart rate may be used to obtain the body temperature. More specifically, the following Equation 19 may be used, and the function includes various functions such as a first-order function, a second-order function, a log function, and an exponential function, in which the first skin temperature, the second skin temperature, the room temperature, and the heart rate are used as variables. may be, but is not limited thereto.

<Equation 19>

In addition, the regression analysis method using the above-described function may be used to obtain the body temperature, but is not limited thereto.

In addition, a machine learning method or a deep learning method using the above-described function may be used to obtain the body temperature, but is not limited thereto.

In addition, it is obvious that various equations may be used in addition to the above-described exemplary equations. For example, a mathematical expression using at least a part of the above-described first skin temperature, second skin temperature, room temperature, and heart rate may be used, and in addition, mathematics using personal and statistical data such as age, gender, race, height, weight, etc. Diet may also be used.

5. Various embodiments of heart rate measurement method

15 is a flowchart illustrating a method for acquiring a heart rate according to an exemplary embodiment.

15 , the method 2100 for acquiring a heart rate according to an embodiment includes acquiring an image (S2110), detecting a skin region (S2120), detecting a region of interest (S2130), It may include at least a part of processing the data for data ( S2140 ), acquiring a characteristic value ( S2150 ), and acquiring a heart rate ( S2160 ), but is not limited thereto.

In this case, since the steps of acquiring the image ( S2110 ), detecting the skin region ( S2120 ), and detecting the region of interest ( S2130 ) have been described above, overlapping descriptions will be omitted.

In addition, the processing of the data on the ROI ( S2140 ) may be performed on at least one image frame among a plurality of acquired image frames.

In addition, the processing of the data for the region of interest ( S2140 ) may be performed to reduce motion artifact, noise caused by external light, and the like.

Also, the step of obtaining the characteristic value ( S2150 ) may be performed on at least some image frame groups among the plurality of obtained image frames.

In addition, the step of acquiring the characteristic value ( S2150 ) may be performed to reduce noise caused by movement, noise caused by external light, and the like.

Also, the step of acquiring the heart rate ( S2160 ) may be performed on at least some image frame groups among the plurality of acquired image frames.

In this case, the image frame group for acquiring the heart rate and the image frame group for acquiring the characteristic value may be the same or different. For example, the image frame group for acquiring the characteristic value may include 18 image frames, and the image frame group for acquiring the heart rate may include 180 image frames, but is not limited thereto.

Hereinafter, the steps of processing data on the region of interest ( S2140 ), obtaining a characteristic value ( S2150 ), and obtaining a heart rate ( S2160 ) will be described in more detail.

16 is a graph of color channel values according to an exemplary embodiment.

According to an embodiment, a color channel value for the ROI may be extracted to process data on the ROI. In this case, the color channel value may be an average value of the color channel values of pixels included in the ROI, and may also be referred to as an average pixel value.

Referring to FIG. 16 , a color channel value according to the RGB color space for the ROI may be extracted in order to process data for the ROI. More specifically, a red channel value that is an average value of red channel pixel values, a blue channel value that is an average value of blue channel pixel values, and a green channel value that is an average value of green channel pixel values may be extracted.

For example, when a color channel value according to an RGB color space is extracted, a red channel pixel value, a green channel pixel value, and a blue channel pixel value of each pixel included in the ROI may be extracted, and the ROI may be A red channel value that is an average of included red channel pixel values, a blue channel value that is an average of blue channel pixel values, and a green channel value that is an average of green channel pixel values may be extracted, but is not limited thereto, and the HSV, YCrCb color space is not limited thereto. Color channel values according to various color spaces such as, etc. may be extracted.

Also, a color channel value extracted according to a specific color space may be converted into another color space. For example, a color channel value extracted according to an RGB color space may be converted into a color channel value according to various color spaces such as HSV and YCrCb color space.

In addition, the color channel value extracted to process the data for the ROI may be a color channel value combined by applying a weight to at least some of the color channel values extracted according to various color spaces.

In addition, the color channel value may be extracted for each of a plurality of continuously acquired image frames, or may be extracted for at least some of the image frames.

Hereinafter, description will be made based on the color channel values extracted according to the RGB color space, but it is obvious that various color channel values may be applied without being limited thereto.

16 (a) is a graph showing the Red channel values extracted according to the RGB color space, (b) is a graph showing the Green channel values extracted according to the RGB color space, (c) is a graph showing the extracted Red channel values according to the RGB color space It is a graph showing the extracted blue channel value.

As shown in FIG. 16 , the value of each color channel may change according to the heartbeat.

However, the value of each color channel may change according to the heartbeat, and at the same time, the value may also change according to the movement of the subject or change in the intensity of external light.

Therefore, in order to obtain the heart rate using the extracted color channel value, it is necessary to reduce the fluctuation of the value due to the movement of the subject or the change in the intensity of external light, and to maximize the fluctuation of the value according to the heart rate. can

5.1 Various embodiments of a method for processing data on a region of interest

It is obvious that the above-described contents can be applied to a method of processing data for the region of interest, and descriptions overlapping with the above-described contents will be omitted.

17 is a graph for explaining a noise reduction method according to an embodiment.

17 (a) is a graph showing the red channel value extracted according to the RGB color space, and (b) is a graph showing the green channel value extracted according to the RGB color space.

Referring to FIGS. 17A and 17B , it can be seen that the extracted color channel value varies with time.

In this case, the extracted color channel value may fluctuate depending on the heartbeat, but may also fluctuate according to the movement of the subject or change in the intensity of external light.

More specifically, large and slow fluctuations in the color channel value are those that are more influenced by the movement of the subject or changes in the intensity of external light, and those that occur quickly and with small fluctuations are the heart of the subject. It may be that the fluctuations are caused by being more influenced by the beat.

Therefore, the relative difference between the values of at least two color channels can be used to reduce the change in the value due to the movement of the subject or the change in the intensity of external light rather than the change according to the heartbeat.

For example, a difference value between a green channel value and a red channel value may be used to reduce noise. More specifically, the Green channel value and the Red channel value obtained from the same image frame can reflect the same motion and the same intensity of external light, and the difference between the Green channel value and the Red channel value of the same frame is the difference between the movement of the subject and the external light. Noise caused by a change in light intensity may be reduced, but the present invention is not limited thereto, and noise may be reduced by using a relative difference between values of at least two color channels.

17C is a graph illustrating a difference value between the green channel value and the red channel value.

As shown in (c) of FIG. 17 , the difference value between the green channel value and the red channel value can reduce noise caused by movement of a subject and change in the intensity of external light.

In addition, the above-described method for reducing noise may be performed on at least one image frame among a plurality of acquired image frames, or may be performed on each of a plurality of consecutive image frames.

Also, although not shown in (c) of FIG. 17, noise may be reduced by using a difference value between the green channel value and the blue channel value, and by using the difference value between the red channel value and the blue channel value. Noise can also be reduced.

Also, as described above, at least two color channel values may be selected to obtain a difference value in order to reduce noise using a relative difference between the at least two color channel values.

In this case, the values of the at least two color channels may be selected in consideration of absorbance of blood.

18 is a diagram showing the absorbance of hemoglobin and oxyhemoglobin in the visible light band.

According to an embodiment, the Red channel may be a channel including at least a portion of a 620 nm to 750 nm wavelength band, the Green channel may be a channel including at least a portion of a 495 nm to 570 nm wavelength band, and the Blue channel may be a 450 nm to 495 nm wavelength band. It may be a channel including at least a part of a wavelength band, but is not limited thereto, and each of a red channel, a green channel, and a blue channel may be a commonly understood color channel.

Referring to FIG. 18 , the absorbance of hemoglobin and oxyhemoglobin according to the wavelength band of light can be seen. For example, as shown in FIG. 18 , the absorbances of hemoglobin and oxygen hemoglobin to light in the 550 nm wavelength band included in the Green channel are higher than the absorbances of hemoglobin and oxygen hemoglobin to light in the 650 nm wavelength band included in the Red channel. can

Also, for example, as shown in FIG. 18 , the absorbances of hemoglobin and oxygen hemoglobin with respect to light in the 550 nm wavelength band included in the Green channel are the absorbances of hemoglobin and oxygen hemoglobin to light in the 470 nm wavelength band included in the Blue channel. can be higher.

In addition, when blood is transported throughout the body by heartbeat, the volume of the blood vessel may change or the amount of blood included in the blood vessel may change due to the blood flow.

Accordingly, a color channel value including a wavelength band of light, which is relatively absorbed by hemoglobin and oxygenated hemoglobin in blood, may be changed relatively a lot due to a change in the amount of blood due to a heartbeat.

On the other hand, a color channel value including a wavelength band of light in which absorption by hemoglobin and oxygenated hemoglobin in blood is relatively small may be changed relatively little due to a change in the amount of blood due to a heartbeat.

Accordingly, according to an embodiment, at least two color channels for reducing noise may be selected in consideration of the absorbance of hemoglobin and oxyhemoglobin.

For example, according to one embodiment, in order to reduce noise, the difference value between the Green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, is the difference value. can be used

Also, for example, according to one embodiment, in order to reduce noise, the Green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Blue channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, are relatively low. A difference value may be used.

Also, for example, according to one embodiment, in order to reduce noise, the difference between the Blue channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, is reduced. A value may be used.

Also, for example, according to an embodiment, a difference value between the green channel value and the red channel value and a difference value between the green channel value and the blue channel value may be used simultaneously.

In addition, although the above-described examples have been described based on the difference value, in order to reduce noise caused by the movement of the subject and noise caused by external light, in addition to the difference value, a processed value processed using a weight for each channel value may be used. have.

For example, as shown in Equation 20 below, a processed value processed by using a weight for each channel value may be used.

<Equation 20>

In addition, at this time, each a, b, and c value may be determined so that a+b+c=0 in order to efficiently remove noise caused by movement of the subject and noise caused by external light, which means that each channel value is Since one image frame may include noise due to a similar degree of motion and noise caused by external light, it may be advantageous to effectively reduce noise.

5.2 Various embodiments of property value acquisition

A characteristic value may be obtained in order to reduce noise caused by the movement of the subject and noise caused by the intensity of external light.

In this case, the characteristic value may be obtained for at least some image frame groups among the plurality of obtained image frames.

Also, the characteristic value may be a value indicating a characteristic of an acquired color channel value or a processing value. For example, the characteristic value may mean an average value, a deviation value, or a standard deviation value of a color channel value or a processing value included in an image frame group, but is not limited thereto.

19 is a diagram for describing a method of acquiring a characteristic value according to an exemplary embodiment.

19A is a graph illustrating a color channel value obtained according to an exemplary embodiment, and more specifically, a graph illustrating a difference value between a green channel value and a red channel value. However, this is only specifically indicated as a difference value between the green channel value and the red channel value for convenience of explanation, and is not limited thereto, and may be various color channel values, difference values, and processing values.

Referring to (a) of FIG. 19 , it can be seen that the difference between the green channel value and the red channel value (hereinafter referred to as a 'G-R value') may not have a constant magnitude of change with time. .

In this case, the G-R value may not be constant due to the movement of the subject. For example, when the movement of the subject is small, the change in the G-R value may be small, and when the movement of the subject is large, the change in the G-R value may be large, but is not limited thereto.

Also, the G-R value may not be constant depending on the intensity of external light. For example, when the intensity of the external light is weak, the change in the G-R value may be small, and when the intensity of the external light is strong, the change in the G-R value may be large, but is not limited thereto.

Accordingly, the characteristic value may be extracted in order to reduce noise caused by the movement of the subject or the intensity of external light as described above.

Also, a window for the characteristic value may be set in order to extract the characteristic value.

In this case, the window for the characteristic value may mean a preset time interval, or may mean the preset number of frames, but is not limited thereto. It may mean a window for setting

19B is a schematic diagram for explaining a window for a characteristic value, and more specifically, a schematic diagram for explaining a window for a characteristic value set to 18 image frames obtained by dividing 180 image frames into 10 equal parts. However, for convenience of explanation, this is only a window for the characteristic value set with 18 image frames divided into 10 equal parts for 180 image frames, and the window for the characteristic value is not limited thereto. .

Referring to FIG. 19B , a plurality of acquired image frames may be set as a group by a window for a characteristic value. For example, as shown in (b) of FIG. 19 , 180 image frames may be set as a group including 18 image frames each by a window for a characteristic value. More specifically, the first image frame to the 18th image frame may be included in the first image frame group 2210, and the 19th image frame to the 36th image frame may be included in the second image frame group 2220. not limited

In this case, the characteristic value may be obtained for an image frame group set by a window for the characteristic value. For example, the characteristic value may be obtained for color channel values for the first image frame group 2210 and may be obtained for color channel values for the second image frame group 2220 .

Also, for example, when the characteristic value is an average value, an average value of color channel values for an image frame group may be obtained. More specifically, an average value of G-R values for the first to eighteenth image frames included in the first image frame group 2210 may be obtained, and the 19th to the nineteenth to be included in the second image frame group 2220 may be obtained. An average value of G-R values for 36 image frames may be obtained, but is not limited thereto.

Also, for example, when the characteristic value is a standard deviation value, a standard deviation value of a color channel value for an image frame group may be obtained. More specifically, standard deviation values of G-R values for the first to eighteenth image frames included in the first image frame group 2210 may be obtained, and the 19th to eighteenth image frames included in the second image frame group 2220 may be obtained. Standard deviation values of G-R values for the 36 th image frames may be obtained, but the present invention is not limited thereto.

However, it is not limited to the above-described examples, and various characteristic values may be obtained for the image frame group.

Also, the characteristic value may be obtained for at least some image frames included in an image frame group divided by a window for the characteristic value. For example, the characteristic value may be obtained with respect to a color channel value of at least some of the 18 image frames included in the first image frame group 2210 , and in the second image frame group 2220 . It may be obtained with respect to a color channel value of at least some of the included 18 image frames.

Also, for example, when the characteristic value is a deviation value, a deviation value of color channel values for at least some image frames included in an image frame group may be obtained. More specifically, a deviation value of the G-R value of the first image frame included in the first image frame group with respect to the average of the G-R value of the first image frame group 2210 may be obtained, and the second image frame group 2220 . A deviation value of the G-R value of the 19th image frame included in the second image frame group with respect to the average of the G-R values of may be obtained, but is not limited thereto.

Also, for example, when the characteristic value is a deviation value, a deviation value of color channel values for at least some image frames included in an image frame group may be obtained. More specifically, a deviation value of the G-R value of the first image frame included in the first image frame group with respect to the average of the G-R value of the first image frame group 2210 may be obtained, and the first image frame group 2210 A deviation value of the G-R value of the second image frame included in may be obtained, but is not limited thereto.

Also, the obtained characteristic values may be normalized.

For example, when the characteristic value is a deviation value, the deviation value may be normalized by a standard deviation value. More specifically, when a deviation value of the G-R value of the first image frame included in the first image frame group 2210 with respect to the average of the G-R values of the first image frame group 2210 is obtained, the G-R of the first image frame The value deviation value may be normalized by the standard deviation value of the first image frame group 2210, but is not limited thereto and may be normalized in various ways.

In addition, when normalized in this way, the magnitude of the change is normalized to better reflect the change in the value due to the heartbeat, and noise caused by the movement of the subject and the noise caused by the change in the intensity of external light can be effectively reduced.

19C is a graph illustrating characteristic values obtained according to an exemplary embodiment, and more specifically, a graph illustrating a deviation value obtained based on a G-R value. However, for convenience of explanation, this is only a specific indication of the deviation value obtained based on the G-R value, and the present invention is not limited thereto, and may be various characteristic values obtained based on various color channel values, difference values, and processing values.

Referring to (c) of FIG. 19 , it can be seen that the magnitude of the change of the value is more constant than that of FIG. 19 (a).

Accordingly, acquiring the characteristic value as described above reduces noise caused by movement of the subject and noise caused by a change in the intensity of external light, and makes it possible to better reflect the change in the value according to the heartbeat.

20 is a diagram for describing a method of acquiring a characteristic value according to another exemplary embodiment.

20A is a graph illustrating a color channel value obtained according to an exemplary embodiment, and more specifically, a graph illustrating a difference value between a green channel value and a red channel value. However, this is only specifically indicated as a difference value between the green channel value and the red channel value for convenience of explanation, and is not limited thereto, and may be various color channel values, difference values, and processing values.

Referring to (a) of FIG. 20, it can be seen that the difference between the green channel value and the red channel value (hereinafter referred to as 'G-R value') may not have a constant magnitude of change over time. .

In this case, the G-R value may not be constant due to the movement of the subject. For example, the overall G-R value may be small in the time period 2301 in which the subject is located in the first state, and the overall G-R value in the time period 2302 in which the subject is located in a second state different from the first state. The value may be large, but is not limited thereto.

Also, the G-R value may not be constant depending on the intensity of external light. For example, the intensity of external light in the time period 2301 in which the subject is located in the first state and the intensity of external light in the time period 2302 in which the subject is located in the second state may be different from each other. And, this may cause a difference in the overall G-R value, but is not limited thereto.

Accordingly, the characteristic value may be extracted in order to reduce noise caused by the movement of the subject or the intensity of external light as described above.

Also, a window for the characteristic value may be set in order to extract the characteristic value.

In this case, the window for the characteristic value may mean a preset time interval or may mean a preset number of frames, but is not limited thereto, and a frame group of at least some of the plurality of frames to obtain the characteristic value. It may mean a window for setting

Also, at least some of the frame groups set by the window may overlap at least partially.

20B is a schematic diagram for explaining a window for a characteristic value, and more specifically, a schematic diagram for explaining a window for a characteristic value set to a size obtained by dividing 180 image frames into 8 equal parts. However, for convenience of explanation, this is only showing a window for a characteristic value set in a size obtained by dividing 180 image frames into eight equal parts, and the window for the characteristic value may be set in various methods and sizes without being limited thereto.

Also, referring to FIG. 20B , a plurality of acquired image frames may be set as a group by a window for a characteristic value. For example, as shown in (b) of FIG. 20 , 180 image frames may be set as a group including 22 or 23 image frames by a window for a characteristic value. More specifically, the first image frame to the twenty-second image frame may be included in the first image frame group 2310 .

Also, referring to FIG. 20B , the image frame group set by the window for the characteristic value may overlap at least partially. For example, as shown in (b) of FIG. 20 , from the first image frame to the 22nd image frame may be included in the first image frame group 2310, and from the 6th image frame to the 28th image frame It may be included in the second image frame group 2320 , the twelfth image frame to the 33rd image frame may be included in the third image frame group 2330 , and the seventeenth image frame to the 39th image frame group 2340 may be included in the group 2340 . However, the present invention is not limited thereto.

Also, referring to FIG. 20B , the image frame groups set by the window for the characteristic value may not overlap. For example, as shown in (b) of FIG. 20 , the first image frame to the 22nd image frame may be included in the first image frame group 2310 , but the fifth image frame from the 23rd image frame to the 45th image frame may be included in the first image frame group 2310 . It may be included in the image frame group 2350, but is not limited thereto.

In this case, the characteristic value may be acquired for an image frame group set by a window for the characteristic value, and may also be acquired for at least some image frames included in the image frame group, but for details on this Since the above contents may be applied, the duplicate description will be omitted.

However, processing of characteristic values obtained with respect to an image frame included in at least partially overlapping image frame groups will be described in detail below.

When the characteristic value is obtained, a plurality of characteristic values may be obtained for an image frame included in an area where at least two image frame groups overlap.

For example, at least two characteristic values may be obtained for sixth to twenty-second image frames in which the first image frame group 2310 and the second image frame group 2320 overlap.

More specifically, for the sixth image frame, a first deviation value that is a deviation value of the G-R values of the sixth image frame with respect to the average of the G-R values of the first image frame group 2310 may be obtained, and the second image frame group 2310 A second deviation value that is a deviation value of the G-R value of the sixth image frame with respect to the average G-R value of 2320 may be obtained, but is not limited thereto.

Also, a plurality of acquired characteristic values may be obtained as one characteristic value through an operation. For example, the deviation value of the sixth image frame may be obtained by adding the first deviation value and the second deviation value, but is not limited thereto.

In addition, the above-described operations may be applied to obtain a characteristic value for an image frame included in an area where a plurality of image frame groups such as three or four overlap.

Also, in addition to the above-described operation, a sliding window method that can be generally understood may be applied.

20( c ) is a graph illustrating characteristic values obtained according to an exemplary embodiment, and more specifically, a graph illustrating a deviation value obtained based on a G-R value. However, for convenience of explanation, this is only a specific indication of the deviation value obtained based on the G-R value, and the present invention is not limited thereto, and may be various characteristic values obtained based on various color channel values, difference values, and processing values.

Referring to (c) of FIG. 20 , it can be seen that the magnitude of the overall value is constant compared to that of (a) of FIG. 20 .

More specifically, the overall characteristic value in the time period 2301 in which the subject is located in the first state and the overall characteristic value in the time period 2302 in which the subject is located in the second state may be similar to each other.

Accordingly, acquiring the characteristic value as described above reduces noise caused by the movement of the subject and noise caused by changes in external light intensity, etc., and makes it possible to better reflect changes in values according to heartbeat.

5.3 Various embodiments of a method using a plurality of characteristic values

The characteristic value obtained according to the above-described methods may be affected by the underlying color channel value, the difference value, and the processing value. Accordingly, obtaining a plurality of characteristic values based on various color channel values, difference values, and processing values and using the plurality of characteristic values may enable more accurate acquisition of the biometric index.

21 is a diagram for explaining a method of using a plurality of characteristic values.

21A is a graph showing two characteristic values obtained according to an embodiment, and more specifically, a first characteristic value obtained based on a G-R value and a second characteristic value obtained based on a G-B value. It is a graph representing However, this is only specifically shown for convenience of description, and is not limited thereto, and may be a characteristic value obtained based on various color channel values, difference values, and processing values.

In this case, the first characteristic value obtained based on the G-R value may be affected by the G-R value. For example, when the external light is light close to the blue channel, the G-R value may not reflect changes in blood according to the heartbeat well.

Alternatively, for example, the change in blood according to the heartbeat may be reflected by being influenced by the difference between the absorbance of the Green channel and the absorbance of the Red channel.

Also, the second characteristic value obtained based on the G-B value may be affected by the G-B value. For example, when the external light is light close to the red channel, the G-B value may not reflect the blood change according to the heartbeat well.

Alternatively, for example, the change in blood according to the heartbeat may be reflected by being affected by the difference between the absorbance of the green channel and the absorbance of the blue channel.

Also, referring to FIG. 21A , the first characteristic value and the second characteristic value may have a complementary relationship. For example, in a section where the first characteristic value does not reflect the change according to the heartbeat well, the second characteristic value may well reflect the change according to the heartbeat, and vice versa.

Accordingly, the first characteristic value and the second characteristic value may be used to reduce noise caused by a change in the wavelength of external light or to better reflect a change in blood according to a heartbeat.

21B is a graph illustrating a third characteristic value obtained by using the first characteristic value and the second characteristic value, and more specifically, a graph obtained by adding the first characteristic value and the second characteristic value. It is a graph showing the third characteristic value. However, this is only specifically shown for convenience of description, and is not limited thereto.

Also, the third characteristic value may be obtained based on calculation of the first characteristic value and the second characteristic value. For example, the third characteristic value is the first characteristic value and the second characteristic value. It may be obtained based on a sum operation of, but is not limited thereto, and may be obtained based on various operations such as a difference operation and a multiplication operation.

Also, the third characteristic value may be obtained by giving various weights to the first characteristic value and the second characteristic value. For example, it may be obtained based on Equation 21 below, but is not limited thereto.

<Equation 21>

In addition, referring to (a) and (b) of FIG. 21 , the third characteristic value may better reflect a change in blood according to the heartbeat than the first characteristic value and the second characteristic value, and Noise caused by a change in wavelength can be reduced.

5.4 Various embodiments of heart rate acquisition method

In order to obtain a heart rate from data obtained from the above-described methods, it may be necessary to detect a periodic change according to a heartbeat. For example, in order to obtain a heart rate from the acquired characteristic value, it is necessary to acquire the wavelength or frequency component most included in the characteristic value.

22 is a graph in which frequency components are extracted from the graph for the characteristic values shown in FIG. 21(b). More specifically, FIG. 22 is a graph obtained by converting a graph of a characteristic value into a frequency domain by fast Fourier transforming. However, for convenience of explanation, the fast Fourier transform is specifically indicated, but the present invention is not limited thereto, and the graph for the characteristic value includes a fast fourier transform (FFT), a discrete fourier transform (DFT), It may be transformed according to a short time fourier transform (STFT) or the like.

Also, the graph of the characteristic value may be transformed into the frequency domain as shown in FIG. 22 .

In this case, a frequency index having the highest intensity may be obtained, and the heart rate may be obtained by Equation 22 below.

<Equation 22>

For example, as shown in FIG. 22 , when the frequency index having the highest intensity is 1.2 Hz, the heart rate may be 72 bpm.

In addition, although the graph of the characteristic value is not shown in FIG. 22, it may be converted into a frequency*measurement time domain.

In this case, the index with the highest intensity (wavelength index) may be obtained, and the heart rate may be obtained by Equation 23 below.

<Equation 23>

For example, although not shown in FIG. 22 , when the index with the highest intensity is 8 and the measurement time is 6.6 seconds, the heart rate may be 8/6.6*60, which may be 72bpm.

Also, in addition to the above-described examples, various equations for obtaining the heart rate may be used.

Also, a preliminary heart rate may be obtained to obtain the heart rate. In this case, the preliminary heart rate may be a calculated heart rate that is a basis for obtaining the heart rate.

23 is a diagram for describing a method for acquiring a heart rate according to an exemplary embodiment.

Prior to description, a 'preliminary heart rate' described below may mean a heart rate obtained according to a heart rate acquisition method, or may mean a heart rate that is a basis for acquiring one heart rate. For example, the at least two heart rates obtained according to the above-described heart rate acquisition method may be a first preliminary heart rate and a second preliminary heart rate, which are a basis for obtaining one final heart rate, but are not limited thereto.

In addition, the preliminary heart rate may be a final heart rate by itself, and a final heart rate may be obtained based on a plurality of preliminary heart rates.

23A is a graph showing values obtained as time series data. For example, (a) of FIG. 23 may mean a color channel value obtained as time series data, but is not limited thereto, and may mean a difference value or a processing value obtained as time series data, or obtained as time series data It can mean the value of the specified characteristic.

Since the above description may be applied to a method of converting a graph representing values obtained as time series data as shown in FIG. 23A into a wavelength domain or a frequency domain, redundant descriptions will be omitted.

23B is a schematic diagram for explaining a window for a preliminary heart rate, and more specifically, a schematic diagram for explaining a window for a preliminary heart rate set to a size of 6 seconds. However, this is only specific for convenience of description, and is not limited thereto, and windows may be set in various sizes.

In this case, the window for the preliminary heart rate may mean a preset time interval or may mean a preset number of frames, but is not limited thereto. It may mean a window for setting.

Also, referring to FIG. 23B , a plurality of acquired image frames may be set as a group by a window for a preliminary heart rate. For example, as shown in (b) of FIG. 20 , an image frame acquired between 0 seconds and 6 seconds may be included in the first image frame group 2410, but is not limited thereto.

Also, referring to FIG. 23B , the image frame group set by the window for the preliminary heart rate may at least partially overlap. For example, as shown in (b) of FIG. 23 , an image frame acquired between 0 seconds and 6 seconds may be included in the first image frame group 2410 , and image frames acquired between 0.5 seconds and 6.5 seconds The image frame may be included in the second image frame group 2420, and image frames acquired between 1 and 7 seconds may be included in the third image frame group 2430, and image frames acquired between 1.5 and 7.5 seconds The image frame may be included in the fourth image frame group 2440, but is not limited thereto.

In this case, the preliminary heart rate may be acquired for an image frame group set by a window for the preliminary heart rate. For example, the first preliminary heart rate may be obtained based on characteristic values obtained from image frames included in the first image frame group 2410 .

In addition, in order to obtain a preliminary heart rate, a value obtained as time series data in each image frame group may be converted into a wavelength domain or a frequency domain. For example, a value obtained as time series data in the first image frame group may be converted into first frequency data 2460 , and a value obtained as time series data in the second image frame group is second frequency data 2470 . may be converted to , and a value obtained as time series data in the third image frame group may be converted into third frequency data 2480, and a value obtained as time series data in the fourth image frame group may be converted to the fourth frequency data ( 2490), but is not limited thereto.

In addition, since the above-described heart rate acquisition method may be applied to acquiring the first to fourth preliminary heart rates from the first to fourth frequency data 2460 , 2470 , 2480 and 2490 , a redundant description will be omitted.

Also, a heart rate may be obtained based on a plurality of preliminary heart rates. For example, an operation for a plurality of preliminary heart rates may be performed to obtain a heart rate, and more specifically, an operation for extracting the average, maximum, and minimum values of the first to fourth preliminary heart rates is performed to determine the heart rate. may be obtained, but is not limited thereto.

Also, a heart rate may be obtained based on a plurality of preliminary heart rates. For example, a heart rate may be obtained by performing an operation on the remaining heart rates except for the preliminary heart rate having a different tenth digit among the plurality of obtained preliminary heart rates.

More specifically, when the first preliminary heart rate is 72 bpm, the second preliminary heart rate is 80 bpm, the third preliminary heart rate is 75 bpm, and the fourth preliminary heart rate is 73 bpm, except for a second preliminary heart rate having a different digit of 10, the first, A heart rate may be obtained by performing an operation on the third and fourth preliminary heart rates. In this case, the operation may be an operation for extracting an average, a maximum value, a minimum value, and the like.

Also, a heart rate may be obtained based on a plurality of preliminary heart rates. For example, if the ten digit of the four acquired preliminary heart rates are different in pairs, the calculation of the remaining heart rates except for the reserve heart rate pair in which the ten digit of the four acquired reserve heart rates is different from the previously acquired heart rate This may be performed to obtain a heart rate.

More specifically, if the first reserve heart rate is 72 bpm, the second reserve heart rate is 80 bpm, the third reserve heart rate is 85 bpm, the fourth reserve heart rate is 73 bpm, and the previously acquired heart rate is 75 bpm, the second and third reserve heart rates are Except this, calculations on the first and fourth preliminary heart rates may be performed to obtain a heart rate.

Also, as described above, when a heart rate is obtained using a plurality of preliminary heart rates, a heart rate that is more robust to noise and more accurate may be obtained.

5.5 Various embodiments of heart rate output

The heart rate obtained by the above-described methods may be output through a display or the like, or may be transmitted to a terminal or a server using a communication unit. Hereinafter, for convenience of description, such an operation will be described as an output of the heart rate.

When the heart rate is continuously measured in real time, correction may be performed on the output heart rate in order to provide stability and reliability to the measured heart rate.

In addition, when correction is performed on the output heart rate, even if a bad image is obtained from some of the plurality of acquired image frames, stability and reliability may be imparted to the acquired and output heart rate.

24 is a flowchart illustrating a method of correcting an output heart rate according to an exemplary embodiment.

Referring to FIG. 24 , the method 2500 for correcting an output heart rate according to an exemplary embodiment includes acquiring a first heart rate ( S2510 ), and comparing a difference between the first heart rate and a first time point with a reference value ( S2520 ). ), but is not limited thereto.

In the step of obtaining the first heart rate ( S2510 ), since the above-described methods for obtaining a heart rate may be applied, a redundant description will be omitted.

In the step of comparing the difference between the first heart rate and the first time point heart rate with a reference value ( S2520 ), the first time point heart rate may mean a heart rate acquired or output before the first heart rate is acquired. For example, when the first heart rate is obtained at 6.5 seconds, the first time point heart rate may be a heart rate obtained at 6 seconds or a heart rate output at 6 seconds, but is not limited thereto.

In addition, the reference value may be determined as a predetermined numerical value or may be determined as a predetermined ratio. For example, the reference value may be set to 10, and in this case, it may be determined whether a difference between the first heart rate and the heart rate at the first time exceeds 10, but is not limited thereto.

In addition, when the difference between the first heart rate and the heart rate at the first time point is equal to or less than a reference value, the step of outputting the first heart rate as the heart rate at the second time point ( S2531 ) may be performed, wherein the second time point is the second time point. It may be a later time point than 1 time point.

For example, when the heart rate at the first time point is 72 bpm, the first heart rate is 75 bpm, and the reference value is 10, the heart rate output at the second time point may be 75 bpm, but is not limited thereto.

In addition, when the difference between the first heart rate and the heart rate at the first time point exceeds a reference value, outputting a heart rate corrected from the first time point heart rate as a second time point heart rate (S2532) may be performed. The second time point may be later than the first time point.

In addition, an operation may be performed on the first time point heart rate to obtain a heart rate corrected from the first time point heart rate. For example, an operation such as adding or subtracting a predetermined value to the first time point heart rate may be performed, but is not limited thereto.

For example, when the heart rate at the first time point is 72 bpm, the first heart rate is 85 bpm, and the reference value is 10, the heart rate output at the second time point may be 75 bpm obtained by adding +3 bpm to the first time point heart rate. , but is not limited thereto.

Also, for example, when the heart rate at the first time point is 72 bpm, the first heart rate is 61 bpm, and the reference value is 10, the heart rate output at the second time point is 69 bpm obtained by adding -3 bpm to the heart rate at the first time point. However, the present invention is not limited thereto.

5.6 Various embodiments of heart rate signal extraction

The heartbeat signal may mean a signal that may vary according to a heartbeat, and may mean a signal that may be estimated to vary according to a heartbeat.

Also, a heartbeat signal may be extracted based on the plurality of acquired image frames.

25 is a diagram for describing a method for extracting a heartbeat signal according to an exemplary embodiment.

First, (a) of FIG. 25 is a graph showing values obtained as time series data. For example, (a) of FIG. 25 may mean a color channel value obtained as time series data, but is not limited thereto, and may mean a difference value or a processing value obtained as time series data, or obtained as time series data It can mean the value of the specified characteristic.

In this case, the value obtained as the time series data may be extracted as a heartbeat signal through a band pass filter. More specifically, the value obtained as the time series data may be extracted as a heartbeat signal through a bandpass filter of a frequency band or a wavelength band corresponding to the heart rate.

In addition, the frequency band or wavelength band corresponding to the heart rate may be a commonly understood frequency band or wavelength band, but is not limited thereto, and a frequency band or wavelength determined based on the heart rate obtained by the above-described methods. It can be a band.

For example, a typical heart rate may be 60 to 100 bpm, and a corresponding frequency band may be 1 Hz to 1.67 Hz, and a corresponding bandpass filter may be used, but is not limited thereto.

Also, for example, when the acquired heart rate is 72 bpm, the corresponding frequency is 1.2 Hz, and a frequency band may be set based on this. More specifically, when the frequency band is set in the range of 0.5 Hz, the frequency band may be set in the range of 0.95 Hz to 1.45 Hz, and a band pass filter corresponding thereto may be used, but is not limited thereto.

FIG. 25B is a graph of a heartbeat signal obtained by extracting a value obtained from the time series data shown in FIG. 25A as a heartbeat signal through a band pass filter.

Referring to (b) of FIG. 25 , it can be seen that a value obtained as time series data can be extracted as a heartbeat signal through a band pass filter.

5.7 Various embodiments of a method for measuring heart rate using infrared rays

Basically, the heart rate measuring methods described above may also be used for the heart rate measuring method using infrared rays.

26 is a diagram for describing a method of acquiring a heart rate using infrared rays according to an exemplary embodiment.

In this case, the infrared may be infrared in the near infrared region, which is a wavelength band of 750 nm to 3000 nm, but is not limited thereto, and the middle infrared, far infrared and extreme infrared rays Infrared of the extreme infrared may be used.

Referring to FIG. 26 , the method 2600 for measuring a heart rate using infrared rays according to an embodiment includes acquiring an image (S2610) for at least one image frame among a plurality of acquired image frames, and detecting a skin region. At least some of the step S2620, the step of detecting the ROI (S2630), and the step of processing the data on the ROI (S2640) may be included, but is not limited thereto.

In addition, since the above contents may be applied to the step of acquiring the image ( S2610 ) and the step of detecting the skin region ( S2620 ), overlapping descriptions will be omitted.

In addition, since the above-described methods for detecting the ROI may be applied to the detailed operations of the step of detecting the ROI ( S2630 ), overlapping descriptions will be omitted.

Also, the step of detecting the region of interest ( S2630 ) may include detecting the first, second, and third regions of interest, and may be performed on at least one image frame among a plurality of acquired image frames. .

Also, the first, second, and third regions of interest may at least partially overlap each other. For example, the first ROI may be set to be included in the second and third ROIs, and the second ROI may be set to be included in the third ROI, but is not limited thereto. The first to third regions of interest may be set to at least partially overlap each other.

Also, the first, second, and third regions of interest may be set not to overlap each other. For example, the first region of interest and the second region of interest may be located vertically on the left cheek of the subject, and the third region of interest may be located on the right cheek of the subject, but is not limited thereto. and the first to third ROIs may be set not to overlap each other.

In addition, the detailed operations of the step S2640 of processing data for the first, second, and third ROIs may be applied to the above-described methods for processing data for the ROI, and thus a redundant description will be omitted. .

In addition, the operation of processing the data on the ROI ( S2640 ) may be performed on at least one image frame among a plurality of acquired image frames.

In addition, the operation of processing the data for the ROI ( S2640 ) may be performed for the first, second, and third ROIs.

In addition, IR intensity values for the first to third ROIs may be extracted with respect to at least one image frame among a plurality of image frames obtained to process data for the first to third ROIs. In this case, the IR intensity value may be an average value of IR intensity values of pixels included in the first to third ROIs, and may be referred to as an average pixel value.

In addition, when the above-described method of processing data for an ROI is applied, an IR intensity value for each ROI may correspond to the above-described color channel value. For example, the IR intensity value of the first ROI may correspond to a Red channel value, the IR intensity value of the second ROI may correspond to a Green channel value, and the IR intensity value of the third ROI may be Blue. It may correspond to a channel value, but is not limited thereto.

Also, for example, the above-described G-R value may correspond to a difference between the IR intensity value of the second region of interest and the IR intensity value of the first region of interest, and the above-described G-B value is the IR intensity value of the second region of interest. and a difference value between IR intensity values of the third ROI.

Accordingly, data may be processed based on the IR intensity value for each region of interest, and detailed operations may follow the data processing method for the region of interest described above.

In addition, the method 2600 for measuring a heart rate using infrared rays according to an embodiment includes the steps of extracting time series data with respect to at least a part of an image frame group among a plurality of acquired image frames ( S2650 ) and acquiring a heart rate ( S2660 ) may include at least a portion of, but is not limited thereto.

In this case, the steps of extracting the time-series data ( S2650 ) and obtaining the heart rate ( S2660 ) may be applied to the above description, and thus the redundant description will be omitted.

27 is a diagram for describing a method for acquiring a heart rate using infrared rays according to an exemplary embodiment.

Referring to FIG. 27 , at least two ROIs may be set in the face region of the subject. More specifically, the first ROI 2710 , the second ROI 2720 , and the third ROI 2730 may be set in the face region of the subject, but is not limited thereto.

Also, IR intensity values for the first to third regions of interest 2710 , 2720 , and 2730 may be extracted.

For example, an IR intensity value for the first ROI 2710 (FIG. 27(a)) may be extracted, and an IR intensity value for the second ROI 2720 (FIG. 27(b)) may be extracted. )) may be extracted, and an IR intensity value ((c) of FIG. 27 ) for the third ROI 2730 may be extracted, but is not limited thereto.

In addition, data on the first to third ROIs 2710 , 2720 , and 2730 may be processed based on IR intensity values extracted with respect to the first to third ROIs 2710 , 2720 and 2730 . . However, since the detailed operation thereof has been described above, the overlapping description will be omitted.

Also, a characteristic value may be obtained based on data processed with respect to the first to third ROIs 2710 , 2720 , and 2730 . However, since the detailed operation thereof has been described above, the overlapping description will be omitted.

In addition, a heart rate may be obtained using a characteristic value obtained based on IR intensity values extracted with respect to the first to third ROIs 2710 , 2720 , and 2730 . However, since the detailed operation thereof has been described above, the overlapping description will be omitted.

6. A method for obtaining a biometric index according to an embodiment.

28 is a flowchart illustrating a method for obtaining a biometric index according to an embodiment.

Referring to FIG. 28 , the method for obtaining a biometric index according to an embodiment includes obtaining a plurality of image frames for a subject (S2810), and obtaining first, second, and third color channel values (S2820) , calculating the first difference value and the second difference value (S2830), obtaining the first characteristic value and the second characteristic value (S2840), and the living body of the subject based on the first and second characteristic values It may include at least a part of determining the index (S2850).

In this case, the plurality of images may be acquired by a camera. For example, the plurality of images may be acquired by a camera such as a visible light camera or an IR camera.

In addition, the plurality of images may be acquired from a camera disposed outside. For example, the plurality of images may be images obtained from a camera, such as a visible light camera or an IR camera, which are disposed outside.

Also, the step of obtaining the first, second, and third color channel values ( S2820 ) may be performed on at least one image frame among a plurality of obtained image frames.

In this case, the first color channel value may mean an average pixel value of a first color channel of an image frame in which the step is performed, and the second color channel value is an average of a second color channel of the image frame. may mean a pixel value, and the third color channel value may mean an average pixel value for a third color channel of the image frame.

For example, the first color channel value may be a green channel value, the second color channel value may be a red channel value, and the third color channel value may be a blue channel value, but is not limited thereto. .

Also, the step of calculating the first difference value and the second difference value ( S2830 ) may be performed on at least one image frame among the plurality of acquired image frames.

In this case, the first difference value may mean a difference value between the first color channel value and the second color channel value. For example, the first difference value is the first color channel value and the first color channel value for the same image frame. It may mean a difference value between the second color channel values, but is not limited thereto.

More specifically, when the first color channel value is a green channel value and the second color channel value is a red channel value, the first difference value may be a G-R value, but is not limited thereto.

Also, the second difference value may mean a difference value between the first color channel value and the third color channel value. For example, the second difference value is the first color channel value and the second color channel value for the same image frame. It may mean a difference value between three color channel values, but is not limited thereto.

More specifically, when the first color channel value is a green channel value and the second color channel value is a blue channel value, the second difference value may be a G-B value, but is not limited thereto.

In addition, the step of acquiring the first characteristic value and the second characteristic value ( S2840 ) may be performed on at least some image frame groups among the plurality of acquired image frames.

For example, the first characteristic value may be obtained for a first image frame group, and the first image frame group may mean an image frame group obtained for a preset time.

Also, for example, the second characteristic value may be obtained for a second image frame group, and the second image frame group may mean an image frame group obtained for a preset time.

Also, the first characteristic value may be obtained based on an average value of the first difference values for the first image frame group and a first difference value for the image frames included in the first image frame group.

For example, the first characteristic value may be a deviation value obtained based on the average value of the G-R values for the first image frame group and the G-R values for the image frames included in the first image frame group. not limited

Also, the second characteristic value may be obtained based on an average value of the first difference values for the second image frame group and a second difference value for the image frames included in the second image frame group.

For example, the second characteristic value may be a deviation value obtained based on the average value of the G-B values for the second image frame group and the G-B value for the image frames included in the second image frame group. not limited

Also, the first image frame group and the second image frame group may be identical to each other, but are not limited thereto and may be different from each other.

Also, the first characteristic value may be an average value or a standard deviation value for the first image frame group, but is not limited thereto.

Also, the first characteristic value may be a deviation value with respect to at least some image frames included in the first image frame group, but is not limited thereto.

In addition, the second characteristic value may be an average value or a standard deviation value for the second image frame group, but is not limited thereto.

Also, the second characteristic value may be a deviation value with respect to at least some image frames included in the second image frame group, but is not limited thereto.

Also, the first and second characteristic values may be normalized. For example, the first characteristic value may be normalized using a standard deviation value for the first image frame group, but is not limited thereto.

Also, for example, the second characteristic value may be normalized using a standard deviation value for the second image frame group, but is not limited thereto.

Also, the step of determining the biometric index of the subject based on the first and second characteristic values ( S2850 ) may be performed on at least some image frame group among the plurality of acquired image frames.

In this case, the biometric index may be a heart rate, blood pressure, oxygen saturation, body temperature, etc., but is not limited thereto.

Also, the biometric index may be obtained based on the first characteristic value and the second characteristic value.

For example, the biometric index may be obtained based on a third characteristic value obtained by adding the first characteristic value and the second characteristic value, but is not limited thereto.

7. Various embodiments of a plurality of biometric indices and methods of obtaining biometric information

29 is a view for explaining a method for obtaining a plurality of biometric indices and biometric information according to an embodiment.

Referring to FIG. 29 , the method for obtaining a plurality of biometric indices and biometric information according to an embodiment may acquire a plurality of biometric indices based on the acquired image 3000 . More specifically, as shown in FIG. 29 , the first biometric index 3010 , the second biometric index 3020 , and the third biometric index 3030 may be obtained based on the acquired image 3000 , but this Without limitation, at least two or more biometric indices may be obtained, for example, four biometric indices may be obtained.

In this case, the obtained image 3000 may be a visible light image obtained by using a visible light camera or a visible light camera disposed outside.

Also, the acquired image 3000 may be an infrared image acquired using an infrared camera or an infrared camera disposed outside.

In addition, the first to third biometric indexes 3010 , 3020 , and 3030 may include at least one of heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

Also, the first to third biometric indices 3010 , 3020 , and 3030 may be obtained to be correlated with each other. For example, the first to third biometric indices 3010 , 3020 , and 3030 may be biometrics obtained from the same or similar state of the subject, but is not limited thereto.

In addition, the first to third biometric indices 3010 , 3020 , and 3030 may be obtained to be mutually independent. For example, the first to third biometric indices 3010 , 3020 , and 3030 may be biometrics obtained from different states of the subject, but is not limited thereto.

Also, the first to third biometric indexes 3010 , 3020 , and 3030 may be output at the same time. For example, when the first to third biometric indexes 3010 , 3020 , and 3030 are measured in real time, they may be output at the same time as each other, but the present invention is not limited thereto.

In addition, the first to third biometric indexes 3010 , 3020 , and 3030 may be output at different times. For example, the second biometric index may be output after the first biometric index is output, and the third biometric index may be output after the second biometric index is output.

Also, referring to FIG. 29 , a plurality of biometric indices and a method for obtaining biometric information according to an embodiment may acquire a plurality of biometric information based on the obtained plurality of biometrics.

More specifically, as shown in FIG. 29 , based on the obtained first to third biometric indices 3010 , 3020 , and 3030 , the first biometric information 3040 , the second biometric information 3050 , and the third biometric information Information 3060 may be acquired, but is not limited thereto.

In this case, the first to third biometric information 3040 , 3050 , and 3060 may include at least one of drowsiness information, stress information, excitement information, and emotion information, but is not limited thereto.

In addition, the first to third biometric information 3040, 3050, and 3060 may be obtained based on at least one of the first to third biometric indices 3010, 3020, and 3030, but is not limited thereto. does not

In addition, the first to third biometric information (3040, 3050, 3060) may be obtained in consideration of personal and statistical data in addition to the first to third biometric indices (3010, 3020, 3030), at this time, The personal and statistical data may mean collectible personal and statistical data of the subject, such as age, gender, height, and weight, and may refer to observable personal and statistical data such as facial expressions, wrinkles, and facial color of the subject , statistical data calculated for groups that include or are related to the subject (e.g., average blood pressure in their 20s, average skin color in yellow people, average height of men in their 30s, average weight of Korean men, etc.) It can also mean personal and statistical data that can be

Also, the first to third biometric information 3040 , 3050 , and 3060 may be obtained based on independent biometric indices. For example, the first to third biometric information 3040 , 3050 , and 3060 may be obtained based on the first to third biometric indices 3010 , 3020 , and 3030 obtained to be independent.

Also, the first to third biometric information 3040 , 3050 , and 3060 may be obtained based on related biometric indices. For example, the first to third biometric information 3040 , 3050 , and 3060 may be obtained based on the first to third biometric indices 3010 , 3020 , and 3030 obtained to be related.

In addition, when the first to third biometric information 3040 , 3050 , and 3060 are obtained based on related biometric indices, the accuracy of the biometric information may be improved. For example, in order to obtain hypertension information that may be included in the first to third biometric information 3040 , 3050 , and 3060 , a heart rate that may be included in the first to third biometric indices 3010 , 3020 , and 3030 . and when blood pressure is used, the hypertension information may be more accurate when the heart rate and blood pressure are correlated with each other.

More specifically, when blood pressure is measured when the subject is exercising or excited, and when the subject measures heart rate in a stable state and hypertension information is obtained based on this, the possibility of detecting high blood pressure even if the subject is not hypertensive There is this. On the other hand, when the subject is exercising or excited, blood pressure and heart rate are measured and hypertension information is obtained based on the measurement, the hypertension information can be more accurately obtained.

Hereinafter, a plurality of biometric indices and a method of obtaining biometric information will be described in more detail.

30 is a diagram for explaining a method for obtaining a plurality of biometrics according to an embodiment.

Referring to FIG. 30 , according to an exemplary embodiment, an image frame 3110 may be obtained to obtain a plurality of biometric indices.

In this case, the image frame 3110 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, the image frame 3110 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 30 , according to an embodiment, at least one pixel value 3120 may be obtained to obtain a plurality of biometric indices.

In this case, the pixel value 3120 may mean an intensity value of at least one pixel constituting the image frame, and more specifically, may mean an intensity value of at least one pixel for at least one color channel. have. For example, when the image frame is a 640*480 image, the image frame may include 640*480 pixels, and a red channel pixel value, a green channel pixel value, and a blue channel pixel value are obtained for each pixel. However, the present invention is not limited thereto, and pixel values for color channels according to various color spaces may be obtained.

Also, the pixel value 3120 may be obtained only with respect to at least some of the plurality of pixels constituting the image frame.

Also, referring to FIG. 30 , according to an embodiment, at least one color channel value 3130 may be obtained to obtain a plurality of biometric indices.

In this case, the color channel value 3130 may mean an average value of color channel pixel values. For example, the red channel value may mean the average value of the red channel pixel values, the green channel value may mean the average value of the green channel pixel values, and the blue channel value may represent the average value of the blue channel pixel values. may mean, but is not limited thereto.

Although the color channel value and the processed value have been separately described throughout the specification, in order to explain a method of obtaining a plurality of biometrics, the processed value may be described as a concept included in the color channel value. .

Also, the color channel value 3130 may be described including a processing value. For example, the color channel value may be described as including a G-R value that is a difference value between a red channel value and a green channel value, but is not limited thereto.

Also, the color channel value 3130 may be obtained for at least one image frame among a plurality of obtained image frames.

Also, the color channel value 3130 may be obtained based on one color channel pixel value or may be obtained based on a plurality of color channel pixel values. For example, the red channel value may be obtained based on the red channel pixel value, and the G-R value may be obtained based on the green channel pixel value and the red channel pixel value, but is not limited thereto.

Also, referring to FIG. 30 , according to an embodiment, at least one or more time series data may be acquired to acquire a plurality of biometric indices.

In this case, the time series data 3140 may be obtained based on the obtained at least one or more color channel values 3130 , and may be obtained for at least some image frame groups among a plurality of obtained image frames.

More specifically, when the time series data 3140 is obtained based on at least one color channel value 3130 obtained for a first image frame, the time series data is an image frame group including the first image frame. can be obtained for

Also, the time series data 3140 may be time series data of the color channel value 3130 . For example, the time series data 3140 may include time series data of a red channel value, time series data of a green channel value, time series data of a blue channel value, time series data of a Hue channel value, time series data of a G-R value, or time series data of a G-B value. and the like, but is not limited thereto.

Also, it may be time series data 3140 of a characteristic value obtained based on the color channel value 3130 . For example, the time series data may include, but is not limited to, a deviation value, a standard deviation value, or an average value of color channel values for at least some image frames included in the image frame group.

For example, the time series data 3140 may be time series data of a red channel value or time series data of a G-R value, but is not limited thereto.

Also, for example, the time series data 3140 may be time series data regarding deviation values of color channel values for at least some image frames included in the image frame group, but is not limited thereto.

Also, the time series data 3140 may be time series data obtained based on a plurality of time series data. For example, the time series data 3140 is based on first time series data for a first image frame group including a first image frame and second time series data for a second image frame group including a second image frame It may be time series data obtained by , but is not limited thereto.

As a more specific example, when the time series data 3140 is time series data for an image frame group including 180 image frames, the time series data 3140 is a first image including first to 18th image frames. The first time series data obtained for the frame group, the second time series data obtained with the second image frame group including the 19th to 36th image frames, and the third image frame group including the 37th to 54th image frames With respect to the third time series data obtained for , the fourth time series data obtained with respect to the fourth image frame group including the 55th to 72nd image frames, and the fifth image frame group including the 73rd to the 90th image frames The obtained fifth time series data, the sixth time series data obtained for the sixth image frame group including the 91st to 108th image frames, and the 7th image frame group including the 109th to the 126th image frame The seventh time series data, the eighth time series data obtained with respect to the eighth image frame group including the 127th to 144th image frames, and the ninth obtained with respect to the ninth image frame group including the 145th to the 162th image frame It may be obtained based on the time series data and the tenth time series data obtained with respect to the tenth image frame group including the 163rd to 180th image frames.

Also, referring to FIG. 30 , according to an embodiment, a feature 3150 may be obtained to obtain a plurality of biometric indices.

In this case, the feature 3150 may mean mathematical and physical features of the acquired time series data. For example, the characteristics may include a frequency component of the obtained time series data, a maximum value, an average of maximum values, a minimum value, an average of minimum values, a difference between a maximum value and a minimum value, a difference between a maximum value average and a minimum value average, an AC value, and a DC Value, mean value, inflection point, first-differential data, second-differential data, can mean mathematical characteristics such as slope at a specific point in time, and physical characteristics such as blood change amount, blood change rate, blood vessel change amount, blood vessel change rate, etc. It may mean a feature, but is not limited thereto.

Also, the feature 3150 may mean mathematical and physical features between a plurality of time series data. For example, the characteristic may mean a mathematical characteristic such as a time difference between a plurality of time series data obtained, a time difference between maxima, a time difference between minima, and a time difference between inflection points, PTT (Pulse Transit Time) , may mean physical characteristics such as a difference in the rate of change in blood, a difference in time of change in blood vessels due to blood, etc., but is not limited thereto.

In addition, the feature 3150 may be acquired for at least some image frame groups among a plurality of acquired image frames.

Also, the feature 3150 may be obtained based on at least one time series data 3140 . For example, the feature 3150 may be obtained based on one time series data or may be obtained based on at least two or more time series data, but is not limited thereto.

Also, referring to FIG. 30 , a biometric index 3160 may be obtained to obtain a plurality of biometric indices according to an embodiment.

In this case, the biometric index 3160 may include, but is not limited to, heart rate, oxygen saturation, blood pressure, body temperature, and the like.

Also, the biometric index 3160 may be obtained based on different characteristics. For example, heart rate may be obtained based on a frequency component of time series data, blood pressure may be obtained based on a PTT value between two time series data, and oxygen saturation may be obtained based on an AC value and DC value of each of the two time series data It may be obtained based on the value, but is not limited thereto.

In addition, a plurality of the biometric index 3160 may be obtained. For example, heart rate, oxygen saturation, and blood pressure may be acquired, but the present invention is not limited thereto.

Also, the biometric index 3160 may be obtained for at least some image frame groups among a plurality of obtained image frames.

In addition, image frame groups that are a basis for obtaining the plurality of biometric indexes 3160 may be identical to each other. For example, the heart rate, oxygen saturation, and blood pressure may be acquired based on an image frame group including the first to 180th image frames, but is not limited thereto.

In addition, image frame groups used as a basis for obtaining the plurality of biometric indexes 3160 may be different from each other. For example, the heart rate may be obtained based on a first image frame group including first to 90th image frames, and oxygen saturation may be obtained based on a second image frame group including 91st to 180th image frames. may be obtained, and the blood pressure may be obtained based on the third image frame group including the 181st to 270th image frames, but is not limited thereto.

In addition, image frame groups that are a basis for obtaining the plurality of biometric indexes 3160 may overlap each other at least partially. For example, the heart rate may be acquired based on a first image frame group including first to 180th image frames, and oxygen saturation may be obtained based on a second image frame group including 30 to 100th image frames. may be obtained, and the shale may be obtained based on a third image frame group including tenth to 180th image frames, but is not limited thereto.

In addition, the number of image frames included in the image frame group which is a basis for obtaining the plurality of biometric indexes 3160 may be the same or different.

7.1 Method of obtaining a plurality of biometric indexes according to an embodiment

31 is a diagram for explaining a method of obtaining a plurality of biometrics according to an embodiment.

Referring to FIG. 31 , according to an exemplary embodiment, an image frame 3210 may be obtained to obtain a plurality of biometric indices.

In this case, the image frame 3210 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, the image frame 3210 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 31 , according to an embodiment, at least one ROI 3220 may be set to obtain a plurality of biometric indices. More specifically, a first ROI, a second ROI, a third ROI, and a fourth ROI may be set.

In this case, the first region of interest may be a region of interest for acquiring oxygen saturation, the second region of interest may be a region of interest for acquiring a heart rate, and the third region of interest may be a region of interest for acquiring blood pressure region, and the fourth region of interest may be a region of interest for acquiring body temperature, but is not limited thereto.

Also, the first to fourth regions of interest may be identical to or different from each other.

Also, the first to fourth regions of interest may overlap each other at least partially.

Also, the sizes and regions of the first to fourth regions of interest may be set based on a biometric index to be acquired. For example, the second region of interest for acquiring the heart rate may be set to be large to include the cheek region of the subject in order to well detect a change in blood due to the heartbeat, and the third region of interest for acquiring the blood pressure The region may be set vertically small and horizontally long to include the cheek region of the subject in order to detect a fine blood flow velocity well, but the present invention is not limited thereto, and the first to fourth regions of interest may be set to various sizes and various regions. can

Also, referring to FIG. 31 , according to an exemplary embodiment, at least one pixel value 3230 may be obtained to obtain a plurality of biometric indices.

Also, the pixel value 3230 may be obtained for at least one image frame among a plurality of obtained image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained with respect to the first region of interest, and the second region of interest is converted into an RGB color space for the second region of interest. At least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value may be obtained according to At least a part of the ROI may be obtained, and a Hue channel pixel value, a saturation channel pixel value, and a Value channel pixel value according to the HSV color space may be obtained for the fourth region of interest, but is not limited thereto.

Also, referring to FIG. 31 , according to an embodiment, at least one color channel value 3240 may be obtained to obtain a plurality of biometric indices.

In this case, the color channel value 3240 may mean an average value of color channel pixel values or may mean a processed value. However, since the detailed description thereof has been described above, the overlapping description will be omitted.

Also, the color channel value 3240 may be obtained in consideration of the biometric to be obtained.

More specifically, at least two color channel values may be obtained for the first ROI for obtaining oxygen saturation.

For example, with respect to the first region of interest for obtaining oxygen saturation, a red channel value, a green channel value, a blue channel value, a Hue channel value according to an HSV color space, a saturation channel value, and a value channel value according to the RGB color space for the first region of interest At least two color channel values may be obtained, but the present invention is not limited thereto.

Also, for example, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and a Hue channel value and a value channel for the first region of interest for obtaining oxygen saturation At least two color channel values among color channel values such as an H-V value that is a value difference value may be obtained, but the present invention is not limited thereto.

In addition, the at least two color channel values may be selected in consideration of absorbance of hemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin and a red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. It may be selected as a blue channel value, but is not limited thereto.

Also, at least two color channel values may be obtained for the second region of interest for obtaining a heart rate.

For example, with respect to the second region of interest for acquiring the heart rate, a red channel value, a green channel value, a blue channel value, a Hue channel value, a saturation channel value, a Value channel value, etc. according to the RGB color space At least two color channel values of the color channel values of ? may be obtained, but the present invention is not limited thereto.

Also, for example, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and a Hue channel value and a Value channel value for the second region of interest for obtaining a heart rate At least two color channel values among color channel values, such as an H-V value that is a difference value of , may be obtained, but the present invention is not limited thereto.

Also, the at least two color channel values may be selected to reduce noise caused by motion and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, the G-R value, and the A G-B value that is a difference value between a green channel value, which is absorbed by oxyhemoglobin, and a blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, at least two color channel values may be obtained for the third ROI for obtaining blood pressure.

For example, with respect to the third region of interest for acquiring blood pressure, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to an HSV color space, a saturation channel value, a value channel value, etc. At least two color channel values of the color channel values of ? may be obtained, but the present invention is not limited thereto.

Also, for example, with respect to the third region of interest for obtaining blood pressure, a G-R value that is a difference between a red channel value and a green channel value, a G-B value that is a difference between a green channel value and a blue channel value, and a Hue channel value and a Value channel value At least two color channel values among color channel values, such as an H-V value that is a difference value of , may be obtained, but the present invention is not limited thereto.

Also, the at least two color channel values may be selected to reduce noise caused by motion and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin. A G-B value, which is a difference value between a Green channel value, which is absorbed by oxyhemoglobin, and a Blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, at least two color channel values may be obtained for the fourth ROI for obtaining body temperature.

For example, with respect to the fourth region of interest for acquiring body temperature, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to an HSV color space, a saturation channel value, a value channel value, etc. At least two color channel values of the color channel values of ? may be obtained, but the present invention is not limited thereto.

Also, for example, with respect to the fourth region of interest for acquiring body temperature, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and a Hue channel value and a Value channel value At least two color channel values among color channel values, such as an H-V value that is a difference value of , may be obtained, but the present invention is not limited thereto.

In addition, the at least two color channel values may be selected in consideration of the characteristics of body temperature and the skin color of the subject.

For example, a saturation channel value related to the body temperature of the subject and a Hue channel value related to the skin color of the subject may be selected, but the present invention is not limited thereto.

Also, referring to FIG. 31 , according to an exemplary embodiment, at least one or more time series data 3250 may be obtained to obtain a plurality of biometric indices.

Also, the time series data 3250 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least two time series data may be acquired for the first ROI for acquiring oxygen saturation. For example, first time series data and second time series data may be obtained with respect to the first ROI for obtaining oxygen saturation.

In this case, the first time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first ROI for obtaining oxygen saturation is a red channel value, the first time series data may be obtained based on the red channel value, but is not limited thereto.

In addition, the first time-series data may be acquired for at least some image frame groups among a plurality of acquired image frames.

For example, when the first time series data is obtained based on a Red channel value obtained for a first image frame, the first time series data is obtained for a first image frame group including the first image frame. However, the present invention is not limited thereto.

Also, the second time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first region of interest for obtaining oxygen saturation is a blue channel value, the second time series data may be obtained based on the blue channel value, but is not limited thereto.

Also, the second time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the second time series data is obtained based on a Blue channel value obtained for a second image frame, the second time series data may be obtained for a second image frame group including the second image frame. However, the present invention is not limited thereto.

In addition, the first and second image frame groups may be identical to each other, different from each other, and may overlap each other at least partially.

In addition, at least one time series data may be acquired with respect to the second ROI for acquiring a heart rate. For example, third time series data may be obtained with respect to the second ROI for obtaining a heart rate.

In this case, the third time series data may be obtained based on the color channel value obtained for the second ROI. For example, if the color channel values obtained for the second region of interest for obtaining the heart rate are G-R values and G-B values, the third time series data may be obtained based on the G-R values and G-B values, but limited thereto. doesn't happen

Also, the third time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the third time series data is obtained based on the G-R value and the G-B value obtained for the third image frame, the third time series data is the third image frame group including the third image frame. may be obtained, but is not limited thereto.

Also, the third time series data may be obtained based on a characteristic value obtained with respect to the second ROI. For example, the third time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the second region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

In addition, at least one time series data may be acquired with respect to the third ROI for acquiring blood pressure. For example, fourth time series data may be obtained with respect to the third ROI for obtaining blood pressure.

In this case, the fourth time series data may be obtained based on a color channel value obtained for the third ROI. For example, when the color channel values obtained for the third region of interest for obtaining blood pressure are G-R values and G-B values, the fourth time series data may be obtained based on the G-R values and the G-B values, but is not limited thereto. does not

Also, the fourth time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the fourth time series data is obtained based on the G-R value and the G-B value obtained for the fourth image frame, the fourth time series data is the fourth time series data for a fourth image frame group including the fourth image frame may be obtained, but is not limited thereto.

Also, the fourth time series data may be obtained based on a characteristic value obtained with respect to the third ROI. For example, the fourth time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the third region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

In addition, the first, second, third, and fourth image frame groups may be the same, but not limited thereto, may be different from each other, and may overlap each other at least partially.

Also, referring to FIG. 31 , according to an embodiment, at least one biometric index acquisition model 3251 may be used to acquire a plurality of biometric indices.

However, with respect to the biometric index acquisition model 3251, descriptions that overlap with the above description will be omitted.

Also, the biometric index acquisition model 3251 may be used to acquire body temperature.

More specifically, the biometric index acquisition model 3251 may use a color channel value acquired for the fourth ROI for acquiring body temperature as an input value. For example, the biometric index acquisition model 3251 may use a Hue channel value and a saturation channel value acquired with respect to the fourth ROI as input values, but is not limited thereto.

Also, the input value of the biometric index acquisition model 3251 may be a color channel value, a characteristic value, an average of color channel values for an image frame group, and the like, but is not limited thereto.

Also, referring to FIG. 31 , according to an exemplary embodiment, at least one feature 3260 may be obtained to obtain a plurality of biometric indices.

In addition, at least one or more of the features 3260 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least one characteristic may be acquired with respect to the first region of interest for acquiring oxygen saturation. For example, a first characteristic may be obtained with respect to the first region of interest for obtaining oxygen saturation.

In this case, the first characteristic may be acquired based on a color channel value or time series data acquired for the first ROI. For example, when the color channel values obtained for the first region of interest are a red channel value and a blue channel value, the first characteristic may be obtained based on the red channel value and the blue channel value, but limited thereto. doesn't happen

Also, for example, the first characteristic may be obtained based on first time series data and second time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the first characteristic may be a characteristic for obtaining oxygen saturation. For example, the first characteristic may be an AC value and a DC value obtained based on the first time series data, an AC value and a DC value obtained based on the second time series data, but is not limited thereto.

In addition, for example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data based on It may include, but is not limited to, at least one of a difference between the obtained local maximum average and the minimum minimum value average and an average value obtained based on the second time series data.

Also, the first feature may include a plurality of features. For example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data obtained based on At least two or more of a difference between the mean of the local maximum and the mean of the minimum value and an average value obtained based on the second time series data may be included, but the present invention is not limited thereto.

Also, at least one characteristic may be acquired with respect to the second ROI for acquiring a heart rate. For example, a second characteristic may be obtained for the second region of interest for obtaining a heart rate.

In this case, the second characteristic may be acquired based on a color channel value or time series data acquired for the second ROI. For example, when the color channel values obtained for the second region of interest are a G-R value and a G-B value, the second characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the second characteristic may be obtained based on third time series data obtained with respect to the second ROI, but is not limited thereto.

Also, the second characteristic may be a characteristic for acquiring a heart rate. For example, the second characteristic may be a frequency value obtained based on the third time series data, a wavelength value, and a value corresponding to the number of repeated cycles during the measurement time, but is not limited thereto.

In addition, at least one or more features may be acquired with respect to the third ROI for acquiring blood pressure. For example, a third characteristic may be obtained for the third region of interest for obtaining blood pressure.

In this case, the third characteristic may be acquired based on a color channel value or time series data acquired for the third ROI. For example, when the color channel values obtained for the third region of interest are a G-R value and a G-B value, the third characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the third characteristic may be acquired based on fourth time series data acquired with respect to the third ROI, but is not limited thereto.

Also, the third characteristic may be a characteristic for acquiring blood pressure. For example, the third characteristic may be a slope value, a maximum value, a minimum value, an average of a maximum value, an average of a minimum value, a difference value between the average of the maximum value and the average of the minimum value obtained based on the fourth time series data, etc. , but is not limited thereto.

Also, the third feature may include a plurality of features. For example, the third characteristic is a slope value, a maximum value, a minimum value, the average of the maximum values, the average of the minimum values, the average of the minimum values, the average of the maximum values, and the average of the minimum values obtained based on the fourth time series data. may be included, but is not limited thereto.

Also, at least one characteristic may be acquired with respect to the fourth region of interest for acquiring body temperature. For example, a fourth characteristic may be acquired with respect to the fourth region of interest for acquiring body temperature.

In this case, the fourth characteristic may be acquired based on a color channel value acquired with respect to the fourth region of interest or an output value of a biometric index acquisition model. For example, when the color channel value obtained for the fourth region of interest is a Hue channel value and a saturation channel value, the fourth characteristic may be obtained based on the Hue channel value and the saturation value, but is not limited thereto. does not

Also, for example, the fourth characteristic may be acquired based on an output value of the biometric index acquisition model, but is not limited thereto.

Also, the fourth characteristic may be a characteristic for acquiring body temperature. For example, the fourth characteristic may be a skin area, skin temperature, etc., but is not limited thereto.

Also, referring to FIG. 31 , according to an embodiment, at least one biometric index 3270 may be acquired to obtain a plurality of biometric indices.

More specifically, each biometric index may be obtained based on a corresponding characteristic. For example, oxygen saturation may be obtained based on the first characteristic, heart rate may be obtained based on the second characteristic, blood pressure may be obtained based on the third characteristic, and body temperature may be It may be obtained based on the fourth characteristic, but is not limited thereto.

In this case, the above-described equations may be used to obtain oxygen saturation based on the first characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the heart rate based on the second characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the blood pressure based on the third characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the body temperature based on the fourth characteristic, and overlapping descriptions will be omitted.

In addition, when there are a plurality of biometric indices 3270 , each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be acquired 6 seconds after measurement, and blood pressure may be acquired 8 seconds after measurement, but is not limited thereto.

In addition, when there are a plurality of biometric indexes 3270 , each biometric index may be obtained based on each image frame group. For example, the oxygen saturation may be obtained based on the fifth image frame group, the heart rate may be obtained based on the sixth image frame group, and the blood pressure may be obtained based on the seventh image frame group.

In addition, each image frame group for obtaining the biometric index 3270 may be the same, but is not limited thereto, may be different from each other, and may overlap each other at least partially. For example, the fifth, sixth, and seventh image frame groups may be the same as each other, different from each other, and at least partially overlap each other.

Also, at least one preliminary biometric index may be obtained to obtain the biometric index 3270 . For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

However, since the above description may be applied to a method of acquiring the heart rate or bio-index using the preliminary heart rate or the preliminary bio-index, the overlapping description will be omitted.

In addition, the number of preliminary bio-indexes for obtaining the bio-index 3270 may be the same for each bio-index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the preliminary oxygen saturation and preliminary blood pressure for acquiring the oxygen saturation and blood pressure may be at least two, but is not limited thereto.

7.2 Method of obtaining a plurality of biometric indexes according to an embodiment

32 is a diagram for explaining a method for obtaining a plurality of biometrics according to an embodiment.

Referring to FIG. 32 , according to an exemplary embodiment, an image frame 3310 may be obtained to obtain a plurality of biometric indices.

In this case, the image frame 3310 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, the image frame 3310 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 32 , according to an embodiment, at least one region of interest 3320 may be set to acquire a plurality of biometric indices. More specifically, a first region of interest, a second region of interest, and a third region of interest may be set.

In this case, the first region of interest may be a region of interest for acquiring oxygen saturation and heart rate, and the second region of interest and the third region of interest may be regions of interest for acquiring blood pressure.

Also, the first to third regions of interest may be identical to or different from each other.

Also, the first to third regions of interest may overlap each other at least partially.

Also, the sizes and regions of the first to third regions of interest may be set based on the biometric index to be acquired. For example, in order to acquire heart rate and oxygen saturation, the size of the first region of interest may be set to include a cheek region of a subject in order to well detect changes in blood due to heartbeat, and to acquire blood pressure The regions of the second and third regions of interest may be set along the direction of blood flow in order to reflect the flow of blood flow, but the present invention is not limited thereto, and the first to third regions of interest may be set to have various sizes and various regions. can

Also, referring to FIG. 32 , according to an embodiment, at least one pixel value 3330 may be obtained to obtain a plurality of biometric indices.

Also, the pixel value 3330 may be obtained for at least one image frame among a plurality of obtained image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained with respect to the first region of interest, and the second region of interest is converted into an RGB color space for the second region of interest. At least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value may be obtained according to At least some of them may be obtained, but the present invention is not limited thereto.

Also, referring to FIG. 32 , according to an exemplary embodiment, at least one color channel value 3340 may be obtained to obtain a plurality of biometric indices.

In this case, the color channel value 3340 may mean an average value of color channel pixel values or may mean a processed value. However, since the detailed description thereof has been described above, the overlapping description will be omitted.

Also, the color channel value 3340 may be obtained in consideration of the biometric to be obtained.

More specifically, at least two color channel values may be acquired for the first ROI for acquiring oxygen saturation and heart rate.

For example, among color channel values such as a red channel value, a green channel value, a blue channel value, a Hue channel value, a saturation channel value, and a Value channel value, according to the RGB color space for the first region of interest At least two or more color channel values may be obtained, but the present invention is not limited thereto.

Also, for example, for the first region of interest, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and an H-V value that is a difference value between the Hue channel value and the Value channel value At least two or more color channel values may be obtained from among the color channel values, but the present invention is not limited thereto.

In addition, in order to obtain oxygen saturation, the at least two color channel values may be selected in consideration of absorbance of hemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin and a red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. It may be selected as a blue channel value, but is not limited thereto.

In addition, in order to obtain a heart rate, the at least two color channel values may be selected to reduce noise caused by movement and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, the G-R value, and the A G-B value that is a difference value between a green channel value, which is absorbed by oxyhemoglobin, and a blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, at least two color channel values may be obtained for the second and third ROIs for obtaining blood pressure.

For example, with respect to the second and third regions of interest for obtaining blood pressure, a red channel value, a green channel value, a blue channel value, a Hue channel value, a saturation channel value, and a value according to the HSV color space for the second and third regions of interest At least two color channel values among color channel values such as channel values may be obtained, but the present invention is not limited thereto.

Also, for example, for the second and third regions of interest for obtaining blood pressure, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and a Hue channel value and At least two color channel values among color channel values such as an H-V value that is a difference value between Value channel values may be obtained, but the present invention is not limited thereto.

Also, the at least two color channel values may be selected to reduce noise caused by motion and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin. A G-B value, which is a difference value between a green channel value, which is absorbed by oxyhemoglobin, and a blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, referring to FIG. 32 , according to an embodiment, at least one or more time series data 3350 may be acquired to acquire a plurality of biometric indices.

Also, the time series data 3350 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least two time series data may be acquired with respect to the first ROI for acquiring oxygen saturation and heart rate. For example, first time series data and second time series data may be obtained to obtain oxygen saturation for the first ROI, and third time series data may be obtained to obtain a heart rate.

In this case, the first time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first ROI is a red channel value, the first time series data may be obtained based on the red channel value, but is not limited thereto.

In addition, the first time-series data may be acquired for at least some image frame groups among a plurality of acquired image frames.

For example, when the first time series data is obtained based on a Red channel value obtained for a first image frame, the first time series data is obtained for a first image frame group including the first image frame. However, the present invention is not limited thereto.

Also, the second time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first region of interest is a blue channel value, the second time series data may be obtained based on the blue channel value, but is not limited thereto.

Also, the second time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the second time series data is obtained based on a Blue channel value obtained for a second image frame, the second time series data may be obtained for a second image frame group including the second image frame. However, the present invention is not limited thereto.

Also, the third time series data may be obtained based on a color channel value obtained for the first region of interest. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the third time series data may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, the third time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the third time series data is obtained based on the G-R value and the G-B value obtained for the third image frame, the third time series data is a third image frame group including the third image frame. may be obtained for, but is not limited to.

Also, the third time series data may be obtained based on a characteristic value obtained with respect to the first ROI. For example, the third time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the first region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

Also, the first, second, and third image frame groups may be the same as, but not limited to, may be different from each other, and may overlap each other at least partially.

In addition, at least one time series data may be acquired for each of the second ROI and the third ROI to obtain blood pressure. For example, fourth time series data may be obtained with respect to the second ROI, and fifth time series data may be obtained with respect to the third ROI, but is not limited thereto.

In this case, the fourth time series data may be obtained based on the color channel value obtained for the second ROI. For example, when the color channel values obtained for the second region of interest are a G-R value and a G-B value, the fourth time series data may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, the fourth time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the fourth time series data is obtained based on the G-R value and the G-B value obtained for the fourth image frame, the fourth time series data is the fourth time series data for a fourth image frame group including the fourth image frame may be obtained, but is not limited thereto.

Also, the fourth time series data may be obtained based on a characteristic value obtained with respect to the second ROI. For example, the fourth time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the second region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

Also, the fifth time series data may be obtained based on a color channel value obtained for the third ROI. For example, when the color channel values obtained for the third region of interest are a G-R value and a G-B value, the fifth time series data may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, the fifth time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the fifth time series data is obtained based on the G-R value and the G-B value obtained for the fifth image frame, the fifth time series data is the fifth image frame group including the fifth image frame. may be obtained, but is not limited thereto.

Also, the fifth time series data may be obtained based on a characteristic value obtained with respect to the third ROI. For example, the fifth time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the third region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

In addition, the first, second, third, fourth, and fifth image frame groups may be the same, but not limited thereto, may be different from each other, and may overlap each other at least partially.

Also, referring to FIG. 32 , according to an embodiment, at least one feature 3360 may be obtained to obtain a plurality of biometric indices.

In addition, at least one or more of the features 3360 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least one characteristic may be acquired with respect to the first region of interest for acquiring oxygen saturation and heart rate. For example, a first characteristic of the first region of interest may be obtained to obtain oxygen saturation, and a second characteristic of the first region of interest may be obtained to obtain a heart rate.

In this case, the first characteristic may be acquired based on a color channel value or time series data acquired for the first ROI. For example, when the color channel values obtained for the first region of interest are a red channel value and a blue channel value, the first characteristic may be obtained based on the red channel value and the blue channel value, but limited thereto. doesn't happen

Also, for example, the first characteristic may be obtained based on first time series data and second time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the first characteristic may be a characteristic for obtaining oxygen saturation. For example, the first characteristic may be an AC value and a DC value obtained based on the first time series data, an AC value and a DC value obtained based on the second time series data, but is not limited thereto.

In addition, for example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data based on It may include, but is not limited to, at least one of a difference between the obtained local maximum average and the minimum minimum value average and an average value obtained based on the second time series data.

Also, the first feature may include a plurality of features. For example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data obtained based on It may include at least two or more of a difference between the mean of the local maximum and the mean of the minimum value and an average value obtained based on the second time series data, but is not limited thereto.

Also, the second characteristic may be obtained based on a color channel value or time series data obtained with respect to the first region of interest. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the second characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the second characteristic may be obtained based on third time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the second characteristic may be a characteristic for acquiring a heart rate. For example, the second characteristic may be a frequency value obtained based on the third time series data, a wavelength value, and a value corresponding to the number of repeated cycles during the measurement time, but is not limited thereto.

Also, at least one or more characteristics may be acquired with respect to the second and third regions of interest for acquiring blood pressure. For example, a third characteristic may be obtained with respect to the second and third regions of interest to obtain blood pressure.

In this case, the third characteristic may be acquired based on color channel values or time series data acquired for the second and third regions of interest. For example, when the color channel values obtained for the second and third regions of interest are a G-R value and a G-B value, the third characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto. does not

Also, for example, the third characteristic may be acquired based on fourth time series data acquired with respect to the second ROI and fifth time series data acquired with respect to the third ROI, but is not limited thereto.

Also, the third characteristic may be a characteristic for acquiring blood pressure. For example, the third characteristic may include a slope value, a maximum value, a minimum value, an average of a maximum value, an average of a minimum value, a difference value between the average of the maximum value and the average of the minimum value obtained based on the fourth and fifth time series data, etc. may be, but is not limited thereto.

Also, for example, the third characteristic may be a time difference between the fourth and fifth time series data, a time difference between maxima, a time difference between minima, and a time difference between inflection points, but is not limited thereto. does not

Also, the third feature may include a plurality of features. For example, the third characteristic may include at least two or more of the above-described characteristics, but is not limited thereto.

Also, referring to FIG. 32 , according to an embodiment, at least one biometric index 3370 may be obtained to obtain a plurality of biometric indices.

More specifically, each biometric index may be obtained based on a corresponding characteristic. For example, the oxygen saturation may be obtained based on the first characteristic, the heart rate may be obtained based on the second characteristic, and the blood pressure may be obtained based on the third characteristic, but is not limited thereto. does not

In this case, the above-described equations may be used to obtain oxygen saturation based on the first characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the heart rate based on the second characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the blood pressure based on the third characteristic, and overlapping descriptions will be omitted.

In addition, when there are a plurality of biometric indices 3370 , each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be acquired 6 seconds after measurement, and blood pressure may be acquired 8 seconds after measurement, but is not limited thereto.

Also, when there are a plurality of biometric indexes 3370 , each biometric index may be obtained based on each image frame group. For example, oxygen saturation may be obtained based on the sixth image frame group, heart rate may be obtained based on the seventh image frame group, and blood pressure may be obtained based on the eighth image frame group.

In addition, each image frame group for obtaining the biometric index 3370 may be the same, but is not limited thereto, may be different from each other, and may overlap each other at least partially. For example, the sixth, seventh, and eighth image frame groups may be the same as each other, different from each other, and may overlap each other at least partially.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3370 . For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

However, since the above description may be applied to a method of acquiring the heart rate or bio-index using the preliminary heart rate or the preliminary bio-index, the overlapping description will be omitted.

In addition, the number of preliminary bio-indexes for obtaining the bio-index 3370 may be the same for each bio-index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the preliminary oxygen saturation and preliminary blood pressure for acquiring the oxygen saturation and blood pressure may be at least two, but is not limited thereto.

7.3 Method of obtaining a plurality of biometric indexes according to an embodiment

33 is a view for explaining a method for obtaining a plurality of biometrics according to an embodiment.

Referring to FIG. 33 , according to an exemplary embodiment, an image frame 3410 may be obtained to obtain a plurality of biometric indices.

In this case, the image frame 3410 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, the image frame 3410 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 33 , according to an exemplary embodiment, at least one region of interest 3420 may be set to obtain a plurality of biometric indices. More specifically, a first region of interest and a second region of interest may be set.

In this case, the first region of interest may be a region of interest for acquiring oxygen saturation, heart rate, and blood pressure, and the first and second regions of interest may be regions of interest for acquiring blood pressure.

Also, the first and second regions of interest may be identical to or different from each other.

Also, the first and second regions of interest may at least partially overlap each other.

Also, the sizes and regions of the first and second regions of interest may be set based on a biometric index to be acquired. For example, in order to acquire heart rate and oxygen saturation, the size of the first region of interest may be set to include the cheek region of a subject to detect changes in blood due to heartbeat well, and to acquire blood pressure The regions of the first and second regions of interest may be set along the direction of blood flow to reflect the flow of blood flow, but the present invention is not limited thereto, and the first and second regions of interest may have various sizes and regions. can

Also, referring to FIG. 33 , according to an exemplary embodiment, at least one pixel value 3430 may be obtained to obtain a plurality of biometric indices.

Also, the pixel value 3430 may be obtained for at least one image frame among a plurality of obtained image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained with respect to the first region of interest, and the second region of interest is converted into an RGB color space for the second region of interest. At least some of the red channel pixel value, the green channel pixel value, and the blue channel pixel value may be obtained, but the present invention is not limited thereto.

Also, referring to FIG. 33 , according to an exemplary embodiment, at least one color channel value 3440 may be obtained to obtain a plurality of biometric indices.

In this case, the color channel value 3440 may mean an average value of color channel pixel values or may mean a processed value. However, since the detailed description thereof has been described above, the overlapping description will be omitted.

Also, the color channel value 3440 may be obtained in consideration of the biometric to be obtained.

More specifically, at least two color channel values may be acquired for the first ROI for acquiring oxygen saturation, heart rate, and blood pressure.

For example, among color channel values such as a red channel value, a green channel value, a blue channel value, a Hue channel value, a saturation channel value, and a Value channel value, according to the RGB color space for the first region of interest At least two or more color channel values may be obtained, but the present invention is not limited thereto.

Also, for example, for the first region of interest, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and an H-V value that is a difference value between the Hue channel value and the Value channel value At least two or more color channel values may be obtained from among the color channel values, but the present invention is not limited thereto.

In addition, in order to obtain oxygen saturation, the at least two color channel values may be selected in consideration of absorbance of hemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin and a red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. It may be selected as a blue channel value, but is not limited thereto.

In addition, in order to obtain a heart rate, the at least two color channel values may be selected to reduce noise caused by movement and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, the G-R value, and the A G-B value that is a difference value between a green channel value, which is absorbed by oxyhemoglobin, and a blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, at least two color channel values may be obtained for the first and second ROIs for obtaining blood pressure.

For example, with respect to the first and second regions of interest for obtaining blood pressure, a red channel value, a green channel value, a blue channel value, a Hue channel value according to the HSV color space, a saturation channel value, and a value according to the RGB color space At least two color channel values among color channel values such as channel values may be obtained, but the present invention is not limited thereto.

In addition, for example, for the first and second regions of interest for obtaining blood pressure, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and a Hue channel value and At least two color channel values among color channel values such as an H-V value that is a difference value between Value channel values may be obtained, but the present invention is not limited thereto.

Also, the at least two color channel values may be selected to reduce noise caused by motion and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin. A G-B value, which is a difference value between a green channel value, which is absorbed by oxyhemoglobin, and a blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, referring to FIG. 33 , according to an embodiment, at least one time sequence data 3450 may be acquired to acquire a plurality of biometric indices.

Also, the time series data 3450 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least two time sequence data may be acquired for the first ROI. For example, first time series data and second time series data may be obtained to obtain oxygen saturation for the first region of interest, and third time series data may be obtained to obtain heart rate and blood pressure.

In this case, the first time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first ROI is the red channel value, the first time series data may be obtained based on the red channel value, but is not limited thereto.

In addition, the first time sequence data may be acquired for at least some image frame groups among a plurality of acquired image frames.

For example, when the first time series data is obtained based on a Red channel value obtained for a first image frame, the first time series data is obtained for a first image frame group including the first image frame. However, it is not limited thereto.

Also, the second time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first region of interest is a blue channel value, the second time series data may be obtained based on the blue channel value, but is not limited thereto.

Also, the second time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the second time series data is obtained based on a Blue channel value obtained for a second image frame, the second time series data may be obtained for a second image frame group including the second image frame. However, the present invention is not limited thereto.

Also, the third time series data may be obtained based on a color channel value obtained for the first region of interest. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the third time series data may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, the third time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the third time series data is obtained based on the G-R value and the G-B value obtained for the third image frame, the third time series data is a third image frame group including the third image frame. may be obtained for, but is not limited to.

Also, the third time series data may be obtained based on a characteristic value obtained with respect to the first ROI. For example, the third time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the first region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

In addition, in order to obtain blood pressure, at least one time series data may be acquired with respect to the second ROI, and third time series data obtained with respect to the first ROI and fourth time series data obtained with respect to the second ROI may be obtained. A blood pressure may be obtained based on the time series data.

In this case, the fourth time series data may be obtained based on the color channel value obtained for the second ROI. For example, when the color channel values obtained for the second region of interest are a G-R value and a G-B value, the fourth time series data may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, the fourth time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the fourth time series data is obtained based on the G-R value and the G-B value obtained for the fourth image frame, the fourth time series data is the fourth time series data for a fourth image frame group including the fourth image frame may be obtained, but is not limited thereto.

Also, the fourth time series data may be obtained based on a characteristic value obtained with respect to the second ROI. For example, the fourth time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the second region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

In addition, the first, second, third, and fourth image frame groups may be the same, but not limited thereto, may be different from each other, and may overlap each other at least partially.

Also, referring to FIG. 33 , according to an embodiment, at least one feature 3460 may be obtained to obtain a plurality of biometric indices.

In addition, at least one or more of the features 3460 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least one feature may be acquired with respect to the first region of interest. For example, a first characteristic of the first region of interest may be obtained to obtain oxygen saturation, and a second characteristic of the first region of interest may be obtained to obtain a heart rate.

In this case, the first characteristic may be acquired based on a color channel value or time series data acquired for the first ROI. For example, when the color channel values obtained for the first region of interest are a red channel value and a blue channel value, the first characteristic may be obtained based on the red channel value and the blue channel value, but limited thereto. doesn't happen

Also, for example, the first characteristic may be obtained based on first time series data and second time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the first characteristic may be a characteristic for obtaining oxygen saturation. For example, the first characteristic may be an AC value and a DC value obtained based on the first time series data, an AC value and a DC value obtained based on the second time series data, but is not limited thereto.

In addition, for example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data based on It may include, but is not limited to, at least one of a difference between the obtained local maximum average and the minimum minimum value average and an average value obtained based on the second time series data.

Also, the first feature may include a plurality of features. For example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data obtained based on At least two or more of a difference between the mean of the local maximum and the mean of the minimum value and an average value obtained based on the second time series data may be included, but the present invention is not limited thereto.

Also, the second characteristic may be obtained based on a color channel value or time series data obtained with respect to the first region of interest. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the second characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the second characteristic may be obtained based on third time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the second characteristic may be a characteristic for acquiring a heart rate. For example, the second characteristic may be a frequency value obtained based on the third time series data, a wavelength value, and a value corresponding to the number of repeated cycles during the measurement time, but is not limited thereto.

Also, at least one or more characteristics may be acquired with respect to the first and second ROIs for acquiring blood pressure. For example, a third characteristic may be obtained for the first and second regions of interest to obtain blood pressure.

In this case, the third characteristic may be acquired based on color channel values or time series data acquired for the first and second ROIs. For example, when the color channel values obtained for the first and second regions of interest are a G-R value and a G-B value, the third characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto. does not

Also, for example, the third characteristic may be obtained based on third time series data obtained with respect to the first area of interest and fourth time series data obtained with respect to the second area of interest, but is not limited thereto.

Also, the third characteristic may be a characteristic for acquiring blood pressure. For example, the third characteristic includes a slope value, a maximum value, a minimum value, an average of a maximum value, an average of a minimum value, a difference value between the average of the maximum value and the average of the minimum value obtained based on the third and fourth time series data. may be, but is not limited thereto.

Also, for example, the third characteristic may be a time difference between the third and fourth time series data, a time difference between maxima, a time difference between minima, and a time difference between inflection points, but is not limited thereto. does not

Also, the third feature may include a plurality of features. For example, the third characteristic may include at least two or more of the above-described characteristics, but is not limited thereto.

Also, referring to FIG. 33 , according to an embodiment, at least one biometric index 3470 may be obtained to obtain a plurality of biometric indices.

More specifically, each biometric index may be obtained based on a corresponding characteristic. For example, the oxygen saturation may be obtained based on the first characteristic, the heart rate may be obtained based on the second characteristic, and the blood pressure may be obtained based on the third characteristic, but is not limited thereto. does not

In this case, the above-described equations may be used to obtain oxygen saturation based on the first characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the heart rate based on the second characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the blood pressure based on the third characteristic, and overlapping descriptions will be omitted.

In addition, when there are a plurality of biometric indexes 3470 , each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be acquired 6 seconds after measurement, and blood pressure may be acquired 8 seconds after measurement, but is not limited thereto.

In addition, when there are a plurality of biometric indexes 3470 , each biometric index may be obtained based on each image frame group. For example, the oxygen saturation may be obtained based on the fifth image frame group, the heart rate may be obtained based on the sixth image frame group, and the blood pressure may be obtained based on the seventh image frame group.

Also, each image frame group for obtaining the biometric index 3470 may be the same, but not limited thereto, may be different from each other, and may overlap each other at least partially. For example, the fifth, sixth, and seventh image frame groups may be the same as each other, different from each other, and at least partially overlap each other.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3470 . For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

However, since the above description may be applied to a method of acquiring the heart rate or bio-index using the preliminary heart rate or the preliminary bio-index, the overlapping description will be omitted.

Also, the number of preliminary bio-indexes for obtaining the bio-index 3470 may be the same for each bio-index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the preliminary oxygen saturation and preliminary blood pressure for acquiring the oxygen saturation and blood pressure may be at least two, but is not limited thereto.

7.4 Method for obtaining a plurality of biometric indexes according to an embodiment

34 is a diagram for explaining a method for obtaining a plurality of biometric indices according to an exemplary embodiment.

Referring to FIG. 34 , according to an exemplary embodiment, an image frame 3510 may be obtained to obtain a plurality of biometric indices.

In this case, the image frame 3510 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, the image frame 3510 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 34 , according to an embodiment, at least one region of interest 3520 may be set to obtain a plurality of biometric indices. More specifically, a first ROI may be set.

In this case, the first ROI may be an ROI for acquiring oxygen saturation, heart rate, and blood pressure.

Also, the size and area of the first ROI may be set based on a biometric index to be acquired. For example, in order to obtain heart rate and oxygen saturation, the size of the first region of interest may be set to include the cheek region of a subject to detect changes in blood due to heartbeat, but is not limited thereto. The first region of interest may be set to various sizes and various regions.

Also, referring to FIG. 34 , according to an exemplary embodiment, at least one pixel value 3530 may be obtained to obtain a plurality of biometric indices.

Also, the pixel value 3530 may be obtained for at least one image frame among a plurality of obtained image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to the RGB color space may be obtained for the first region of interest, but the present invention is not limited thereto.

Also, referring to FIG. 34 , according to an embodiment, at least one color channel value 3540 may be obtained to obtain a plurality of biometric indices.

In this case, the color channel value 3540 may mean an average value of color channel pixel values or may mean a processed value. However, since the detailed description thereof has been described above, the overlapping description will be omitted.

Also, the color channel value 3540 may be obtained in consideration of the biometric to be obtained.

More specifically, at least two color channel values may be acquired for the first ROI for acquiring oxygen saturation, heart rate, and blood pressure.

For example, among color channel values such as a red channel value, a green channel value, a blue channel value, a Hue channel value, a saturation channel value, and a Value channel value, according to the RGB color space for the first region of interest At least two or more color channel values may be obtained, but the present invention is not limited thereto.

Also, for example, for the first region of interest, a G-R value that is a difference value between a red channel value and a green channel value, a G-B value that is a difference value between a green channel value and a blue channel value, and an H-V value that is a difference value between the Hue channel value and the Value channel value At least two or more color channel values may be obtained from among the color channel values, but the present invention is not limited thereto.

In addition, in order to obtain oxygen saturation, the at least two color channel values may be selected in consideration of absorbance of hemoglobin and oxyhemoglobin.

For example, a blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin and a red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. It may be selected as a blue channel value, but is not limited thereto.

In addition, in order to obtain the heart rate and blood pressure, the at least two color channel values may be selected to reduce noise caused by movement and noise caused by external light.

For example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the Red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, the G-R value, and the A G-B value that is a difference value between a green channel value, which is absorbed by oxyhemoglobin, and a blue channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, the G-R value, which is the difference between the Green channel value, which reflects a relatively large amount of change due to the heartbeat, and the Red channel value and the Blue channel value, which relatively reflects the change due to the heartbeat, relatively little, and A G-B value may be selected, but is not limited thereto.

Also, referring to FIG. 34 , according to an exemplary embodiment, at least one or more time series data 3550 may be obtained to obtain a plurality of biometric indices.

In addition, the time series data 3550 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least two time sequence data may be acquired for the first ROI. For example, first time series data and second time series data may be obtained to obtain oxygen saturation for the first region of interest, and third time series data may be obtained to obtain heart rate and blood pressure.

In this case, the first time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first ROI is the red channel value, the first time series data may be obtained based on the red channel value, but is not limited thereto.

In addition, the first time sequence data may be acquired for at least some image frame groups among a plurality of acquired image frames.

For example, when the first time series data is obtained based on a Red channel value obtained for a first image frame, the first time series data is obtained for a first image frame group including the first image frame. However, the present invention is not limited thereto.

Also, the second time series data may be obtained based on a color channel value obtained for the first ROI. For example, when the color channel value obtained for the first region of interest is a blue channel value, the second time series data may be obtained based on the blue channel value, but is not limited thereto.

Also, the second time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the second time series data is obtained based on a Blue channel value obtained for a second image frame, the second time series data may be obtained for a second image frame group including the second image frame. However, the present invention is not limited thereto.

Also, the third time series data may be obtained based on a color channel value obtained for the first region of interest. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the third time series data may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, the third time series data may be obtained for at least some image frame groups among the plurality of obtained image frames.

For example, when the third time series data is obtained based on the G-R value and the G-B value obtained for the third image frame, the third time series data is a third image frame group including the third image frame. may be obtained for, but is not limited to.

Also, the third time series data may be obtained based on a characteristic value obtained with respect to the first ROI. For example, the third time series data may be obtained based on a characteristic value obtained based on the G-R value obtained for the first region of interest and a characteristic value obtained based on the G-B value, but is not limited thereto. .

Also, the first, second, and third image frame groups may be the same, but not limited thereto, may be different from each other, and may overlap each other at least partially.

Also, referring to FIG. 34 , according to an embodiment, at least one feature 3560 may be acquired to acquire a plurality of biometric indices.

In addition, at least one or more of the features 3560 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least one feature may be acquired with respect to the first region of interest. For example, a first characteristic of the first region of interest may be obtained to obtain oxygen saturation, a second characteristic of the first region of interest may be obtained to obtain a heart rate, and blood pressure may be obtained In order to do this, a third characteristic of the first region of interest may be acquired, but the present invention is not limited thereto.

In this case, the first characteristic may be acquired based on a color channel value or time series data acquired for the first ROI. For example, when the color channel values obtained for the first region of interest are a red channel value and a blue channel value, the first characteristic may be obtained based on the red channel value and the blue channel value, but limited thereto. doesn't happen

Also, for example, the first characteristic may be obtained based on first time series data and second time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the first characteristic may be a characteristic for obtaining oxygen saturation. For example, the first characteristic may be an AC value and a DC value obtained based on the first time series data, an AC value and a DC value obtained based on the second time series data, but is not limited thereto.

In addition, for example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data based on It may include, but is not limited to, at least one of a difference between the obtained local maximum average and the minimum minimum value average and an average value obtained based on the second time series data.

Also, the first feature may include a plurality of features. For example, the first characteristic may include a difference between a maximum average and a minimum average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data obtained based on It may include at least two or more of a difference between the mean of the local maximum and the mean of the minimum value and an average value obtained based on the second time series data, but is not limited thereto.

Also, the second characteristic may be obtained based on a color channel value or time series data obtained with respect to the first region of interest. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the second characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the second characteristic may be obtained based on third time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the second characteristic may be a characteristic for acquiring a heart rate. For example, the second characteristic may be a frequency value obtained based on the third time series data, a wavelength value, and a value corresponding to the number of repeated cycles during the measurement time, but is not limited thereto.

Also, the third characteristic may be obtained based on a color channel value or time series data obtained with respect to the first ROI. For example, when the color channel values obtained for the first region of interest are a G-R value and a G-B value, the third characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the third characteristic may be obtained based on third time series data obtained with respect to the first ROI, but is not limited thereto.

Also, the third characteristic may be a characteristic for acquiring blood pressure. For example, the third characteristic may be a gradient value, a maximum value, a minimum value, the average of the maximum values, the average of the minimum values, the average of the minimum values, the difference between the average of the maximum values and the average of the minimum values obtained based on the third time series data. , but is not limited thereto.

Also, the third feature may include a plurality of features. For example, the third characteristic includes at least two or more of a difference value between a slope value, a maximum value, a minimum value, an average of a maximum value, an average of a minimum value, an average of a maximum value, and an average of a minimum value obtained based on the third time series data. may be included, but is not limited thereto.

Also, referring to FIG. 34 , according to an embodiment, at least one biometric index 3570 may be obtained to obtain a plurality of biometric indices.

More specifically, each biometric index may be obtained based on a corresponding characteristic. For example, the oxygen saturation may be obtained based on the first characteristic, the heart rate may be obtained based on the second characteristic, and the blood pressure may be obtained based on the third characteristic, but is not limited thereto. does not

In this case, the above-described equations may be used to obtain oxygen saturation based on the first characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the heart rate based on the second characteristic, and overlapping descriptions will be omitted.

In addition, the above-described equations may be used to obtain the blood pressure based on the third characteristic, and overlapping descriptions will be omitted.

In addition, when there are a plurality of biometric indices 3570, each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be acquired 6 seconds after measurement, and blood pressure may be acquired 8 seconds after measurement, but is not limited thereto.

In addition, when there are a plurality of biometric indices 3570 , each biometric index may be obtained based on each image frame group. For example, the oxygen saturation may be obtained based on the fourth image frame group, the heart rate may be obtained based on the fifth image frame group, and the blood pressure may be obtained based on the sixth image frame group.

In addition, each image frame group for obtaining the biometric index 3570 may be the same, but is not limited thereto, may be different from each other, and may overlap each other at least partially. For example, the fourth, fifth, and sixth image frame groups may be the same as each other, different from each other, and at least partially overlap each other.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3570 . For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

However, since the above description may be applied to a method of acquiring the heart rate or bio-index using the preliminary heart rate or the preliminary bio-index, the overlapping description will be omitted.

In addition, the number of preliminary bio-indexes for obtaining the bio-index 3570 may be the same for each bio-index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the preliminary oxygen saturation and preliminary blood pressure for acquiring the oxygen saturation and blood pressure may be at least two, but is not limited thereto.

7.5 Method for obtaining a plurality of biometric indices according to an embodiment

35 is a diagram for explaining a method for obtaining a plurality of biometric indices according to an embodiment.

Referring to FIG. 35 , according to an embodiment, a plurality of image frames 3600 may be acquired to acquire a plurality of biometric indices.

In this case, the image frame 3600 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, referring to FIG. 35 , at least one or more biometric indexes may be obtained based on at least one or more characteristics.

For example, as shown in FIG. 35 , a first biometric index 3660 may be obtained based on the first feature 3610 , and a second biometric index 3670 may be obtained based on the second feature 3620 . may be obtained, a third biometric index 3680 may be obtained based on the third characteristic 3630 , and a fourth biometric index 3690 may be obtained based on the fourth characteristic 3640 . Without limitation, one biometric index may be obtained based on a plurality of characteristics, or a plurality of biometric indexes may be obtained based on one characteristic.

In this case, the first to fourth characteristics may include an AC value, a DC value, a difference between the average maximum value and the minimum value average obtained based on at least one time series data, an average value, a frequency component value, a wavelength component value, and a measurement time value for the number of times the period is repeated during It may include at least one of characteristics such as difference and skin temperature, but is not limited thereto.

In addition, the first to fourth bioindices may include at least one of bioindices such as heart rate, oxygen saturation, blood pressure, body temperature, and blood flow, but is not limited thereto.

However, since detailed information regarding the characteristics and biometric index has been described above, overlapping descriptions will be omitted.

In addition, the first to fourth characteristics and/or the biometric index may be obtained based on a group of at least some of the plurality of image frames.

For example, the first characteristic and/or the first biometric index may be obtained based on a first image frame group, and the second characteristic and/or the second biometric index may be obtained based on a second image frame group. The third characteristic and/or the third biometric index may be obtained based on a third image frame group, and the fourth characteristic and/or the fourth biometric index may be obtained based on a fourth image frame group. may be obtained based on, but is not limited thereto.

In this case, the first to fourth image frame groups may include at least two image frames.

For example, as shown in FIG. 35 , the first image frame group may include N image frames, the second image frame group may include M image frames, and the third image The frame group may include K image frames, and the fourth image frame group may include L image frames, but is not limited thereto.

Also, the number of image frames included in the first to fourth image frame groups may be the same.

For example, the first image frame group may include 180 image frames, the second image frame group may include 180 image frames, and the third image frame group may include 180 image frames. may include, and the fourth image frame group may include 180 image frames, but is not limited thereto.

In this case, the first to fourth biometric indexes may be obtained or output at the same time point, but are not limited thereto, and may be obtained or output at different time points.

Also, in this case, the first to fourth biometric indexes may be obtained from the same state of the subject, and each biometric index may be correlated with each other.

Also, the number of image frames included in the first to fourth image frame groups may be different from each other.

For example, the first image frame group may include 180 image frames, the second image frame group may include 90 image frames, and the third image frame group may include 240 image frames. may include, and the fourth image frame group may include 360 image frames, but is not limited thereto.

In this case, the first to fourth biometric indexes may be obtained or output at different time points, but are not limited thereto, and may be obtained or output at the same time point.

In addition, the number of image frames included in the first to fourth image frame groups may be different depending on the desired biometric index.

For example, when the first biometric index is heart rate, the second biometric index is oxygen saturation, the third biometric index is blood pressure, and the fourth biometric index is body temperature, the first image frame group may include 180 image frames. The second image frame group may include 120 image frames, the third image frame group may include 60 image frames, and the fourth image frame may include 60 image frames. However, the present invention is not limited thereto, and the number of image frames included in the image frame group may be set in consideration of the desired biometric index.

36 is a diagram for describing a method of obtaining a plurality of related biometric indices according to an embodiment.

Referring to FIG. 36 , according to an embodiment, a plurality of image frames 3700 may be acquired to acquire a plurality of related biometric indices.

In this case, the image frame 3700 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, referring to FIG. 36 , at least one biometric index may be obtained based on at least one image frame group.

More specifically, as shown in FIG. 36 , the first biometric index 3760 may be obtained based on the first image frame group 3710 , and the second biometric index 3770 is the second image frame group ( 3720 , the third biometric index 3780 may be obtained based on the third image frame group 3730 , and the fourth biometric index 3790 is the fourth image frame group 3740 . can be obtained based on

At this time, since the above-described contents may be applied to each image frame group and each biometric index, overlapping descriptions will be omitted.

Referring to FIG. 36 , in order to obtain the first biometric index 3710 , the second biometric index 3720 , the third biometric index 3730 , and the fourth biometric index 3740 to be correlated with each other, the The first image frame group 3710 , the second image frame group 3720 , the third image frame group 3730 , and the fourth image frame group 3740 may overlap each other at least partially.

For example, as shown in FIG. 36 , the first image frame group 3710 includes first image frames to thirteenth image frames, and the second image frame group 3720 includes sixth image frames to a nineteenth image frame, the third image frame group 3730 includes a fourth image frame to a seventeenth image frame, and the fourth image frame group 3740 includes a ninth image frame to a twenty-third image frame may include, but is not limited thereto.

Accordingly, the first to fourth image frame groups 3710 , 3720 , 3730 , and 3740 may include ninth to thirteenth image frames in common.

Also, as the first to fourth image frame groups overlap each other at least partially as described above, the first to fourth biometric indexes may be obtained to be correlated with each other.

For example, since the first to fourth image frame groups commonly include the ninth to thirteenth image frames, the first to fourth biometric indexes may be obtained including the same state of the subject, and thus Accordingly, the first to fourth bio-indexes may be obtained to be correlated with each other.

As a more specific example, when the state of the subject at the point in time when the ninth image frame is acquired is referred to as the first state, the first to fourth biometric indexes commonly reflect the first state of the subject. may be obtained, and accordingly, the first to fourth biometrics may be obtained so as to be correlated with each other.

In addition, when the first to fourth biometric indices 3760, 3770, 3780, and 3790 are obtained to be related, it is possible to more accurately determine the subject's biometric index.

In addition, at least one or more pieces of biometric information may be acquired based on the first to fourth biometric indices 3760 , 3770 , 3780 and 3790 .

In addition, when the biometric information is obtained based on related biometric indices, the accuracy of the biometric information may be improved. For example, when the heart rate and blood pressure that can be included in the first to fourth biometric indexes are used to obtain the hypertension information that can be included in the biometric information, when the heart rate and blood pressure are correlated with each other, the hypertension information is could be more accurate.

More specifically, when blood pressure is measured while the subject is exercising or excited, and the subject measures heart rate in a stable state and hypertension information is obtained based on this, the possibility of detecting hypertension even if the subject is not hypertensive There is this. On the other hand, when the subject is exercising or excited, blood pressure and heart rate are measured and hypertension information is obtained based on the measurement, the hypertension information can be more accurately obtained.

37 is a diagram for describing a method for obtaining a plurality of biometric indices according to an embodiment.

Referring to FIG. 37 , at least one or more biometric indexes may be obtained based on at least one or more preliminary biometric indexes.

For example, as shown in FIG. 37 , the first biometric index may be obtained based on N preliminary first biometric indices, and the second biometric index may be obtained based on M preliminary second biometric indices. In addition, the third biometric index may be obtained based on K preliminary third biometric indices, and the fourth biometric index may be obtained based on L preliminary fourth biometric indices.

However, since the above description may be applied to the method of obtaining the biometric index using the preliminary biometric index, the overlapping description will be omitted.

The number of preliminary first to fourth bio-indexes for obtaining each of the first to fourth bio-indexes may be the same.

For example, the preliminary first biometric index for obtaining the first biometric index may be four, the preliminary second biometric index for obtaining the second biometric index may be four, and the second biometric index may be four. The third preliminary bio-index for obtaining the 3 bio-index may be four, and the fourth preliminary bio-index for obtaining the fourth bio-index may be four, but is not limited thereto.

Also, the number of preliminary first to fourth bio-indexes for obtaining each of the first to fourth bio-indexes may be different from each other.

For example, the preliminary first biometric index for obtaining the first biometric index may be four, the preliminary second biometric index for obtaining the second biometric index may be two, and the second biometric index may be two. The third preliminary bio-index for obtaining the 3 bio-index may be two, and the fourth preliminary bio-index for obtaining the fourth bio-index may be one, but is not limited thereto.

In addition, the number of preliminary first to fourth bio-indexes for obtaining each of the first to fourth bio-indexes may be different depending on the desired bio-index.

For example, when the first bio-index is heart rate, the second bio-index is oxygen saturation, the third bio-index is blood pressure, and the fourth bio-index is body temperature, the preliminary first bio-index for obtaining the heart rate is four may be, the preliminary second biometric index for obtaining the oxygen saturation may be two, the preliminary third biometric index for obtaining the blood pressure may be two, and the second biometric index for obtaining the body temperature may be two The fourth preliminary biometric index may be one, but is not limited thereto, and the number of basic preliminary biometric indexes may be set in consideration of the desired biometric index.

8. Drowsiness detection device and method

8.1 Drowsiness detection device

The drowsiness detecting device provided herein may refer to a device for detecting a state related to drowsiness of a subject. Specifically, the drowsiness detecting device may detect whether the subject is drowsy and the intensity of drowsiness.

People can feel drowsy during their daily life. And in some cases, there may be situations in which drowsiness poses a safety hazard. For example, if a driver feels drowsy while driving, he or she may not be able to concentrate on driving, which may lead to a traffic accident. In this case, the drowsiness detection device detects the driver's drowsiness state and notifies the driver or the manager of the drowsiness state, thereby preventing a situation threatening the driver's safety, such as a traffic accident. As such, it goes without saying that the drowsiness detecting device may be used not only in situations where drowsiness poses a safety risk, but also in places and fields where drowsiness acts as important information, such as a reading room and infant monitoring.

The drowsiness detecting device may detect a state related to drowsiness by using a contact type device accompanying physical contact with a subject, such as an attachable electrode sensor or a wearable device. In addition, the drowsiness detecting apparatus may detect a state related to the drowsiness of the subject in a non-contact manner using a camera. When the non-contact method is used, as drowsiness of the subject can be sensed without being constrained by the touch-type device, the user's convenience is increased and the drowsiness detecting device can be used in various places and fields. .

The non-contact drowsiness detecting device may detect a state related to drowsiness by detecting an abnormal state of the subject, such as measuring the number of blinks of the subject's eyes or tracking the position of the pupil. However, when the subject consciously controls the eye blinking or pupil position, it may be difficult to accurately detect a state related to drowsiness. In order to supplement the accuracy, the state related to drowsiness can be more accurately detected if information appearing in the subject's body, which appears when the subject feels drowsy, is used. In addition, if the subject does not feel drowsy but uses biometric information that appears in a situation in which he or she will soon feel drowsy, the state related to drowsiness may be detected in advance in the initial stage.

Due to these advantages, the drowsiness detecting device 6000 provided in the specification detects whether the subject is in a state of drowsiness and to what degree of drowsiness by using the biometric index of the subject obtained in a non-contact manner. can

Referring to FIG. 38 according to an exemplary embodiment, the drowsiness detecting device 6000 includes an acquired heart rate information acquisition unit 6100 that acquires a heart rate from the subject, and detects drowsiness of the subject based on the acquired heart rate of the subject. It may include a drowsiness detection unit 6200 to detect.

Of course, the drowsiness detecting device 6000 of the present invention may further include other components in addition to the above-mentioned heart rate information obtaining unit 6100 and drowsiness detecting unit 6200 . For example, the drowsiness detecting apparatus 6000 may further include a notification unit 6300 in addition to the heartbeat information obtaining unit 6100 and the drowsiness detecting unit 6200 . Here, the notification unit 6300 may give a notification for waking up the subject from drowsiness or a notification to an object other than the subject based on the result detected by the drowsiness detecting unit 6200 . Here, the entity other than the subject may refer to a manager who manages the site to be measured or a third party in the vicinity of the subject.

In an embodiment, the drowsiness detecting apparatus 6200 may detect the drowsiness of the subject at various time points. That is, some or all of the components included in the drowsiness detecting apparatus 6000 may operate at various time points, and each or all steps of various methods for detecting drowsiness to be described below may also be implemented at various time points.

For example, the drowsiness detecting apparatus 6000 may detect the drowsiness of the subject according to a predetermined period (eg, every minute or every hour). As another example, the drowsiness detecting apparatus 6000 may detect the drowsiness of the subject without following a predetermined time period. As an example, the drowsiness detecting apparatus 6000 may detect the drowsiness of the subject whenever information on the heartbeat of the subject is obtained, that is, in real time. In addition, the drowsiness detecting device may continuously detect the drowsiness of the subject, and may detect the drowsiness of the subject when receiving a signal for triggering the drowsiness detection or receiving a request to detect drowsiness.

In another embodiment, the drowsiness detecting apparatus 6000 may start detecting drowsiness when first acquiring a heart rate. For example, the heart rate information acquisition unit 6100 may first acquire a heart rate several seconds or minutes after starting measurement for obtaining a heart rate for the subject. In this case, the drowsiness detecting apparatus 6000 may start detecting drowsiness with respect to the subject from the time the first heart rate is acquired.

Hereinafter, components included in the drowsiness detecting apparatus 6000 will be described in detail with reference to FIG. 38 according to an exemplary embodiment.

8.1.1 Heart rate information acquisition unit

In an embodiment, the heart rate information obtaining unit 6100 may obtain information about the heart rate of the subject, for example, at least one of a heart rate of the subject or a ratio of sympathetic nerve activity to parasympathetic nerve activity.

Here, the heart rate may mean the number of times the heart beats during the reference time. The heart rate generally refers to the heart rate for a reference time of 1 minute, but the reference time may be arbitrarily set. For example, it can be set to 30 seconds or another length. In addition, the ratio of the sympathetic nerve activity to the parasympathetic nerve activity is due to the heartbeat signal, where the heartbeat signal may mean a signal that can fluctuate depending on the heartbeat, and a signal that can be estimated to fluctuate with the heartbeat can mean In general, sympathetic nerve activity and parasympathetic nerve activity are highly correlated with the subject's drowsiness state. Accordingly, the drowsiness detecting device may detect the drowsiness of the subject by using the ratio of the sympathetic nerve activity to the parasympathetic nerve activity resulting from the heartbeat signal.

In addition, the activity of the sympathetic nerve may be characterized in the low frequency band of the heartbeat signal, and the activity of the parasympathetic nerve may be characterized in the high frequency band of the heartbeat signal. Accordingly, in the present specification, the ratio of sympathetic nerve activity to parasympathetic nerve activity of the heartbeat signal may be expressed as LF/HF. LF may mean a characteristic value in a low frequency band (LF) of the heartbeat signal, and HF may mean a characteristic value in a high frequency band (HF) of the heartbeat signal.

In an embodiment, the low frequency band and the high frequency band may be divided based on a reference frequency. For example, the low frequency band and the high frequency band are divided based on a reference frequency of 0.15 Hz, the low frequency band includes a signal having a frequency in the range of 0.04 to 0.15 Hz, and the high frequency band has a frequency in the range of 0.15 to 0.4 Hz It may contain signals. Of course, the reference frequency may be set to a frequency other than 0.15 Hz.

In an embodiment, the heartbeat information acquisition unit 6100 may convert the heartbeat signal into a signal in the frequency domain through various methods such as Fourier transform, and through the conversion, the heartbeat signal is converted to a low frequency band (0.04 - 0.15 Hz) and a high frequency signal. By extracting the signal in the band (0.15 - 0.4 Hz), the LF/HF value can be obtained.

In an embodiment, the heart rate information acquisition unit 6100 may acquire information about the heart rate invasively or non-invasively.

Also, according to another exemplary embodiment, the heart rate information obtaining unit 6100 may obtain information on the heart rate using a contact method or a non-contact method.

In more detail, the heart rate information obtaining unit 6100 may obtain heart rate information in a non-contact method using a device that does not physically contact the subject. For example, the heart rate information obtaining unit 6100 may obtain an image of the subject by using a camera, and may obtain information on the heart rate of the subject by analyzing the obtained image. A method and apparatus for non-contactingly acquiring heart rate information using a camera have been described above, and thus detailed descriptions thereof will be omitted.

Also, it goes without saying that the heart rate information acquisition unit 6100 may acquire information on the heart rate through a contact method using a device in physical contact with the subject. For example, the heart rate information acquisition unit 6100 may obtain heart rate information by using an attachable sensor for measuring heart rate or a wearable device including various attachable sensors and optical sensors. For convenience of explanation, a configuration of a drowsiness detecting device and a method for detecting drowsiness will be described below, focusing on an example of acquiring heart rate information in a non-contact manner.

In an embodiment, the heart rate information acquisition unit 6100 may obtain information on the heart rate by directly measuring it. In more detail, the heart rate information obtaining unit may include an apparatus for measuring a heart rate, and may obtain information about a heart rate measured by the apparatus for measuring a heart rate. For example, the heart rate information obtaining unit 6100 may include a camera, and may directly measure heart rate information by analyzing an image obtained from the camera.

Also, the heart rate information acquisition unit 6100 may obtain information about the heart rate by receiving information about the heart rate measured by an external device. For example, the heart rate information is measured by a device including an electrode sensor for measuring the heart rate, and the heart rate information acquisition unit 6100 may receive the measured heart rate information. Also, the heart rate information obtaining unit 6100 may receive information about the heart rate measured through the heart rate information obtaining unit 6100 and another device (eg, a server) that mediates the external device.

In an embodiment, when the drowsiness detecting apparatus 6000 is included in the biometric index obtaining apparatus 10 , the heart rate information obtaining unit 6100 reads the heart rate stored in the memory of the biometric index obtaining apparatus 10 to obtain a heart rate can do. In another embodiment, when the drowsiness detecting device 6000 is not included in the biometric index obtaining device 10 , the heart rate information obtaining unit 6100 uses the heart rate measured by the biometric index obtaining device 10 as the drowsiness detecting device ( 6000) may be received through a communication unit (not shown). Of course, the drowsiness detection device 6000 may acquire the heart rate from a device other than the biometric index acquisition device 10 .

8.1.2 Drowsiness Detector

In an embodiment, the drowsiness detecting unit 6200 may detect whether the subject is drowsy or the intensity of drowsiness felt by the subject based on the information on the heart rate. Specifically, the drowsiness detecting unit 6200 may detect a drowsy state and a normal state of the subject based on the information on the heart rate. Here, the drowsy state may be divided into a plurality of stages.

In the present specification, the drowsiness state may mean a state in which the subject is temporarily sleeping or a state in which the subject is likely to fall asleep within a predetermined time before falling asleep, and the normal state is not a drowsiness state, but the subject within a predetermined time. It can mean a state in which you are less likely to fall asleep.

In an embodiment, the drowsy state may be divided into a plurality of levels according to the intensity of drowsiness. The drowsy state may be classified from the first stage to the Nth stage depending on the case. In the present specification, for convenience of explanation, the level with the lowest intensity of drowsiness is indicated as level 1, and the level with the highest intensity of drowsiness is indicated as level N. For example, the drowsy state may be divided into first to third stages. In this case, the first stage may mean a stage with the lowest intensity of drowsiness, and the third stage may mean a stage with the highest intensity of drowsiness.

In an embodiment, the first stage drowsiness state may mean a state in which the subject physically enters the drowsiness state, but does not realize that the subject is drowsy. For example, in the first stage drowsiness state, the information about the heartbeat of the subject objectively indicates that the subject is drowsy, but the subject may not feel drowsy by himself. In the present specification, the drowsiness state may be expressed as an unconscious drowsiness state or an unconscious drowsiness state.

In an embodiment, the second stage drowsiness state and the third stage drowsiness state may refer to a state in which the subject physically enters the drowsiness state and the subject becomes aware of drowsiness. In the present specification, the corresponding states may be expressed as a conscious drowsiness state or a conscious drowsiness state.

In general, the subject enters the unconscious drowsiness state before entering the conscious drowsiness state. If the subject receives a notification of drowsiness after entering the conscious drowsiness state, since the notification is received when the subject is already asleep, a dangerous situation is highly likely to occur. Also, even if a notification of the drowsiness state is received before a dangerous situation occurs, it may take a long time for the subject to get out of the drowsiness state.

In order to prevent such a dangerous situation, the drowsiness detecting device 6000 may detect the unconscious drowsiness state before the subject enters the conscious drowsiness state, and may provide a notification based on the detection result to the outside. The drowsiness detecting device 6000 may make the subject become conscious of the drowsy state from an involuntary drowsiness state in which the subject does not feel drowsy through the notification.

Therefore, detection and notification of involuntary drowsiness can increase the possibility of getting the subject out of drowsiness before a dangerous situation occurs, and can more reliably protect the subject from dangerous situations due to drowsiness.

In an embodiment, the drowsiness detecting unit 6200 may measure the drowsiness state of the subject by using the heart rate information. In the present specification, the drowsiness state is a state in which the subject temporarily falls asleep or a state in which the subject is likely to fall asleep within a predetermined time immediately before falling asleep, and is a state in which the parasympathetic nervous system is relatively activated compared to the normal state, and the parasympathetic nervous system is activated, the heart rate decreases. That is, the change in heart rate may serve as a biomarker for estimating the drowsiness state of the subject. For example, when the subject enters the drowsy state, the heart rate of the subject may decrease below a predetermined level, and the reduced heart rate may continue for a predetermined time. In some cases, the drowsiness detecting unit 6200 may estimate the drowsiness state of the subject based on the duration of the reduced heart rate.

In an embodiment, the drowsiness detecting unit 6200 may measure the normal state of the subject by using heart rate information. For example, when the subject enters a normal state, the heart rate of the subject may increase to a certain level or more due to activation of the sympathetic nerve, and the increased heart rate may also be continued for a certain period of time. In this case, the drowsiness detecting unit 6200 may estimate the normal state of the subject based on the duration of the increased heart rate.

Also, in another embodiment, the drowsiness detecting unit 6200 may measure the drowsiness state and the normal state of the subject by using LF/HF. As the subject enters the drowsy state, the parasympathetic nervous system of the subject is activated, and the LF value of the heartbeat signal with respect to the HF value may decrease. Also, when the subject enters the normal state, the subject's sympathetic nervous system is activated, and the LF value with respect to the HF value may increase. Accordingly, the drowsiness detecting unit may estimate the drowsiness state and the normal state of the subject through LF/HF.

In another embodiment, the drowsiness detecting unit 6200 may estimate the drowsiness or normal state of the subject by considering the change in the heart rate and the LF/HF at the same time.

Such a method for detecting drowsiness will be described in detail below.

8.1.3 Notification

In an embodiment, the notification unit 6300 may provide information about the externally sensed drowsiness state. Specifically, the notification unit 6300 may provide various information related to whether the subject is in a drowsy state, a drowsy state, and in addition to this, a sensed drowsy state. For example, the notification unit 6300 may externally provide a time for maintaining a drowsy state, a time for entering each drowsiness state, a recovery time for the drowsiness state, and the like.

In an embodiment, the notification unit 6300 may provide information about a drowsy state to the subject. For example, the notification unit 6300 may give a notification of an intensity corresponding to a stage of drowsiness to the subject. The notification may induce the subject to recover from the drowsy state by applying a certain stimulus to the subject who is in a drowsy state.

In another embodiment, information about the drowsiness state may be provided to a subject other than the subject. For example, the notification unit 6300 may provide information about a drowsy state to an administrator or a server. Here, the manager may mean a manager of the subject or an entity managing the drowsiness state of the subject. Provision of such information can help the administrator to effectively manage the safety of the subject based on the information on the drowsiness state.

In an embodiment, the notification unit 6300 may provide information on the drowsiness state of all detected stages to the outside. For example, the notification unit 6300 may provide information on the drowsiness state to the subject for the first to third stage drowsiness states.

Also, in another embodiment, the notification unit 6300 may provide only information about at least a portion of the detected drowsiness states to the outside.

For example, the notification unit 6300 may provide the manager with information on the time and stage at which the third stage drowsiness state is detected only when the third stage drowsiness state is detected. If the notification of all stages of drowsiness interferes with the subject, the notification unit 6300 provides only information about the high-risk drowsiness stage (eg, information on the third stage drowsiness state), and the risk By not providing information on this low level of sleepiness (eg, information on the first level of sleepiness), it is possible to increase the work efficiency of the subject.

Also, according to an embodiment, when the subject is detected as being in a normal state, the notification unit 6300 may provide information on the detected normal state. Specifically, the notification unit 6300 may provide various information related to the sensed normal state, including whether the subject is in a normal state. For example, it is possible to provide a time for maintaining a steady state, a time for entering a steady state, and the like.

In an embodiment, the notification unit 6300 may provide information about the normal state to the subject. For example, the notification unit 6300 may output a message to notify the fact that the drowsy subject recovers to the normal state, when the subject recovers to the normal state.

In another embodiment, the notification unit 6300 may provide information about the normal state to the subject other than the subject. For example, the notification unit 6300 may provide information about a normal state to an administrator or a server. Providing such information can help the administrator to recognize that the subject is in a safe situation.

In an embodiment, the notification unit 6300 generates a notification signal indicating information related to the drowsiness state, and outputs the notification signal to an output unit (not shown) so that information related to the normal state and the drowsiness state is output to the outside of the drowsiness detecting device. can provide

Here, the output unit is a device capable of notifying information to the outside, and for example, the output unit is a display unit that visually displays information to the outside, an audio unit that can inform information aurally, and a tactile system that informs information It may include at least one of the tactile stimulation unit. The output unit may be included in the drowsiness detecting device 6000 or may be included in an external device of the drowsiness detecting device 6000 .

When the output unit is included in another device outside the drowsiness sensing device 6000, the drowsiness sensing device 6000 may further include a communication unit (not shown), and the notification unit 6300 sends a notification signal to the output unit through the communication unit. can provide The output unit may receive the notification signal provided by the notification unit 6300 and output information about drowsiness to the outside according to the notification signal.

For example, the notification unit 6300 may provide an auditory notification through the audio unit. Such an auditory notification is effective because it can provide an immediate stimulus to the notification recipient and can vary the intensity of the stimulus according to the loudness of the sound.

For another example, the notification unit 6300 may provide a notification by displaying a message on the display through the display unit. Such a visual notification can minimize a situation in which an entity other than the notification recipient is disturbed by the notification.

For another example, the notification unit 6300 may provide a notification through the tactile stimulation unit, for example, vibration, electrical stimulation, heat element, or cooling element. Such a tactile notification is effective because a situation in which an entity other than the notification recipient is not disturbed by the notification does not occur, and an immediate stimulation can be given, and the strength of the stimulation can be varied according to the strength of the stimulation.

Also, according to an embodiment, the notification unit 6300 may provide notifications of different types and/or strengths to the subject according to the detected drowsiness stage. Accordingly, the notification unit 6300 may provide a notification with a type suitable for recognizing that the subject is in a drowsy state and/or with an intensity required for the subject to get out of the drowsiness state.

For example, when a first stage drowsiness state, which is a low-intensity drowsiness state, is detected, the notification unit 6300 may recognize the drowsiness state by giving a notification using a visual message to the subject. In addition, when the third stage drowsiness, which is a high intensity drowsiness state, is detected, the notification unit 6300 may provide a notification to the subject using electrical stimulation so that the subject can escape from the drowsiness state.

As another example, when a first stage drowsiness state, which is a low intensity drowsiness state, is detected, the notification unit 6300 provides a low-intensity alarm to the subject, and when a third stage drowsiness is detected, the notification unit 6300 ) may provide a high-intensity alarm to the subject.

Of course, according to an embodiment, the notification unit 6300 may provide a notification of the same type and/or intensity to the subject regardless of the stage of the sensed drowsiness state.

Also, the notification unit 6300 may give a notification to at least one of the subject and the non-measured subject according to the detected drowsiness stage.

In an embodiment, the notification unit 6300 may give a notification to the subject when the first stage and the second stage drowsiness state is detected. In this case, the notification unit 6300 may give at least one of an auditory notification, a visual notification, and a tactile notification to the subject.

Also, when the third stage drowsiness state is detected, the notification unit 6300 may give a notification to not only the subject but also the subject. Specifically, the notification unit 6300 may give a notification to a non-measured entity through a device or server that mediates the notification unit 6300 and a non-measured entity. For example, when an entity other than the subject is an administrator who manages the subject, such a notification may be effective in a situation in which the subject turns off the power of the notification through the output unit or ignores the notification. Hereinafter, a method for detecting drowsiness performed by the drowsiness detecting device 6000 will be described in detail.

8.2 Method for detecting drowsiness based on heart rate

Hereinafter, a method for detecting drowsiness based on a heart rate according to an exemplary embodiment will be described with reference to FIGS. 39 to 43 .

In an embodiment, the method for detecting drowsiness based on the heart rate includes: acquiring a heart rate (S6110), comparing the acquired heart rate with a reference heart rate (S6120), and measuring the duration of the changed heart rate according to the comparison result It may include a step of measuring the duration ( S6130 ) and a step of detecting drowsiness step by step ( S6140 ) of detecting drowsiness step by step based on the measured duration of the heart rate change.

In the heart rate obtaining step S6110 , the drowsiness detecting apparatus 6000 may obtain the heart rate of the subject by using the heart rate information obtaining unit 6100 .

In an embodiment, the drowsiness detecting apparatus 6000 may acquire the heart rate of the subject according to a predetermined time period. Here, the predetermined time period may include various time periods such as 1 second, 1 minute, 5 minutes, and the like. Also, it goes without saying that the predetermined time period may be fixed or variable.

In another embodiment, the drowsiness detecting apparatus 6000 may acquire the heart rate of the subject without following a predetermined time period. For example, the drowsiness detecting apparatus 6000 may acquire the heart rate of the person to be measured whenever the heart rate of the person to be measured is measured by the biometric index obtaining apparatus 10 or another device, that is, in real time.

Of course, if the heart rate of the subject is measured according to a predetermined time period in the biometric index obtaining device 10 or another device, the drowsiness detecting device 6000 may measure the predetermined time period measured by the biometric index obtaining device 10 or another device. The heart rate of the subject may be acquired based on the measurement.

As another example, the drowsiness detecting device 6000 may request the biometric index acquisition device 10 or another device to provide a heart rate according to an input or request from the outside, and whenever the request is made, the heart rate of the subject can obtain

In an embodiment, the drowsiness detecting apparatus 6000 may acquire the average heart rate of the subject for a predetermined time period. The predetermined time period may be arbitrarily determined. Here, the drowsiness detecting apparatus 6000 may calculate an average heart rate for a predetermined time period using the heart rate of the subject received from the external device. Also, the drowsiness detecting device 6000 may receive the average heart rate for a predetermined time period of the subject from the external device.

Hereinafter, an effect of obtaining an average heart rate as a heart rate of a subject will be described with reference to FIG. 40 .

Referring to FIG. 40 according to an embodiment, the drowsiness detecting apparatus 6000 may acquire a noise-free heart rate as the average heart rate 6003 is obtained.

Here, the noise indicates an error caused by various causes such as movement of the subject or external light, and when noise is included in the heart rate, it may be difficult to accurately reflect the state of the subject.

For example, when the heart rate information acquisition unit 6100 acquires a heart rate at every second interval from 1 second to 10 seconds and noise is generated at 5 seconds, the drowsiness detecting apparatus 6000 may not accurately detect drowsiness. However, when the heart rate information obtaining unit 6100 obtains the average heart rate 6003, the deviation between noise and other values may be corrected in the process of calculating the average heart rate 6003, so the drowsiness detecting device 6000 is more It can accurately detect drowsiness.

The method of obtaining the heart rate of the subject by the heart rate information acquisition unit 6100 is described in Table of Contents 8.1.1, and detailed description thereof will be omitted.

Hereinafter, referring to FIG. 41 , comparing the heart rate of the subject with a reference heart rate ( S6120 ) and measuring the duration for which the acquired heart rate of the subject continues in a decreased/increased state ( S6130 ) to be explained.

Referring to FIG. 41 according to an exemplary embodiment, in the step of comparing the heart rate of the subject with the reference heart rate ( S6120 ), the drowsiness detecting device 6000 determines the heart rate ( 6001 ) and a reference heart rate 6002 may be compared.

Here, the reference heart rate 6002 may mean a heart rate set as a reference for distinguishing between a drowsy state and a normal state. As described above, according to the activation of the parasympathetic nervous system, the heart rate in the drowsy state is lower than that in the normal state. Accordingly, in order to distinguish the drowsy state from the normal state, a heart rate equal to or similar to the reduced heart rate may be set as the reference heart rate 6002 . In this case, when the subject's heart rate is less than or equal to the reference heart rate 6002, the subject may be identified as drowsy. it could be

Hereinafter, various methods for setting the reference heart rate 6002 will be described.

In an embodiment, the drowsiness detecting apparatus 6000 may set the reference heart rate 6002 by using an average of heart rates acquired for a predetermined time from the time the heart rate is first acquired. The predetermined time may be set to various values such as 1 minute and 5 minutes. In general, there is a high possibility that the subject is in a normal state rather than in a drowsy state for a certain period of time from the time the heart rate of the subject is first acquired. Accordingly, the average heart rate of the subject acquired for the predetermined time is obtained, and the average heart rate is multiplied by a predetermined value a (eg, a represents a real number less than or equal to 1. As a more specific example, a is 0.9). A value may be obtained as a reference heart rate 6002 . The predetermined value a is not limited to the above example.

In another embodiment, the drowsiness detecting device 6000 may set the reference heart rate 6002 using the resting heart rate and/or the active heart rate. Specifically, the resting heart rate may mean a heart rate in a state in which the subject does not move (or when the amount of movement is small). In the present specification, the heart rate when the subject is awake but not moving (or when the amount of movement is small) is defined as the first resting heart rate, and the heart rate when the subject is sleeping is defined as the second resting heart rate to do it

In addition, the active-phase heart rate may mean a heart rate when a person to be measured actively moves.

In an embodiment, the drowsiness detecting apparatus 6000 may acquire the measured heart rate as the first resting heart rate in a situation in which it is recognized that the subject is awake but does not move (or when the amount of movement is small). For example, when the drowsiness detecting device 6000 is in an environment in which the subject cannot move actively (eg, when the subject is sitting in the driver's seat), the drowsiness detecting device 6000 may measure the drowsiness for a certain period of time from when the heart rate is first acquired. Assuming that the self is awake but not moving, the heart rate measured for the predetermined time may be set as the first resting heart rate.

In addition, the drowsiness detecting device 6000 acquires a heart rate measured in a situation in which the subject is recognized as sleeping as the second resting heart rate, and obtains the heart rate measured in a situation in which the subject is recognized as actively moving. It can be obtained from active heart rate.

Also, the drowsiness detecting device 6000 may acquire a first resting heart rate, a second resting heart rate, and an active heart rate using a camera or a wearable device.

For example, the drowsiness detecting apparatus 6000 may obtain information on whether the subject moves and whether the pupil of the subject is tracked using a camera. For example, the drowsiness detecting apparatus 6000 may acquire a heart rate measured when the subject moves as an active heart rate. In addition, the drowsiness detecting device 6000 acquires the measured heart rate as the first resting heart rate when the subject's pupil is continuously tracked even if the subject does not move using the camera, and When the pupil is not continuously tracked, the measured heart rate may be acquired as the second resting heart rate.

As another example, the drowsiness detecting device 6000 may obtain information on whether the subject moves using the wearable device and whether the body part (eg, wrist) on which the wearable device is worn moves. As an example, the drowsiness detecting apparatus 6000 may acquire the heart rate measured when the movement of the subject is detected using the wearable device as the active heart rate. In addition, the drowsiness detecting device 6000 acquires the measured heart rate as the first resting heart rate when the movement of the subject is not detected but the movement of the body part on which the wearable device is worn is detected, and the movement of the subject is not detected. , when the movement of the body part is not detected for a predetermined time, the measured heart rate may be acquired as the second resting heart rate.

In addition, the drowsiness detecting device 6000 may preset the first resting heart rate, the second resting heart rate, and the active heart rate before starting the drowsiness detection, and the first resting heart rate from the outside before or after starting the drowsiness detection; A second resting heart rate and an active heart rate may be received.

According to an embodiment, the reference heart rate 6002 may be set based on the first resting heart rate. Specifically, the reference heart rate 6002 may be set to a constant ratio of the first resting heart rate. For example, the reference heart rate 6002 may be calculated by multiplying the first resting heart rate by a predetermined value a (eg, a represents a real number less than or equal to 1. As a more specific example, a is 0.9). Of course, the predetermined value a is not limited to the above example.

According to another embodiment, the reference heart rate 6002 may be set based on the second resting heart rate. Specifically, the reference heart rate 6002 is set equal to or equal to the second resting heart rate, or a predetermined value b for the second resting heart rate (eg, b represents a real number greater than 1. As a more specific example, b is 1.1) can be calculated. Of course, the predetermined value b is not limited to the above example.

Also, according to another embodiment, the reference heart rate 6002 may be set based on the active phase heart rate. Specifically, the reference heart rate 6002 may be calculated by multiplying the active phase heart rate by a predetermined value c (eg, c represents a real number less than or equal to 1. As a more specific example, c is 0.8). In addition, in consideration of the fact that the active phase heart rate is generally higher than the first resting phase heart rate, the predetermined value c may be set to be lower than the predetermined value a. Of course, the predetermined value c is not limited to the above example.

The method of setting the reference heart rate 6002 is not limited to a method of calculating a constant ratio of the first and second resting heart rates and the active heart rate, and various calculations such as sum and difference may be applied.

Specifically, according to an embodiment, the drowsiness detecting apparatus 6000 may set the reference heart rate 6002 by adding or subtracting a predetermined value to the first and second resting heart rates or active heart rates. For example, when the first resting heart rate is 75, the drowsiness detecting apparatus may set the reference heart rate 6002 as (first resting heart rate - 10) = 65 (times/minute) based on this. For example, when the second resting heart rate is 65, the drowsiness detecting apparatus 6000 may set the reference heart rate 6002 as (second resting heart rate + 10) = 65 (times/minute) based on this.

Also, in another embodiment, the drowsiness detecting apparatus 6000 may change the previously set reference heart rate 6002 to a new reference heart rate 6002 . That is, the drowsiness detecting apparatus 6000 may set a new reference heart rate 6002 by updating the previously set reference heart rate 6002 .

For example, as described above, the drowsiness detecting apparatus 6000 may set the reference heart rate 6002 based on the heart rate acquired for a predetermined time from the time point when the heart rate of the subject is first acquired. The drowsiness detecting device 6000 may detect a normal state based on the heart rate of the subject acquired after the predetermined time, and may change the average heart rate in the time period detected as the normal state to a new reference heart rate 6002. have. In addition, the drowsiness detecting apparatus 6000 may change to a new reference heart rate 6002 based on the first and second resting and active heart rates acquired for the same subject after the predetermined time. Accordingly, the drowsiness detecting device 6000 may set the reference heart rate 6002 reflecting the most recent state of the subject, and thus the drowsiness detecting apparatus 6000 may set the target heart rate 6002 based on the more accurate reference heart rate 6002. drowsiness can be detected.

As another example, as described above, the drowsiness detecting apparatus 6000 may preset the reference heart rate 6002 based on the first and second resting phase and active phase heart rates. In this case, the drowsiness detecting device 6000 may newly acquire the first and second resting or active heart rates for a predetermined time after the reference heart rate 6002 is set based on the first and second resting or active heart rates. . The drowsiness detecting device 6000 may use the newly acquired first and second resting or active heart rates to set a new reference heart rate 6002 according to the method of setting the reference heart rate 6002 described above.

Referring to FIG. 41 according to an embodiment, in the step of comparing the heart rate 6001 and the reference heart rate 6002 of the subject ( S6120 ), the drowsiness detecting device 6000 sets the heart rate 6001 and the reference heart rate of the subject ( S6120 ). Heart rate 6002 may be compared.

In an embodiment, the drowsiness detecting device 6000 compares the heart rate 6001 and the reference heart rate 6002 of the subject, and the time point t1 when the heart rate 6001 of the subject becomes less than / equal to the reference heart rate 6002 ) can be detected. The sensed time point t1 may be a starting point for measuring the drowsy state and the normal state for each stage.

That is, in the above embodiment, the detection of the time point t1 when the heart rate 6001 of the subject becomes less than/more than the reference heart rate 6002 is to act as a trigger for measuring the drowsiness state and the steady state by stage. can

In the duration measuring step (S6130), the drowsiness detecting device 6000 determines that the heart rate 6001 of the subject is less than or equal to the reference heart rate 6002 based on the time when the heart rate 6001 of the subject becomes less than or equal to the reference heart rate 6002. You can measure how long a state lasts. In addition, the drowsiness detecting device 6000 measures the duration for which the heart rate 6001 of the person to be measured is equal to or greater than the reference heart rate 6002, based on the point in time when the heart rate 6001 of the subject becomes greater than or equal to the reference heart rate 6002 can do.

Heart rate can be fluctuated by various physical factors besides drowsiness. Among various physical factors, since the decrease/increase in heart rate due to drowsiness persists in a decreased/increased state, the continued decrease or increase in heart rate for several seconds to several minutes may be a good biomarker indicative of drowsiness.

For example, when a person's tension is relieved, the heart rate temporarily decreases, but the reduced heart rate can be easily recovered. If the drowsiness detecting device 6000 detects a drowsiness state based on the duration of the reduced heart rate, the drowsiness detecting device 6000 avoids detecting a situation in which the heart rate temporarily decreases due to tension relief as the drowsy state. can Accordingly, the drowsiness detecting device 6000 may more accurately detect the drowsy state.

Also, as the subject enters a state of drowsiness with a higher intensity, the duration of the decreased heart rate due to drowsiness will be longer. Accordingly, if the drowsiness detecting device 6000 measures the length of time during which the decreased heart rate is maintained, it may be possible to determine which stage of the drowsiness state the subject has entered. That is, by dividing the stage of the drowsiness state according to the length of the duration, the drowsiness detecting apparatus 6000 may more specifically detect the drowsiness state of the subject.

Referring to FIG. 41 according to an embodiment, the drowsiness detecting device 6000 determines the heart rate 6001 of the subject based on a time point t1 when the heart rate 6001 and the reference heart rate 6002 become less than or equal to the heart rate 6001 of the subject. It is possible to measure the duration of the state of the reference heart rate 6002 or less.

For example, the reference heart rate 6002 may be set to 72 (times/minute) corresponding to 90% of the first resting heart rate. Here, when the heart rate of the person to be measured at the first time point is 74 (times/minute) and the heart rate of the person to be measured at the second time point immediately after the first time point is 72 (times/minute), the drowsiness detecting device 6000 is set at the second time point. From there, it is possible to measure the duration of the subject's heart rate below 72 (beats/minute).

Of course, according to another embodiment, the drowsiness detecting apparatus 6000 may measure the duration of a state in which the heart rate of the person to be measured is equal to or greater than the reference heart rate based on the time when the heart rate of the person to be measured and the reference heart rate are abnormal.

For example, the reference heart rate 6002 may be set to 72 (times/minute) corresponding to 90% of the first resting heart rate. Here, when the heart rate 6001 of the subject at the third time point is 70 (times/minute) and the heart rate 6001 of the person under measurement at the fourth time point immediately after the third time point is 72 (times/minute), the drowsiness detecting device ( 6000) may measure the duration from the fourth time point in which the heart rate 6001 of the person to be measured is 72 (times/minute) or more.

Referring to FIG. 42 according to an embodiment, the drowsiness detecting device 6000 determines the heart rate including noise when measuring the duration for which the heart rate 6001 of the subject is equal to or less than/higher than the reference heart rate 6002. The duration may be measured by correcting the measured time period Δtn. For example, the drowsiness detecting apparatus 6000 may correct a heart rate detected as noise based on a previously measured heart rate. As another example, the time period Δtn in which the heart rate including noise is measured may be excluded from the duration measurement.

Here, noise represents an error caused by various causes such as movement of the subject or external light, and heart rate including noise may be difficult to reflect the state of the subject. In general, a heart rate including noise is obtained temporarily, and has a large deviation from a previously obtained heart rate of a subject and a heart rate obtained thereafter.

Based on this, in an embodiment, the drowsiness detecting device 6000 determines the previously acquired heart rate 6001 and/or the subsequently acquired heart rate 6001 and the heart rate having a large deviation by more than a predetermined value. When obtained, the heart rate 6001 of the subject may be detected as a heart rate including noise. For example, when the drowsiness detecting device acquires a heart rate that has a deviation of 20 or more from a previously sensed heart rate and/or a later obtained heart rate for a time period of 3 seconds or less, the heart rate of the subject acquired during the 3 seconds (6001) may be detected as a heart rate including noise. Here, the time at which the noise-containing heart rate is acquired and the deviation between the noise-containing heart rate and other heart rates are not limited to a specific value.

According to an embodiment, although the heart rate including noise is a heart rate greater than or equal to the reference heart rate 6002, the drowsiness detecting device 6000 measures the duration during which the heart rate 6001 of the subject is equal to or less than the reference heart rate 6002. , the noise can be treated as a heart rate below the reference heart rate, and the duration can be measured.

According to an embodiment, even though the heart rate including noise is a heart rate less than or equal to the reference heart rate, the drowsiness detecting device 6000 considers the noise to be a heart rate greater than or equal to the reference heart rate when measuring a duration in which the subject's heart rate is equal to or greater than the reference heart rate. It can be processed and its duration can be measured.

According to an embodiment, even though the heart rate including noise is a heart rate less than or equal to the reference heart rate, the drowsiness detecting apparatus 6000 determines that the heart rate including noise is measured when measuring a duration in which the subject's heart rate is equal to or less than the reference heart rate. It can be measured except for the time period (Δtn).

According to an embodiment, even though the heart rate including noise is a heart rate that is less than or equal to the reference heart rate, the drowsiness detecting device 6000 determines that the heart rate including noise is measured when measuring a duration in which the subject's heart rate is equal to or greater than the reference heart rate. It can be measured except for the time period (Δtn).

Referring to FIG. 41 according to an embodiment, in step S6140 of detecting the drowsiness state, the drowsiness detection device 6000 may detect a drowsy state or a normal state by comparing the measured duration with the reference duration. have.

The drowsiness detecting device 6000 measures the duration of a state in which the heart rate 6001 of the subject is below the reference heart rate, and detects the drowsiness state when the measured duration reaches the reference duration (Δta, Δtb, Δtc). can In this case, the reference durations Δta, Δtb, and Δtc may be plural. For example, the reference duration (Δta, Δtb, Δtc) may be three, the first reference duration (Δta) is 30 seconds, the second reference duration (Δtb) is 60 seconds, and the third reference duration ( Δtc) may be set to 90 seconds.

At this time, when the duration for which the heart rate 6001 of the person to be measured remains below the reference heart rate 6002 reaches 30 seconds, the drowsiness detecting device may detect that the person to be measured is in the first stage drowsiness state.

At this time, when the duration in which the heart rate 6001 of the subject is below the reference heart rate 6002 reaches 60 seconds, the drowsiness detecting device 6000 may detect that the subject is in the second stage drowsiness. .

At this time, when the duration for which the subject's heart rate 6001 is below the reference heart rate 6002 reaches 90 seconds, the drowsiness detecting device 6000 may detect that the subject is in the third stage drowsiness state. .

In addition, the drowsiness detecting device 6000 may maintain the sensed drowsiness state until a drowsiness state or a normal state of another stage is detected with respect to the subject whose drowsiness state has been detected.

In another embodiment, if the duration for which the heart rate 6001 of the subject is less than or equal to the reference heart rate 6002 does not reach the first reference duration Δta, the drowsiness detecting device 6000 determines that the subject is in a normal state can be detected as being

For example, if the first reference duration Δta is 30 seconds and the subject's heart rate 6001 is below the reference heart rate 6002 for 10 seconds and then recovers to a heart rate greater than or equal to the reference heart rate, the drowsiness detecting device 6000 ) can be detected that the subject is in a normal state.

Hereinafter, a method for detecting recovery of a drowsy state based on a heart rate will be described with reference to FIG. 43 .

In addition, in the drowsiness-related state detection step (S6140), the drowsiness detection device 6000 detects that the subject has recovered from the drowsy state when the measured duration reaches the recovery reference duration Δtd, Δte, Δtf can do.

The recovery of the drowsiness state may mean a state in which the subject is out of the drowsiness state sensed after the subject enters the drowsiness state. Specifically, the recovery of the drowsiness state may refer to a situation in which a drowsiness state of a lower stage than the drowsiness state of the sensed stage is sensed after the drowsiness state of a predetermined stage is sensed. In addition, the recovery of the drowsy state may mean a situation in which the normal state is detected after the drowsy state is detected.

According to an embodiment, when the time for which the heart rate 6001 of the subject for which the drowsiness has been detected remains equal to or greater than the recovery reference heart rate 6004 reaches the recovery reference duration Δtd, Δte, Δtf, drowsiness is detected The device 6000 may detect recovery of the drowsiness state for the subject.

Specifically, referring to FIG. 46 (a) according to an embodiment, the drowsiness detecting device 6000 maintains a state in which the heart rate 6002 of the subject is greater than or equal to the recovery reference heart rate 6004 from the time point t1. may be measured, and it may be detected that the subject recovers from the drowsy state at time points (t1, t2, t3) when the duration reaches the recovery reference duration.

Here, the recovery reference duration duration The recovery reference duration may be set in various ways. For example, the recovery duration recovery standard duration may be set in various units such as several seconds, tens of seconds, minutes, and tens of minutes.

For example, the recovery reference duration may be set to 30 seconds. In this case, the drowsiness detecting device 6000 detects that the subject has recovered to a normal state when the time for which the heart rate of the subject detected in the third stage drowsiness is equal to or higher than the recovery reference heart rate 6004 reaches 30 can do.

According to an embodiment, a plurality of recovery reference durations may be set. For example, the duration of the recovery reference may be 3, the duration of the first recovery reference may be set to 20 seconds, the duration of the second recovery reference may be set to 40 seconds, and the duration of the third recovery reference may be set to 60 seconds. Here, the first to third recovery reference durations are not limited to the values suggested in the above example.

Referring to (a) of FIG. 43 according to an embodiment, the drowsiness detecting device 6000 indicates that the heart rate 6001 of the subject detected in the third stage drowsiness state is the heart rate 6002 of the subject, the recovery reference heart rate ( 6004) or higher, the duration is measured from the time point (t1) to the recovery standard heart rate (6004) or higher, and at the time point (t2) when the duration reaches the first recovery standard duration time (t2), the subject is drowsy in the second stage It can be detected that the state has been restored.

In addition, after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the drowsiness detecting device 6000 starts at the time t1 when the subject's heart rate 6002 is equal to or higher than the recovery reference heart rate 6004. Measure the duration that the heart rate 6001 of the recovery standard heart rate 6004 or higher continues, and when the duration reaches the second recovery reference duration (t3), the subject is drowsy detecting device (6000) may detect that the subject has recovered to the first stage drowsiness state.

However, referring to FIG. 43B according to an embodiment, after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the subject's heart rate 6002 is equal to or greater than the recovery reference heart rate 6004 From the time point t4, the drowsiness detecting device 6000 sets the heart rate ( When 6001) detects the time point t6 below the recovery reference heart rate 6004, the drowsiness detecting device 6000 may maintain the subject as being in the second stage drowsiness state, and resetting it to the third stage drowsiness state again You may.

It goes without saying that the drowsiness detecting device 6000 may detect the recovery of the drowsy state based on the duration for which the average heart rate 6003 of the subject remains equal to or greater than the recovery reference heart rate 6004 .

The recovery reference heart rate 6004 may be the same as or different from the reference heart rate 6002 for detecting a drowsy state.

The recovery reference duration may be the same as or different from the duration for drowsiness detection.

8.3 Drowsiness detection method based on LF/HF

Hereinafter, a method for detecting drowsiness based on LF/HF according to an embodiment will be described with reference to FIGS. 44 to 47 .

In an embodiment, the method for detecting drowsiness based on LF/HF includes obtaining the LF/HF 6005 of the subject (S6210), and comparing the LF/HF 6005 of the subject and the size of the reference LF/HF It may include the step of doing (S6220) and the step of detecting drowsiness step by step based on the comparison result (S6230).

The method for detecting drowsiness may further include giving a notification according to the drowsiness stage to the outside according to an embodiment.

In the step of obtaining the LF/HF ( S6210 ), the heartbeat information obtaining unit 6100 may obtain the LF/HF 6005 of the subject.

In an embodiment, the drowsiness detecting apparatus 6000 may acquire the LF/HF 6005 of the subject according to a predetermined time period. In another embodiment, the drowsiness detecting device 6000 may acquire the LF/HF 6005 of the subject without following a predetermined time period.

Of course, if the LF/HF 6005 of the subject is measured according to a predetermined time period in the biometric index obtaining device 10 or another device, the drowsiness detecting device 6000 is measured by the biometric index obtaining device 10 or another device. It is possible to obtain the LF/HF 6005 of the subject based on a predetermined time period.

As another example, the drowsiness detecting device 6000 may request the biometric index acquiring device 10 or other device to provide LF/HF according to an input or request from the outside, and whenever the request is made, the subject to be measured LF/HF (6005) of

In an embodiment, the drowsiness detecting apparatus 6000 may acquire the average LF/HF of the subject for a predetermined time period.

The drowsiness detecting device 6000 may have an effect of accurately detecting drowsiness by correcting noise as the average LF/HF is acquired. .

Referring to FIG. 45 according to an embodiment, in the step S6220 of comparing the acquired LF/HF 6005 of the subject and the reference LF/HF 6006, 6007, 6008, the drowsiness detecting device 6000 is acquired It is possible to compare the LF/HF (6005) of the measured subject and the reference LF/HF.

Here, the reference LF/HF (6006, 6007, 6008) may mean a value set as a reference for distinguishing the drowsy state from the normal state. As described above, depending on the activation of the parasympathetic nervous system, the LF/HF in the drowsy state may be smaller than the LF/HF in the normal state.

In an embodiment, the drowsiness detecting device 6000 sets the reference LF/HF (6006, 6007, 6008) by using the LF/HF average of the heartbeat signals acquired for a predetermined time from the time of first acquiring the LF/HF of the heartbeat signal. can be set. The predetermined time may be set to various values such as 1 minute and 5 minutes. In general, there is a high possibility that the subject is in a normal state for a certain period of time from the time when the LF/HF 6005 of the subject is first acquired. Accordingly, the average LF/HF of the subject acquired for the predetermined time is obtained, and the average LF/HF has a predetermined value a (for example, a represents a real number less than or equal to 1. As a more specific example, a is 0.9) can be obtained as a reference LF/HF. In addition, the drowsiness detecting device may set a plurality of reference LF/HF using other predetermined real numbers. The predetermined value a is not limited to the above example.

In another embodiment, the drowsiness detecting device 6000 may set the reference LF/HF 6006 , 6007 , 6008 using the resting phase LF/HF and/or the active phase LF/HF. At this time, the reference LF/HF (6006, 6007, 6008) may be a plurality.

Specifically, the drowsiness detecting device 6000 uses a camera or a wearable device based on a first resting heart rate LF/HF, a second resting LF/HF based on the second resting heart rate, and an active heart rate based on the first resting heart rate. Active phase LF/HF can be obtained.

According to one embodiment, the reference LF/HF (6006, 6007, 6008) may be set based on the first rest period LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) may be set to a constant ratio of the first rest period LF/HF. For example, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF multiplied by a predetermined value 0.9, the first reference LF/HF (6006), and the second reference LF/HF (6007) multiplied by 0.8 and a third reference LF/HF (6008) multiplied by 0.7. Of course, the predetermined value is not limited to the above example.

According to another embodiment, the reference LF/HF (6006, 6007, 6008) may be set based on the second rest period LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) may be set to a constant ratio of the first rest period LF/HF. For example, the reference LF/HF(6006, 6007, 6008) is the first reference LF/HF(6006) multiplied by the predetermined value 1.5 to the first resting period LF/HF(6006), the second reference LF/HF(6007) multiplied by 1.3 and a third reference LF/HF 6008 multiplied by 1.1. Of course, the predetermined value is not limited to the above example.

Also according to another embodiment, the reference LF/HF 6006 , 6007 , 6008 may be set based on the active phase LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006) multiplied by the predetermined value 0.8 by the active phase LF/HF, the second reference LF/HF (6007) multiplied by 0.7, and 0.6 It may be set as the third reference LF/HF 6008 multiplied by . Of course, the predetermined value is not limited to the above example.

The method of setting the reference LF/HF (6006, 6007, 6008) is not limited to a method of calculating a constant ratio of the first and second resting periods LF/HF and the active period LF/HF, and various calculations such as sum and difference may be applied. Of course you can. Also, in another embodiment, the drowsiness detecting device 6000 may change the previously set reference LF/HFs 6006 , 6007 , and 6008 to new reference LF/HFs 6006 , 6007 , and 6008 .

For example, as described above, the drowsiness detecting device 6000 sets the reference LF/HF (6006, 6007, 6008) based on the heart rate acquired for a predetermined time from the time of first acquiring the LF/HF (6005) of the subject. ) can be set. The drowsiness detecting device 6000 may detect the normal state based on the LF/HF 6005 of the subject acquired after the predetermined time, and use the average LF/HF in the time period detected as the normal state as a new standard. It can be changed to LF/HF (6006, 6007, 6008). In addition, the drowsiness detecting device 6000 changes to a new standard LF/HF (6006, 6007, 6008) based on the first and second LF/HF and active phase LF/HF obtained for the same subject after the predetermined time. can

As another example, as described above, the drowsiness detecting device 6000 may preset the reference LF/HFs 6006 , 6007 , and 6008 based on the first and second resting phases and the active phase LF/HF. At this time, the drowsiness detecting device 6000 sets the reference LF/HFs 6006, 6007, and 6008 based on the first and second resting phases LF/HF and the active phase LF/HF, and then, in the same way, New reference LF/HF (6006, 6007, 6008) may be set based on the newly set first and second rest periods LF/HF and active period LF/HF.

Also, the drowsiness detecting device 6000 may set new reference LF/HFs 6006 , 6007 , and 6008 based on the LF/HF acquired for a predetermined time from the time of first acquiring the LF/HF.

In addition, the drowsiness detecting apparatus 6000 may set new reference LF/HFs 6006 , 6007 , and 6008 based on the average LF/HF for a time period in which the subject is detected as a normal state.

Referring to FIG. 45 according to an embodiment, the drowsiness detecting device 6000 is configured such that the LF/HF 6005 of the subject is greater than or less than any one of the plurality of reference LF/HFs 6006, 6007, and 6008. Time points t1, t2, t3, and t4 may be detected. The sensed time point may be a time point for measuring the step-by-step drowsiness state and the normal state.

In an embodiment, the drowsiness detecting device 6000 may detect a drowsiness state at a time point (t1, t2, t3) when the LF/HF (6005) of the subject becomes less than or equal to the reference LF/HF (6006, 6007, 6008). have.

In one embodiment, the drowsiness detecting device 6000 is a time point (t1, t2, t1, t2, when the LF / HF 6005 of the subject becomes less than or equal to one of the plurality of reference LF / HF (6006, 6007, 6008). At t3), the drowsiness state of the stage corresponding to the one reference LF/HF may be detected. Here, the drowsiness detecting device 6000 may detect the drowsiness state based on a result of additionally comparing the LF/HF 6005 of the subject and the one reference LF/HF as well as another reference LF/HF. .

Specifically, the plurality of reference LF/HFs 6006 , 6007 , and 6008 may be set as the first reference LF/HF 6006 , the second reference LF/HF 6007 , and the third reference LF/HF 6008 . . For convenience of description below, the first reference LF/HF 6006 is defined as the reference LF/HF for detecting the lowest intensity drowsiness state.

For example, if the LF/HF 6005 of the subject at the first time point is less than or equal to the first reference LF/HF 6006 and greater than or equal to the second reference LF/HF 6007, the drowsiness detecting device 6000 is the subject. The drowsiness state may be determined as the first stage drowsiness state.

In addition, when the LF/HF 6005 of the subject at the first time point is equal to or less than the second reference LF/HF 6007 and greater than or equal to the third reference LF/HF 6008, the drowsiness detecting device 6000 determines the subject's drowsiness. The state may be determined as a second stage drowsy state.

Also, when the LF/HF 6005 of the subject at the first time point is less than or equal to the third reference LF/HF 6008, the drowsiness detecting device 6000 may determine the drowsiness state of the subject as the third stage drowsiness state. have.

Hereinafter, with reference to FIG. 46, a method for detecting recovery of a drowsy state based on LF/HF will be described.

Referring to FIG. 46 (a) according to an embodiment, in the step of detecting a state related to drowsiness (S6230), the drowsiness detecting device 6000 determines that the measured LF/HF 6005 of the subject is the recovery reference LF. At a time point (t1, t2, t3, t4) equal to or greater than /HF (6006, 6007, 6008, 6010), it can be detected that the subject has recovered from the state of drowsiness. Here, the drowsiness detecting device 6000 detects the recovery of the drowsy state based on the result of additional comparison of the subject's LF/HF 6005 with the one recovery standard LF/HF and the other recovery standard LF/HF. can

In one embodiment, the plurality of reference LF/HFs 6006 , 6007 , 6008 , 6010 includes a first recovery reference LF/HF 6006 , a second recovery reference LF/HF 6007 , and a third recovery reference LF/HF (6008). For convenience of description below, the first recovery reference LF/HF 6006 is set to be the recovery reference LF/HF having the largest value among the recovery reference LF/HF of 1 or less. For example, the LF/HF (6005) of the subject at the first time point is equal to or less than the third recovery reference LF/HF (6008), and the LF/HF (6005) of the subject at the second time point thereafter is the third recovery point. When the reference LF/HF (6008) or more and the second recovery reference LF/HF (6007) or less, the drowsiness detection device 6000 indicates that the drowsiness state of the subject at the second time point is changed from the third stage drowsiness state to the second stage drowsiness state. It can be judged that the state has been restored. In addition, at a third time point after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the drowsiness detecting device 6000 determines that the LF/HF 6005 of the subject is set to the second recovery standard LF/HF ( 6007) and less than the first recovery criterion LF/HF 6006, the drowsiness detecting device 6000 determines that the drowsiness state of the subject at the third time point is restored from the second stage drowsiness state to the first stage drowsiness state can do.

In an embodiment, the LF/HF 6005 of the subject at the first time point is less than or equal to the third recovery reference LF/HF 6008, and the LF/HF 6005 of the subject at the second time point thereafter is the second When the recovery standard LF/HF (6007) or more and the first recovery reference LF/HF (6006) or less, the drowsiness detecting device 6000 determines that the drowsiness state of the subject at the second time point is the third stage drowsiness state and the first stage It can be judged that he has recovered to a state of drowsiness.

However, referring to FIG. 46 (b) according to an embodiment, after a time point t6 when the subject recovers from the third stage drowsiness state to the first stage drowsiness state, the drowsiness detecting device 6000 is When the LF/HF (6005) of LF/HF (6005) is reduced to less than the second recovery reference LF/HF (6007) within a predetermined time (t7) is sensed, the drowsiness detection device 6000 determines that the subject is in the second stage drowsiness. can be perceived as In addition, it may be detected that the subject has reset to the third stage drowsy state.

The first recovery reference LF/HF 6006 is a first reference LF/HF 6006 in FIG. 45, and the second recovery reference LF/HF 6010 is a second reference LF/HF ( 6007) and the third recovery reference LF/HF 6011 may be the same as or different from the third reference LF/HF 6006 in FIG. 45 .

In addition, in order to detect that the drowsiness detecting device 6000 recovers from a drowsy state to a normal state, a fourth recovery reference LF/HF 6010 having a value greater than the first to third recovery reference LF/HF may be set. have. For example, when the drowsiness detecting device 6000 detects a point in time when the LF/HF 6005 of the subject sensed as drowsy is greater than or equal to the fourth recovery reference LF/HF 6010, the subject It may be judged that the drowsy state has returned to a normal state. In general, when the subject is in a steady state, LF/HF is measured within the range of 0.9~1. Therefore, even when the subject is detected as drowsy with a very small intensity, it may be difficult for the LF/HF of the subject to have a value of 1 or more. That is, by setting the fourth recovery reference LF/HF 6010 having a value of 1 or more as the recovery reference LF/HF, the drowsiness detection device 6000 determines that the state is in a normal state only when the subject fully recovers from the drowsiness state. You may.

8.4 Drowsiness detection method based on heart rate and LF/HF

Hereinafter, a method for detecting drowsiness based on a heart rate and LF/HF will be described with reference to FIG. 47 .

In an embodiment, the step of detecting drowsiness based on the heart rate and LF/HF (S6100, S6200) and determining the final drowsiness state (S6300) may be included.

The method for detecting drowsiness may further include giving a notification according to the stage of the drowsiness state to the subject/non-measurement subject according to an embodiment.

This table of contents describes the drowsiness detection method in which both the heart rate-based drowsiness detection method described in Table 8.3 and the LF/HF-based drowsiness detection method described in Table 8.4 are applied.

Although the heart rate and LF/HF are based on heart rate information obtained from the same subject, in some cases, the stage of the drowsy state determined based on the heart rate and the stage of the drowsiness state determined based on the LF/HF may be different. have. For example, the drowsiness detecting apparatus may detect the third-stage drowsiness state based on the heart rate and detect the second-stage drowsiness state based on the LF/HF at the same time point from the same subject. As such, the fact that different levels of drowsiness were determined according to the drowsiness detection method despite the drowsiness detection device determining the drowsiness state for the same subject at the same time point means that the drowsiness state determination result by any one drowsiness detection method may be wrong. This may mean that there is

Accordingly, in order to reduce such errors and detect drowsiness more accurately, a drowsiness detection method based on both the heart rate and LF/HF will be described below.

In an embodiment, the state related to drowsiness detected by the method for detecting drowsiness based on the heart rate and LF/HF may be expressed as a final drowsiness state and a final steady state. In an embodiment, the final drowsiness state may be divided into a plurality of stages according to the intensity of drowsiness. For example, it may be divided into a first final drowsiness state, a second final drowsiness state, and a third final drowsiness state. Here, the first final drowsiness state may mean a final drowsiness state with the lowest intensity, and the third final drowsiness state may mean a final drowsiness state with the highest intensity.

In an embodiment, the final drowsiness state and the final steady state may mean a state corresponding to the drowsiness state and the normal state detected based on the drowsiness detection method based on the heart rate and the sleepiness detection method based on the LF/HF, respectively. can That is, the final drowsiness state may mean a state in which the subject is temporarily sleeping or a state with a high probability of falling asleep within a predetermined time before falling asleep. It may mean a state in which the probability of falling asleep in time is low.

As described above, the drowsiness detection method based on the heart rate change and the LF/HF value is divided into four stages: a normal state, a first stage drowsiness state, a second stage drowsiness state, and a third stage drowsiness state, depending on the intensity of drowsiness, respectively. can be divided Here, the first stage drowsiness state may mean an unconscious/unconscious drowsiness state, and the second and third stage drowsiness states may mean a conscious/conscious drowsiness state.

In an embodiment, the final drowsiness state divided into a plurality of stages according to the intensity of drowsiness is the first stage drowsiness detected based on each of the heart rate-based drowsiness sensing method and the LF/HF-based drowsiness sensing method It may mean a state corresponding to the state, the second stage drowsiness state, and the third stage drowsiness state. Accordingly, the first final drowsiness state may mean an unconscious/unconscious drowsiness state. Also, the second final drowsiness state and the third final drowsiness state may mean a conscious/conscious drowsiness state. In an embodiment, the drowsiness detecting device may acquire the final drowsiness state of the subject according to a predetermined time period. Of course, the predetermined time period may be fixed or variable.

In another embodiment, the drowsiness detecting apparatus may acquire the final drowsiness state of the subject without following a predetermined time period. For example, the drowsiness detecting device may acquire the drowsiness of the subject whenever the heart rate and/or LF/HF of the subject is measured by the biometric index obtaining apparatus 10 or another device, that is, in real time.

Of course, if the heart rate or LF/HF of the subject is measured according to a predetermined time period in the biometric index obtaining device 10 or another device, the drowsiness detecting device may perform a predetermined time period measured by the biometric index obtaining device 10 or another device. Based on , the final drowsiness state of the subject can be obtained.

As another example, the drowsiness detection device may request the biometric index acquisition device 10 or another device to provide LF/HF and heart rate according to an input or request from the outside, and whenever the request is made, the subject's You can get the final drowsiness state

In an embodiment, the drowsiness detecting apparatus may acquire the average heart rate and average LF/HF of the subject for a predetermined time period.

As the drowsiness detection device acquires the average heart rate and average LF/HF, noise is corrected to have the effect of accurately detecting drowsiness, and the effect obtained by acquiring the average heart rate and average LF/HF The description will be omitted.

In an embodiment, the drowsiness detecting device may determine the final drowsiness stage according to the drowsiness stage detected based on the heart rate and the drowsiness state stage detected based on the LF/HF. In some cases, the phase of the sleepy state detected based on the heart rate may or may not coincide with the phase of the sleepy state detected based on the LF/HF.

Hereinafter, for convenience of description, the normal state sensed based on the heart rate is defined as a drowsy state of a lower stage than the first stage drowsiness state. In addition, the normal state sensed based on LF/HF is also defined as a drowsy state of a lower stage than the first stage drowsiness state.

When the stage of the drowsy state detected based on the heart rate matches the stage of the drowsiness state detected based on the LF/HF, the drowsiness detecting device may determine the matching stage as the final stage of the drowsiness state. For example, when the drowsiness detecting device detects the first stage drowsiness state based on the heart rate of the subject and detects the first stage drowsiness state based on LF/HF, the drowsiness detecting device sets the first stage drowsiness state as the final drowsiness state. The final state of drowsiness can be determined. In addition, when the drowsiness detecting device detects the second stage drowsiness state based on the heart rate of the subject and detects the second stage drowsiness state based on the LF/HF, the drowsiness detecting device returns the final drowsiness state to the second stage final drowsiness state It can determine the state of drowsiness. In addition, when the drowsiness detecting device detects the third stage drowsiness state based on the heart rate of the subject and detects the third stage drowsiness state based on the LF/HF, the drowsiness detecting device enters the third stage final drowsiness state It can determine the state of drowsiness.

When the stage of the drowsiness state sensed based on the heart rate and the stage of the drowsiness state sensed based on the LF/HF do not match, the drowsiness detection device may determine the final drowsiness stage in consideration of various situations.

In an embodiment, when the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on the LF/HF do not match, the drowsiness detection device sets the higher stage of the two stages to the final stage of the drowsiness state can be decided with For example, when the drowsiness detecting device detects the first stage drowsiness state based on the heart rate of the subject and detects the second stage drowsiness state based on LF/HF, the drowsiness sensing device sets the final drowsiness state to the second stage drowsiness state. The final state of drowsiness can be determined.

In another embodiment, when the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, the drowsiness detection device sets the lower stage of the two stages to the final stage of the drowsiness state can be decided with For example, when the drowsiness detecting device detects the first stage drowsiness state based on the heart rate of the subject and detects the second stage drowsiness state based on LF/HF, the drowsiness sensing device sets the first drowsiness state as the final drowsiness state. The final state of drowsiness can be determined.

In another embodiment, when the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, the drowsiness detection device determines the average of the two stages as the final drowsiness stage. can Here, when the average is not an integer, rounding or rounding may be applied to the number of the first decimal place. For example, if the drowsiness detecting device detects the first stage drowsiness state based on the heart rate and detects the third stage drowsiness state based on the LF/HF, the drowsiness detecting device sets two final drowsiness states as the final drowsiness state. It is possible to determine the final drowsiness state of the second stage, which is the average of the stages. As another example, when the first stage drowsiness state is detected based on the heart rate and the second stage drowsiness state is detected based on the LF/HF, the drowsiness detecting device sets the final drowsiness state to 1.5, which is the average of the two stages. Rounding off to one decimal place can be applied to determine the final state of sleepiness in the first stage. Also, the drowsiness detecting device may determine the final drowsiness state of the second stage by rounding up to the first decimal place with respect to 1.5, which is the average of the two stages, as the final drowsiness state.

Of course, in some cases, the drowsiness detection device newly defines a drowsiness stage between the drowsiness stage detected based on the heart rate and the drowsiness stage detected based on the LF/HF, and finalizes the newly defined drowsiness stage. It can be determined as the drowsiness stage. For example, when the first stage drowsiness state is detected based on the heart rate and the second stage drowsiness state is detected based on the LF/HF, the drowsiness detecting device is configured to be drowsy between the first stage drowsiness state and the second stage drowsiness state An intermediate stage drowsiness state representing the intensity may be defined, and the intermediate stage drowsiness state may be determined as the final drowsiness stage.

In another embodiment, when the stage of the drowsiness state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, the drowsiness detection device sets the previously determined final drowsiness state or the final steady state to drowsiness. can be maintained as related. For example, in a situation in which the drowsiness detecting device detects the final drowsiness state of the first stage for the subject, the first stage drowsiness state is detected based on the heart rate and the third stage drowsiness state is detected based on the LF/HF. In one case, the drowsiness detecting device may maintain the first stage final drowsiness state, which is the previously determined final drowsiness state, as a state related to drowsiness. For example, in a situation in which the drowsiness detecting device detects the final normal state for the subject, when detecting the first stage drowsiness state based on the heart rate and detecting the third stage drowsiness state based on the LF/HF, drowsiness detection The device may maintain the previously determined final steady state as the state relating to drowsiness.

Hereinafter, a case in which the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state sensed based on LF/HF do not coincide will be described in detail.

8.4.1 Specific Examples

In an embodiment, when the stage of the drowsy state sensed based on the heart rate and the stage of the drowsiness state sensed based on the LF/HF coincide with the normal state, the drowsiness detecting device may determine that the subject is in the final normal state. .

In one embodiment, when the stage of the drowsiness state detected based on the heart rate and the stage of the drowsiness state sensed based on the LF/HF coincide as the first stage drowsiness state, the drowsiness detecting device determines that the subject is drowsy in the first stage final drowsiness status can be determined.

In an embodiment, when the stage of the drowsiness state sensed based on the heart rate and the stage of the drowsiness state sensed based on LF/HF coincide as the second stage drowsiness state, the drowsiness detecting device determines that the subject is drowsy in the second stage final drowsiness status can be determined.

In an embodiment, when the stage of the drowsiness state detected based on the heart rate and the stage of the drowsiness state sensed based on the LF/HF coincide as the third stage drowsiness state, the drowsiness detecting device determines that the subject is drowsy in the third stage final drowsiness status can be determined.

In an embodiment, the phase of the sleepy state detected based on the heart rate and the phase of the sleepy state detected based on LF/HF do not match, and the phase of the sleepy state detected based on the heart rate and the detection based on LF/HF The method of determining the lower stage among the stages of the drowsy state as the stage of the final drowsiness state (hereinafter referred to as the first method) can be applied as a method for detecting a stage in which the need to be sensitively sensed is low. .

For example, a notification may be given when first, second, and third stage drowsiness states are detected. In some cases, the first and second stage drowsiness states may have little effect on the safety of the subject. In this case, if the notification is given frequently, the subject may feel uncomfortable or may not operate the drowsiness detecting device. In addition, as the lower stage of the drowsiness stage detected based on heart rate and the drowsiness stage detected based on LF/HF is determined as the final drowsiness stage, the drowsiness detection device more accurately determines the drowsiness state. may detect it. That is, it may be advantageous to improve user convenience and accuracy to determine the final drowsiness state using the drowsiness detecting device according to the first method.

In one embodiment, the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, one of the drowsiness stages indicates the normal state, and the other drowsy state When the drowsiness state is detected as one or two stages higher than the normal state, the drowsiness detecting device may determine that the subject is in the final normal state based on the lower stage of the steady state.

For example, when the drowsiness detecting device detects the normal state based on the heart rate and detects the first stage drowsiness based on the LF/HF, the drowsiness detecting device may determine that the subject is in the final normal state.

In one embodiment, the phase of the sleepy state detected based on the heart rate and the phase of the sleepy state detected based on LF/HF do not match, and any one of the sleepy phases indicates the first phase sleepiness, and the other When the drowsiness stage of the drowsiness state is detected as a drowsiness state that is one level higher than the normal state, the drowsiness detecting device may determine that the subject is in the first stage drowsiness state based on the first stage drowsiness state, which is a lower stage.

For example, when the drowsiness detecting device detects the first stage drowsiness state based on the heart rate and detects the second stage drowsiness state based on the LF/HF, the drowsiness detecting device determines that the subject is in the first stage drowsiness state it can be decided that

In one embodiment, the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, and any one of the drowsy stages indicates the second stage drowsiness, and the other When it is detected that the level of drowsiness is higher than that of the normal state, the drowsiness detecting device may determine that the subject is in the second level of sleepiness based on the second level of sleepiness, which is the lower level. On the other hand, the heart rate The stage of the sleepy state detected based on LF/HF does not match the stage of the sleepy state detected based on A method of determining the highest stage among the final drowsiness stages (hereinafter referred to as a second method) may be applied as a method for detecting a stage with a high necessity to be sensitively detected.

For example, if the first, second, and third stage drowsiness is detected, a notification may be given. can At this time, if the drowsiness detecting device does not give a strong notification to the subject as soon as the third stage drowsiness is detected, the subject may be in a serious danger. In addition, as the higher stage of the sleepy state detected based on heart rate and the sleepy state detected based on LF/HF is determined as the final sleepy state stage, the drowsiness detection device tries to detect more various situations. This stage can be determined as the stage of final drowsiness. That is, it may be advantageous to prepare for user risk to determine the final drowsiness state using the drowsiness detecting device according to the second method.

In one embodiment, the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, and any one of the drowsiness stages indicates the third stage drowsiness state, and the other When the drowsiness stage of the drowsiness state is detected to be one, two or three stages lower than the third stage drowsiness state, the drowsiness detecting device determines the third stage based on the third stage drowsiness state, which is the higher stage. It can be determined that the final state of drowsiness is.

For example, when the drowsiness detecting device detects the third stage drowsiness state based on the heart rate and detects the first stage drowsiness state based on the LF/HF, the drowsiness detecting device determines that the subject is in the third stage drowsiness state can decide

In another embodiment, the drowsiness detecting device may follow the second method to detect the second final drowsiness state. This is because, in some cases, the drowsiness detecting device may need to detect not only the third stage drowsiness state but also the second stage drowsiness state in more diverse situations. Specifically, when the subject is a driver of a freight vehicle or a driver of a public transportation vehicle, when the driver is at risk of a traffic accident, a situation may occur in which the driver and passengers of surrounding vehicles other than the driver are in serious danger. Therefore, the stage of the drowsy state detected based on the heart rate and the stage of the drowsiness state detected based on LF/HF do not match, one of the drowsy stages indicates the second stage drowsiness, and the other drowsy state When the stage of drowsiness is detected as one or two stages lower than the second stage drowsiness state, the drowsiness detecting device determines that the subject is in the second stage final drowsiness state based on the second stage drowsiness state, which is a higher stage. may be

For example, when the drowsiness detecting device detects the second stage drowsiness state based on the heart rate and detects the first stage drowsiness state based on the LF/HF, the drowsiness detecting device determines that the subject is in the second stage drowsiness state can decide

9. Various applications using the biometric index acquisition device

9.1 Various embodiments of the display device

The above-described method for obtaining a biometric index or a method for detecting drowsiness may be used in various applications. For example, the method for obtaining the biometric index or the method for detecting drowsiness may be used in a display device.

In this case, the display device includes, but is not limited to, a smart mirror including a mirror display, and may refer to a device including a display, such as a smart phone or a tablet.

In addition, the biometric index or biometric information obtained according to the method for obtaining the biometric index may be output through the display, and the drowsiness information obtained according to the method for detecting drowsiness may also be output through the display.

In addition, the display device may perform an operation corresponding to the biometric index or biometric information obtained according to the method for obtaining the biometric index.

Also, the display device may perform an operation corresponding to the drowsiness information obtained according to the drowsiness detection method.

In addition, the user of the display device may set the biometric of interest, and when the biometric of interest is set, only the biometric of interest may be obtained or output. For example, when the first user sets the heart rate and the second user sets the blood pressure, only the heart rate may be obtained from the first user and only the blood pressure may be obtained from the second user, but is not limited thereto.

In addition, the display device may be used to measure the biometric index of people who are provided in public places such as a hotel lobby, a hotel front, and a public toilet, but is not limited thereto.

In addition, although the operation of the biometric index measuring device will be described below, the described content may be performed by other processors such as ECUs mounted on a vehicle, and may also be performed by a processor included in a server.

Also, the device described as the biometric index measuring device may mean an image acquisition device, and in this case, an operation according to the description may be performed by another processor such as an ECU or a processor included in a server.

48 is a diagram for explaining a smart mirror device according to an embodiment.

48 , the smart mirror device 7000 according to an embodiment may include at least one of an image sensor 7010 , a mirror display 7020 , and a controller 7030 . For example, the smart mirror device 7000 may include the mirror display 7020 and the controller 7030, but is not limited thereto.

At this time, the image sensor 7010 may be provided as a visible light camera for obtaining a visible light image, an infrared camera for obtaining an infrared image, etc., but is not limited thereto, and a visible light image and A hybrid type camera for acquiring an infrared image may be provided.

Also, the image sensor 7010 may be provided as a single package with the mirror display 7020 , but is not limited thereto and may be provided as a separate unit separate from the mirror display 7020 .

Also, the image sensor 7010 may acquire a plurality of image frames and transmit the acquired images to the controller 7030 .

In addition, the mirror display 7020 may refer to a medium capable of transmitting information while functioning as a mirror, but is not limited thereto.

In addition, the mirror display 7020 may include a mirror capable of functioning as a mirror and a display capable of transmitting information, and may be provided in a form in which a mirror film is added to the display, but is not limited thereto.

In addition, the mirror display 7020 may include a half mirror that transmits light in one direction and reflects light in the other direction, but is not limited thereto.

In addition, the mirror display 7020 may include a polarizing plate, but is not limited thereto.

In addition, the mirror display 7020 may include a mirror display that can be commonly understood in addition to the above-described exemplary principles and exemplary elements, and more specifically, a device that functions as a mirror and transmits information. can be understood

In addition, the controller 7030 may acquire a biometric index and biometric information based on a plurality of image frames obtained from the image sensor 7010 .

At this time, the method for the controller 7030 to obtain the biometric index and biometric information based on the plurality of image frames obtained from the image sensor 7010 can be applied to the above description, and thus the overlapping description will be omitted. .

Also, the control unit 7030 may obtain a biometric index and biometric information from an external sensor (not shown). For example, the controller 7030 may acquire heart rate information through an ECG sensor attached to the subject, but is not limited thereto.

Also, the controller 7030 may acquire personal and statistical data through an input device (not shown). For example, the control unit 7030 may acquire personal and statistical data such as height, age, and weight of the subject through a keyboard, but is not limited thereto.

In this case, the input device may be a keyboard, a mouse, a key pad, a dome switch, a touch pad (static pressure/capacitance), a jog wheel, a jog switch, etc. However, the present invention is not limited thereto.

Also, the controller 7030 may acquire personal and statistical data through an external device (not shown). For example, the control unit 7030 may obtain personal and statistical data such as height, age, and weight of the subject through the smartphone of the subject, but is not limited thereto.

In this case, the external device may include a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a navigation device, etc. In addition, it may include a fixed terminal such as a digital TV, a desktop computer, etc., but is not limited thereto.

Also, the controller 7030 may control the operation of the mirror display 7020 . For example, the controller 7030 may control the operation of the mirror display 7020 to output the biometric index or biometric information obtained based on the image frame, but is not limited thereto.

Also, the controller 7030 may control the operation of the mirror display 7020 to output various information. For example, the control unit 7030 may control the operation of the mirror display 7020 to output various information such as weather information, date information, calendar information, internal humidity information, and internal temperature information, but is not limited thereto. .

Also, the controller 7030 may control the operation of the mirror display 7020 to output various personal information. For example, the controller 7030 may control the operation of the mirror display 7020 to output various personal information such as schedule information and medication information of the subject, but is not limited thereto.

49 is a view for explaining a smart mirror device according to an embodiment.

49 , a smart mirror device 7100 according to an embodiment may include an image sensor 7110 and a mirror display 7120 .

In this case, since the above-described operations may be applied to the image sensor 7110 and the mirror display 7120 , overlapping descriptions will be omitted.

According to an embodiment, basic information may be output to the mirror display 7120 . For example, at least one of basic information such as date information, time information, external temperature information, weather information, calendar information, and news information may be output to the mirror display 7120 as shown in FIG. However, the present invention is not limited thereto.

Also, personal information may be output to the mirror display 7120 . For example, as shown in FIG. 49 , at least one piece of personal information such as the age, height, and weight of the subject may be output on the mirror display 7120 , but the present invention is not limited thereto.

Also, for example, at least one of personal information such as schedule information and medication information of a subject may be output to the mirror display 7120 as shown in FIG. 49 , but the present invention is not limited thereto.

In addition, the biometric index of the subject may be output to the mirror display 7120 . For example, as illustrated in FIG. 49 , at least one bio-index among bio-indexes such as heart rate, oxygen saturation, and blood pressure may be output to the mirror display 7120 , but the present invention is not limited thereto.

In this case, the biometric index may be obtained based on the image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained by an external sensor or the like.

In addition, biometric information of the subject may be output to the mirror display 7120 . For example, at least one of biometric information such as condition information may be output to the mirror display 7120 as shown in FIG. 49 , but the present invention is not limited thereto.

In this case, the biometric information may be obtained based on the image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained by an external sensor or the like.

In addition, the biometric information may be obtained based on at least one biometric index among the biometric indices, but is not limited thereto.

Also, a biosignal of the subject may be output to the mirror display 7120 . For example, at least one biosignal among biosignals such as a heartbeat signal may be output to the mirror display 7120 as shown in FIG. 49 , but the present invention is not limited thereto.

In this case, the biosignal may be obtained based on an image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained based on an external sensor.

Also, the mirror display 7120 may include an input device. For example, as shown in FIG. 49 , the mirror display 7120 may include at least one of input devices such as a touch panel, but is not limited thereto.

Also, for example, although not shown in FIG. 49 , the smart mirror device 7100 may receive an input from the user by tracking the user's pupil or by recognizing the user's gesture, but is not limited thereto.

Also, the mirror display 7120 may acquire information on the subject through the input device, and the obtained information on the subject may be output to the mirror display 7120 .

In addition, the drawings and descriptions of the above-described mirror display 7120 are only examples, and it is obvious that various information can be output in various ways without being limited to FIG. 49 and related descriptions.

9.1.1 Various embodiments of a display device outputting a guide area

50 is a diagram for explaining a smart mirror device outputting a guide area according to an exemplary embodiment.

Referring to FIG. 50 , a smart mirror device 7150 according to an embodiment may include an image sensor 7160 and a mirror display 7170 , and the image sensor 7160 and the mirror display 7170 have the above-mentioned details. Since one content may be applied, redundant descriptions will be omitted.

Also, a guide area 7180 may be displayed on the mirror display 7170 according to an embodiment. In this case, the guide area 7180 may function to roughly hold the measurement position of the measurement target.

When the biometric index is obtained using an image sensor, measurement accuracy may differ depending on the measurement location of the person under test. It is possible to improve the accuracy of the biometric index measurement.

In addition, the guide area 7180 may be displayed in a rectangular shape as shown in FIG. 50 , but is not limited thereto, and may be displayed in various shapes such as a human face outline, a circle shape, an oval shape, and the like.

Also, the displayed position of the guide area 7180 may be changed according to the subject. For example, the guide area 7180 for a tall person may be located on the mirror display 7170, and the guide area 7180 for a short person is relatively small. It may be located under the mirror display 7170, but is not limited thereto.

Also, the position of the guide area 7180 may be changed in real time. For example, when the subject moves during measurement, the location of the guide area 7180 may change in real time in response to the movement of the subject, but is not limited thereto.

Also, the guide area 7180 may be displayed before measuring the biometric index of the subject. For example, the guide area 7180 may be displayed before the person to be measured enters the measurement area and may function to inform an approximate measurement position of the person to be measured, but is not limited thereto.

In addition, the guide area 7180 may function to notify the subject to be measured when there are a plurality of persons who may be the subject of measurement. For example, when person A and person B enter the measurement area, but only the person B becomes a subject, the guide area 7180 may be displayed corresponding to the person B, but is not limited thereto.

9.1.2. Various embodiments of a display device for displaying predetermined information during measurement of a biometric index

When a biometric index is obtained using an image sensor, measurement accuracy may differ depending on the movement of the subject, so a function for minimizing the movement of the subject between measurements of the biometric index may be required.

Accordingly, when a display device that displays predetermined information during measurement of a biometric index is used, the movement of the subject can be induced to be minimized by using the displayed information.

51 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment.

Referring to FIG. 51 , different information may be displayed on the smart mirror device 7200 according to the biometric index measurement time according to an embodiment.

More specifically, at the first time point 7210 , the measurement of the biometric index for the subject may be started.

In this case, the first time point 7210 may be a time point at which the measurement target area of the person to be measured is located within the measurement area, but is not limited thereto, and may be a time point at which the measurement target person is located within the field of view of the image sensor. can

Also, the first time point 7210 may be a time point at which the person to be measured inputs an intention for measurement. For example, the first time point 7210 may be a time point when the subject touches a measurement button, but is not limited thereto.

Also, the first information may be displayed at the second time point 7220 .

In this case, the second time point 7220 may be a time point at which face recognition for the subject is completed, but is not limited thereto.

In addition, the second time point 7220 may mean a time point after a predetermined time has elapsed after the measurement of the biometric index for the subject starts, but is not limited thereto.

Also, the first information may be recognition information about the subject. For example, as shown in FIG. 51 , the first information displayed at the second time point 7220 may be greeting information about the subject whose face has been recognized, but is not limited thereto.

Also, the first information may be information about a measurement progress time. For example, when the second time point 7220 is a time point 2 seconds after the start of measurement, information corresponding to 2 seconds may be displayed.

In addition, the first information may be information about the remaining time for measurement. For example, it takes 6 seconds to measure the biometric index, and when the second time point 7220 is a time point 2 seconds after the measurement starts, information corresponding to the remaining 4 seconds may be displayed.

Also, second information may be displayed at the third time point 7230 .

In this case, the third time point 7230 may mean a time point after a predetermined time has elapsed after the face recognition of the subject is completed, but is not limited thereto.

In addition, the third time point 7230 may mean a time point after a predetermined time has elapsed after the measurement of the biometric index for the subject starts, but is not limited thereto.

Also, the second information may be health information about the subject. For example, as shown in FIG. 51 , the second information displayed at the third time point 7230 may be a monthly average biometric index of the subject, but is not limited thereto.

Also, the second information may be information about a measurement progress time. For example, when the third time point 7230 is a time point 4 seconds after the start of measurement, information corresponding to 4 seconds may be displayed.

In addition, the second information may be information about the remaining time for measurement. For example, it takes 6 seconds to measure the biometric index, and when the third time point 7230 is a time point 4 seconds after the measurement starts, information corresponding to the remaining 2 seconds may be displayed.

Also, third information may be displayed at the fourth time point 7240 .

In this case, the fourth time point 7240 may mean a time point at which the measurement of the biometric index for the subject is completed, but is not limited thereto.

Also, the third information may be information related to a biometric index for the subject. For example, as shown in FIG. 51 , the third information displayed at the fourth time point 7240 may be a heart rate, oxygen saturation, and blood pressure of the subject, but is not limited thereto.

Also, the third information may be information related to biometric information about the subject. For example, although not shown in FIG. 51 , the fourth information displayed at the fourth time point 7240 may be biometric information such as a condition index of the subject, but is not limited thereto.

9.1.3 Various embodiments of a smart mirror device that displays an image acquired through an image sensor

52 is a view for explaining a smart mirror device according to an embodiment.

Referring to FIG. 52 , a smart mirror device 7250 according to an embodiment may include an image sensor 7260 and a mirror display 7270 .

In this case, since the above-described operations may be applied to the image sensor 7260 and the mirror display 7270 , overlapping descriptions will be omitted.

According to an embodiment, an image may be displayed on the mirror display 7270 . For example, an image acquired through the image sensor 7260 may be displayed on the mirror display 7270 as shown in FIG. 52 , but is not limited thereto.

In addition, information on a subject to be measured may be displayed on the displayed image. For example, an area displaying a measurement target included in the image may be displayed, but is not limited thereto.

In this way, even though the subject can see his/her own appearance through the mirror display 7270, the additional display of the image makes it possible to clearly display the subject who is the subject of the biometric index measurement when a plurality of measurement subjects exist. It is possible to prevent confusion among multiple personnel.

In addition, the displayed image of the subject and the image of the subject reflected through the mirror display 7270 may be in different positions, but the present invention is not limited thereto and may overlap each other.

9.1.4. Various embodiments of a display device for measuring a bio-index in real time

53 is a diagram for describing a display device for measuring a biometric index in real time, according to an exemplary embodiment.

Referring to FIG. 53 , the display device 7300 according to an embodiment may measure the biometric index in real time, and the display device 7300 may display the biometric index in real time.

More specifically, at the first time point 7310 , the biometric index for the subject acquired in the first time interval may be displayed.

In this case, the first time point 7310 may be a time point after the biometric index for the subject is obtained in the first time interval.

Also, at the second time point 7320 , the biometric index for the subject acquired in the second time interval may be displayed.

In this case, the second time point 7320 may be a time point after the biometric index for the subject is obtained in the second time interval.

In addition, the second time period may be different from the first time period, and may overlap at least partially.

Also, the second time point 7320 may be a later time point than the first time point 7310 .

Also, at the third time point 7330 , the biometric index for the subject acquired in the third time period may be displayed.

In this case, the third time point 7330 may be a time point after the biometric index for the subject is obtained in the third time interval.

In addition, the third time interval may be different from the first and second time intervals, and may overlap at least partially.

Also, the third time point 7330 may be a later time point than the first and second time points 7310 and 7320 .

Also, at the fourth time point 7340 , the biometric index for the subject acquired in the fourth time period may be displayed.

In this case, the fourth time point 7340 may be a time point after the biometric index for the subject is obtained in the fourth time interval.

Also, the fourth time interval may be different from the first, second, and third time intervals, and may overlap at least partially.

Also, the fourth time point 7340 may be a later time point than the first, second, and third time points 7310 , 7320 , and 7330 .

In addition, when the biometric index is acquired in real time as described above, the biometric index obtained at a specific time may be used to store the final biometric index. For example, the biometric index obtained at the time point when the biometric index is first obtained may be stored, but the present invention is not limited thereto. It is also possible to store the biometric index.

In addition, when the biometric index is obtained in real time as described above, a plurality of biometric indexes may be used to store the final biometric index. For example, an average between a plurality of biometrics acquired during a predetermined time period may be stored as a final biometric index, but is not limited thereto.

In addition, when the biometric index is acquired in real time as described above, even if noise is generated according to the movement of the subject between the biometric index measurements, since a plurality of biometric indices are acquired, more accurate biometric indexes can be obtained.

In addition, when the biometric index is acquired in real time as described above, a period for updating the biometric index to be displayed may be set. For example, the user may set a cycle at which the biometric index is updated to be displayed, and the biometric index may be updated according to the set cycle, but is not limited thereto.

9.1.5. Various embodiments of a display device for displaying predetermined information during measurement of a biometric index

54 is a diagram for explaining a smart mirror device outputting predetermined information according to an embodiment.

Referring to FIG. 54 , different information may be displayed on the smart mirror device 7350 according to the measurement time of the biometric index according to an embodiment.

More specifically, the first time point 7360 may be a time point before or before the measurement of the biometric index for the subject is started.

In this case, the first time point 7360 may be a time point at which the measurement target area of the person to be measured is located within the measurement area, but is not limited thereto, and may be a time point at which the measurement target person is located within the field of view of the image sensor. can

Also, first information may be displayed at the first time point 7360 .

In this case, the first information may include basic information. For example, as shown in FIG. 54 , basic information such as date, time, and weather information may be included, but is not limited thereto.

Also, the first information may include information about the guide area. For example, the guide area may be displayed at the first viewpoint 7360 as shown in FIG. 54 , but the present invention is not limited thereto.

Also, second information may be displayed at a second time point 7370 .

In this case, the second time point 7370 may be a time point at which the biometric index of the subject is measured, but is not limited thereto.

Also, the second time point 7370 may mean a time point after a predetermined time has elapsed after the measurement of the biometric index for the subject starts, but is not limited thereto.

In addition, the second information may include personal information about the subject. For example, as shown in FIG. 54 , the second information displayed at the second time point 7370 may include personal information such as the main schedule and medication information of the subject, but is not limited thereto.

Also, the second information may include information related to measurement of a biometric index. For example, as shown in FIG. 54 , the second information displayed at the second time point 7370 may include information indicating that the biometric index of the subject is being measured, but is not limited thereto.

Also, third information may be displayed at a third time point 7380 .

At this time, the third time point 7380 may be a time point at which the measurement of the biometric index for the subject is completed, but is not limited thereto, and may be a time point at which at least one measurement is completed in the case of measurement in real time.

In addition, the third information may include a measured biometric index. For example, as shown in FIG. 54 , the third information displayed at the third time point 7380 may include the measured heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

Also, as shown in FIG. 54 , the first information and the second information may be simultaneously displayed at the second time point 7370 , and the first, second and third information may be displayed at the third time point 7380 . Information can be displayed simultaneously.

9.1.6. Various embodiments of a smart mirror device including an opening and closing device

55 is a view for explaining a smart mirror device including an opening and closing device according to an embodiment.

Referring to FIG. 55 , a smart mirror device 7400 according to an embodiment may include an image sensor 7410 , a mirror display 7420 , and an opening/closing device 7430 .

In this case, since the above-described contents may be applied to the image sensor 7410 and the mirror display 7420 , overlapping descriptions will be omitted.

In addition, the open state of the opening and closing device 7430 described below may mean a state in which a window for the image sensor 7410 to acquire an image is secured, and the closed state of the opening and closing device 7430 is the This may mean a state in which a window for the image sensor 7410 to acquire an image is not secured.

In addition, the image sensor 7410 may be used to detect the open state and the closed state. For example, when the illuminance sensed by the image sensor 7410 is less than or equal to a reference value, the closed state may be detected. In addition, when the illuminance is equal to or greater than the reference value, the open state may be sensed, but the present invention is not limited thereto, and the open state and the closed state may be detected using the image sensor 7410 in various ways.

In addition, an external sensor may be used to detect the open state and the closed state. For example, the external sensor detects the degree of openness of the opening/closing device 7430 and, when opened more than a reference value, detects an open state. and the closed state may be detected when the opening is lower than the reference value, but the present invention is not limited thereto, and the open state and the closed state may be detected using the external sensor in various ways.

According to an embodiment, the opening/closing device 7430 may open/close the image sensor 7410 . For example, the opening/closing device 7430 opens or closes the front of the image sensor 7410 accommodated in the internal storage space to adjust the image acquired by the image sensor 7410 or externally the image sensor 7410 . can be prevented from being observed.

More specifically, at a first time point 7440 at which the opening and closing device 7430 is positioned in a closed state, the image sensor 7410 may not be observed from the outside due to the opening and closing device 7430 .

Also, the image sensor 7410 may not acquire an image at the first time point 7440 . For example, the power to the image sensor 7410 may be cut off through a closing operation of the opening/closing device 7430 , but is not limited thereto.

In addition, the image sensor 7410 may be observable from the outside at a second time point 7450 at which the opening/closing device 7430 is positioned in an open state.

Also, at the second time point 7450 , the image sensor 7410 may acquire an image. For example, power may be supplied to the image sensor 7410 through an opening operation of the opening/closing device 7430, but is not limited thereto.

In addition, the surface of the normally open/close device 7430 may be formed of a mirror, and may be formed of the same material as the outer surface of the mirror display 7420 , but is not limited thereto.

In addition, the opening/closing device 7430 may perform a function of protecting the image sensor 7410 from external dust. For example, when the opening/closing device 7430 is in a closed state, it is possible to prevent external dust from coming into contact with the image sensor 7410 , but is not limited thereto.

9.1.7. Various embodiments of a smart mirror device disposed in a shoe closet

A smart mirror device disposed in a shoe cabinet of a house or an entrance of a building may measure a biometric index of a person entering and leaving the house. Usually, a shoe closet at home or a doorway of a building is a place where people wear clothes and there is little privacy problem due to the image sensor, so it may be easy to install an image sensor for measuring a biometric index.

Hereinafter, a smart mirror device disposed at the entrance of a building, which may be represented by a shoe closet, will be described, and for convenience of explanation, a smart mirror device disposed at a shoe closet will be described.

56 is a view for explaining a smart mirror device disposed in a shoe closet according to an embodiment.

Referring to FIG. 56 , a smart mirror device 7460 according to an embodiment may include an image sensor 7470 and a mirror display 7480 .

In this case, since the above-described contents may be applied to the image sensor 7470 and the mirror display 7480 , overlapping descriptions will be omitted.

The image sensor 7470 according to an embodiment may acquire an image of the subject 7490 . For example, the image sensor 7470 may acquire an image of the subject 7490 entering and exiting a shoe closet in which the smart mirror device 7460 is located.

Also, the smart mirror device 7460 may acquire the biometric index based on the image frame obtained from the image sensor 7470 .

In addition, the mirror display 7480 may reflect and provide the image of the subject 7490 . For example, the subject 7490 may observe his/her own appearance through the mirror display.

Also, the mirror display 7480 may display the biometric index of the subject 7490 . For example, the mirror display 7480 may display a biometric index obtained based on an image frame obtained from the image sensor 7470 , but is not limited thereto.

Also, the mirror display 7480 may display biometric information of the subject 7490 . For example, the mirror display 7480 may display biometric information obtained based on the obtained biometric index, but is not limited thereto.

Also, the mirror display 7480 may be formed on at least a part of the smart mirror device 7460 . For example, as shown in FIG. 56 , the mirror display 7480 may be arranged to occupy a predetermined area in the center of the smart mirror device 7460 .

Also, although not shown in FIG. 56 , it is obvious that the mirror display 7480 can display basic information, personal information, and the like.

9.1.7.1. Various embodiments of a smart mirror device using a trigger signal

When the smart mirror device for measuring the biometric index continuously operates, power consumption for acquiring an image and analyzing it may continue. Thus, the trigger signal can be used to save energy and provide a smarter smart mirror device.

57 is a flowchart illustrating a method of operating a smart mirror device according to an exemplary embodiment.

Referring to FIG. 57 , a method 7500 of operating a smart mirror device according to an embodiment includes the steps of acquiring an on-trigger (S7510), acquiring a biometric index of a subject (S7520), and turning off At least one of a step of acquiring a trigger (S7530) and a step of stopping at least one operation of the smart mirror device (S7540) may be included, but is not limited thereto.

More specifically, the method 7500 of operating a smart mirror device according to an embodiment may include acquiring an on-trigger ( S7510 ).

In this case, the on-trigger may be a trigger signal for starting at least one operation of the smart mirror device. For example, the on-trigger may be a trigger signal for an image sensor to acquire an image, but is not limited thereto. It may be a trigger signal for starting at least one operation.

In addition, the on trigger may be provided in various ways.

For example, when the smart mirror device includes a motion detection sensor, the on trigger may be obtained from the motion detection sensor. More specifically, when a predetermined motion is detected by the motion detection sensor, the smart mirror device may acquire a corresponding trigger signal, and this trigger signal may be the ON trigger, but is not limited thereto.

Also, in this case, the motion detection sensor may be disposed inside the smart mirror device, but is not limited thereto and may also be disposed outside the smart mirror device.

Also, for example, when the smart mirror device includes an input device such as a touch panel, the on trigger may be obtained from the input device. More specifically, a predetermined input may be obtained from the input device, and the smart mirror device may obtain a corresponding trigger signal, and such a trigger signal may be the on trigger, but is not limited thereto.

Also, for example, when the smart mirror device includes an image sensor, the on trigger may be obtained from the image sensor. More specifically, the lighting device located in the shoe closet may emit light by detecting a predetermined operation, the image sensor may detect the light emitted from the lighting device, and the smart mirror device may generate a corresponding trigger signal may be obtained, and such a trigger signal may be the on trigger, but is not limited thereto.

Also, for example, when the door is included in the angle of view of the image sensor, the on trigger may be acquired based on a change in the door. More specifically, the image sensor may determine that the door is opened, and the smart mirror device may acquire a trigger signal corresponding thereto, and this trigger signal may be the on trigger, but is not limited thereto. .

Also, the operating method 7500 of the smart mirror device according to an embodiment may include obtaining the biometric index of the subject ( S7520 ).

In this case, the bio-signal for the subject may be obtained based on an image frame obtained through an image sensor, but the detailed description thereof will be omitted as described above.

Also, the operating method 7500 of the smart mirror device according to an embodiment may include acquiring an off-trigger (S7530).

In this case, the off-trigger may be a trigger signal for stopping at least one operation of the smart mirror device. For example, the off trigger may be a trigger signal for stopping an operation of the image sensor acquiring an image, but is not limited thereto, and at least one operation of the smart mirror device, such as turning off the power of the smart mirror device It may be a trigger signal for stopping.

In addition, the off trigger may be provided in various ways.

For example, when the smart mirror device includes a motion detection sensor, the off trigger may be obtained from the motion detection sensor. More specifically, when a predetermined motion is not detected from the motion detection sensor for a certain period of time, the smart mirror device may acquire a corresponding trigger signal, and this trigger signal may be the off trigger, but is limited thereto. doesn't happen

Also, for example, when the smart mirror device includes an input device such as a touch panel, the off trigger may be obtained from the input device. More specifically, a predetermined input may be obtained from the input device, and the smart mirror device may obtain a corresponding trigger signal, and such a trigger signal may be the off trigger, but is not limited thereto.

Also, for example, when the smart mirror device includes an image sensor, the off trigger may be obtained from the image sensor. More specifically, the lighting device located in the shoe closet may be turned off by detecting that there is no predetermined operation for a certain period of time, the image sensor may detect the intensity of the turned off light, and the smart mirror device may generate a corresponding trigger signal. may be obtained, and such a trigger signal may be the off trigger, but is not limited thereto.

Also, for example, the off trigger may be acquired based on the acquisition of the biometric index of the smart mirror device. More specifically, if the measurement target area does not exist within the field of view of the image sensor, the biometric index may not be acquired. , such a trigger signal may be the off trigger, but is not limited thereto.

Also, for example, when the door is included in the angle of view of the image sensor, the off trigger may be acquired based on a change in the door. More specifically, the image sensor may determine that the door is opened, and the smart mirror device may acquire a trigger signal corresponding thereto, and this trigger signal may be the off trigger, but is not limited thereto. .

58 and 59 are diagrams for explaining an operation of a smart mirror device using a trigger signal according to an embodiment.

58 and 59 , a smart mirror device 7550 according to an embodiment may include an image sensor 7560 and a mirror display 7570 .

In this case, since the above-described contents may be applied to the image sensor 7560 and the mirror display 7570 , overlapping descriptions will be omitted.

Referring to FIG. 58 according to an embodiment, when the subject 7580 is located in a predetermined on-trigger generating area, at least one operation of the smart mirror device may be started.

In this case, the predetermined on-trigger generation region may be a detection range of a motion detection sensor when the smart mirror device includes a motion detection sensor, and may be a detection range of a motion detection sensor of a lighting device installed in a shoe closet. not limited

Also, referring to FIG. 59 according to an embodiment, when the smart mirror device 7550 receives a predetermined input from the subject 7580, at least one operation of the smart mirror device may start.

In this case, the predetermined input may be obtained through an input device such as a touch panel, but is not limited thereto.

9.1.7.2. Various embodiments of a smart mirror device for measuring a biometric index according to priority

When a plurality of people exist in the biometric index measurement area of the smart mirror device, the biometric index may be measured according to priority.

60 is a diagram for explaining a method of operating a smart mirror device according to an embodiment.

Referring to FIG. 60 , a method 7600 of operating a smart mirror device according to an embodiment includes acquiring an image including a plurality of people ( S7610 ) and determining a priority for measuring a biometric index ( S7620 ) , may include at least one of obtaining the biometric index according to the priority (S7630) and outputting the obtained biometric index (S7640), but is not limited thereto.

More specifically, the method 7600 of operating a smart mirror device according to an embodiment may include acquiring an image including a plurality of people ( S7610 ), to which the above-described image acquisition content is applied. Therefore, redundant descriptions will be omitted.

In addition, the operating method 7600 of the smart mirror device according to an embodiment may include determining a priority for measuring the biometric index (S7620) and obtaining the biometric index according to the priority (S7630). have.

In this case, the ease of measuring the biometric index may be considered in order to determine the priority. For example, a biometric index of a person occupying a larger area in the acquired image among the plurality of people may be measured, but is not limited thereto.

Also, for example, a biometric index of a person recognized as a person in the acquired image among the plurality of people may be measured, but the present invention is not limited thereto.

Also, for example, a biometric index of a person occupying an area closer to the center in the acquired image among the plurality of people may be measured, but the present invention is not limited thereto.

Also, for example, a biometric index of a person located closer to the image sensor in the acquired image among the plurality of people may be measured, but the present invention is not limited thereto.

In addition, pre-stored information may be utilized to determine the priority. For example, a biometric index of a person previously stored as a subject among the plurality of persons may be measured, but the present invention is not limited thereto.

In addition, for example, a biometric index may be obtained according to a preset priority, and more specifically, when the priority for a child among family members is set high, when the child and the father are located in the biometric index measurement area at the same time, the child's A biometric index may be measured, but is not limited thereto.

In addition, biometric index measurement order information may be utilized to determine the priority. For example, when the image of the first subject is first obtained and the image of the second subject is obtained later, the biometric index of the first subject may be measured, but the present invention is not limited thereto.

Also, the operating method 7600 of the smart mirror device according to an embodiment may include outputting the obtained biometric index (S7640).

In this case, the biometric index may be a biometric index for a person determined according to the priority.

In addition, while the biometric index is output, an indicator for a person who has been measured may also be output. For example, a guide area for a person who is determined according to the priority and is a target of biometric index measurement may be output, but is not limited thereto.

61 is a diagram for explaining an operation of a smart mirror device according to an embodiment.

Referring to FIG. 61 , a smart mirror device 7650 according to an embodiment may include an image sensor 7660 and a mirror display 7670 .

In this case, since the above-described contents may be applied to the image sensor 7660 and the mirror display 7670 , overlapping descriptions will be omitted.

According to an exemplary embodiment, referring to FIG. 61 , a first person 7680 and a second person 7690 may be located within a measurement range of the image sensor 7660 .

Also, the first person 7680 may be the target of measuring the biometric index of the smart mirror device 7650 .

For example, as shown in FIG. 61 , when the first person 7680 is located closer to the image sensor 7660 than the second person 7690 , the first person 7680 is the target for measuring the biometric index. may be, but is not limited thereto.

Also, for example, when the area occupied by the image of the first person 7680 in the image acquired by the image sensor 7660 is larger than the area occupied by the image of the second person 7690 , the first person 7680 may be a measurement target of the biometric index, but is not limited thereto.

Also, for example, when only the first person 7680 is recognized as a person in the smart mirror device 7650 , the first person 7680 may be the target of measuring the biometric index, but is not limited thereto.

Also, for example, when the area occupied by the image of the first person 7680 in the image acquired by the image sensor 7660 is closer to the center than the area occupied by the image of the second person 7690 , the second One person 7680 may be the target of measuring the biometric index, but is not limited thereto.

Also, for example, if the priority for the first person 7680 is stored higher than the priority for the second person 7690, the first person 7680 may be the target for measuring the biometric index. , but is not limited thereto.

Also, for example, when the first person 7680 is stored as a subject to be measured, the first person 7680 may be a target for measuring the biometric index, but is not limited thereto.

Also, when the biometric index for the first person 7680 is measured, the biometric index for the first person 7680 may be output. For example, as shown in FIG. 61 , the heart rate, oxygen saturation, and blood pressure of the first person 7680 may be output, but the present invention is not limited thereto.

Also, when the biometric index for the first person 7680 is measured, an indicator for the first person 7680 may be output. For example, as shown in FIG. 61 , a guide area for the first person 7680 may be output, but the present invention is not limited thereto.

9.1.7.3. Various embodiments of a smart mirror device for measuring the bio-indices of a plurality of subjects

When a plurality of people exist in the biometric index measurement area of the smart mirror device, the biometric index may be measured according to priority.

62 is a view for explaining a method of operating a smart mirror device according to an embodiment.

Referring to FIG. 62 , the operating method 7700 of the smart mirror device according to an embodiment includes the steps of obtaining an image including a plurality of subjects ( S7710 ), obtaining biometric indices for the plurality of subjects ( S7710 ) S7720) and outputting the biometric index for the plurality of subjects (S7730) may include at least one step, but is not limited thereto.

More specifically, the operating method 7700 of the smart mirror device according to an embodiment may include acquiring an image including a plurality of subjects (S7610), and the content of acquiring the image described above is Since it can be applied, overlapping descriptions will be omitted.

In addition, the operating method 7700 of the smart mirror device according to an embodiment may include a step (S7720) of obtaining biometric indices for a plurality of subjects. Duplicate descriptions will be omitted.

According to an embodiment, regions for each of the plurality of subjects may be set in order to obtain the biometric index for the plurality of subjects. For example, when the first subject and the second subject are located within the biometric index measurement area, the area for the first subject and the area for the second subject may be set.

Also, a plurality of biometric indices for a plurality of subjects may be obtained. For example, heart rate, oxygen saturation, and blood pressure for a plurality of subjects may be respectively obtained, but the present invention is not limited thereto.

Also, the types of the plurality of biometric indexes obtained for each of the plurality of subjects may be different from each other. For example, the heart rate of the first subject may be obtained and the oxygen saturation and blood pressure of the second subject may be obtained, but the present invention is not limited thereto.

In addition, a detailed area for each of the plurality of subjects may be set in order to obtain a plurality of biometric indices for the plurality of subjects. For example, a region for the first subject is set, and for the first subject, a first sub-region for measuring heart rate, a second sub-region for measuring oxygen saturation, and a third sub-region for measuring blood pressure A detailed area may be set, an area for a second subject is set, and a fourth detailed area for measuring heart rate, a fifth detailed area for measuring oxygen saturation, and blood pressure are measured for the second subject A sixth detailed area may be set, but is not limited thereto.

Also, the first to sixth detailed regions may be different from each other, and may overlap at least partially.

Also, the operating method 7700 of the smart mirror device according to an embodiment may include outputting biometric indices for a plurality of subjects ( S7730 ).

In this case, the biometric index may be at least one or more biometric indexes for each of the plurality of subjects.

In addition, while the biometric index is output, indicators for a plurality of subjects may be output together. For example, when the biometric indexes of the first and second subjects are obtained, guide regions for the first and second subjects may be output, but the present invention is not limited thereto.

In addition, while outputting the biometric index, an indicator indicating which subject the output biometric index belongs to may be output together. For example, when the biometric index for the first subject is output, an indicator for identifying the first subject may be output together, and when the biometric index for the second subject is output, the second subject An indicator that can be identified may be output together.

63 is a diagram for explaining an operation of a smart mirror device according to an embodiment.

Referring to FIG. 63 , a smart mirror device 7750 according to an embodiment may include an image sensor 7760 and a mirror display 7770 .

In this case, since the above-described contents may be applied to the image sensor 7760 and the mirror display 7770 , overlapping descriptions will be omitted.

According to an exemplary embodiment, referring to FIG. 63 , a first subject 7780 and a second subject 7790 may be located within a measurement range of the image sensor 7760 .

Also, a region for each of the plurality of subjects may be set in an image frame obtained to obtain the biometric index for the first subject 7780 and the second subject 7790 .

In addition, biometric indexes for the first subject 7780 and the second subject 7790 may be obtained. For example, heart rates of the first subject 7780 and the second subject 7790 may be obtained, but the present invention is not limited thereto.

In addition, a plurality of biometric indices for each of the first subject 7780 and the second subject 7790 may be obtained. For example, as shown in FIG. 63 , the heart rate, oxygen saturation, and blood pressure of the first subject 7780 may be obtained, and the heart rate, oxygen saturation and blood pressure of the second subject 7790 may be obtained. may be obtained, but is not limited thereto.

Also, for example, although not shown in FIG. 63 , the heart rate for the first subject 7780 may be obtained, and oxygen saturation and blood pressure for the second subject 7790 may be obtained. However, the present invention is not limited thereto.

9.1.7.4. Various embodiments of a smart mirror device that outputs different information depending on the situation

When the smart mirror device is disposed at the entrance of a building, such as a shoe cabinet, information required for a user may be different from entering the building and leaving the building.

Accordingly, the smart mirror device may output different information to provide different information to the user according to circumstances.

In addition, the smart mirror device may output different biometric indices to provide different biometrics to the user according to circumstances.

64 is a diagram for explaining a method of operating a smart mirror device according to an embodiment.

Referring to FIG. 64 , a method 7800 of operating a smart mirror device according to an embodiment includes the steps of determining an information provision situation ( S7810 ), outputting first information according to the first situation ( S7820 ), and the At least one step of outputting the second information according to the second situation (S7830) may be included, but is not limited thereto.

More specifically, the operating method 7800 of the smart mirror device according to an embodiment may include determining an information provision situation (S7810).

In this case, the information provision situation may mean the situation of the smart mirror device user. For example, the information providing situation may include, but is not limited to, a situation in which the smart mirror device user enters a building or a situation in which the user leaves the building.

Also, the information provision situation may refer to a situation determined to simply provide information. For example, the information provision situation may include the first situation or the second situation, but is not limited thereto.

Also, the information provision situation may mean a situation related to whether or not a biometric index is measured. For example, the information provision situation may include a situation in which it is not necessary to measure the biometric index or a situation in which the biometric index measurement is required, but is not limited thereto.

In addition, the information provision situation may mean a situation related to time. For example, the information provision situation may include a morning situation or an afternoon situation, but is not limited thereto.

Also, the information provision situation may mean a situation according to an external environment. For example, the information provision situation may include a situation according to an external environment such as an external infectious disease situation, but is not limited thereto.

In addition, the smart mirror device may include an additional sensor to determine the information provision situation. For example, a motion detection sensor may be additionally included to determine the user's entry/exit situation, and when the user's motion is first detected by the first motion detection sensor, it is determined as the first situation and the second motion detection sensor When the user's motion is first detected, the second situation may be determined, but the present invention is not limited thereto.

Also, an image sensor included in the smart mirror device may be used to determine the information provision situation. For example, an image frame acquired through the image sensor may be used to determine the user's entry/exit situation, and more specifically, an area corresponding to the user in a plurality of image frames acquired in time series. The first situation may be determined when the first side approaches the central portion, and the second situation may be determined when the second side approaches the central portion, but is not limited thereto.

Also, an image sensor included in the smart mirror device may be used to determine the information provision situation. For example, an image frame obtained through the image sensor may be used to determine whether a biometric index measurement and information provision are necessary, and more specifically, in a plurality of image frames obtained temporally, the user When the movement speed of the applicable region exceeds the reference value, the first situation may be determined, and when the movement speed is less than the reference value, the second situation may be determined, but is not limited thereto.

In addition, time information may be used to determine the information provision situation. For example, if it is before the reference time, it may be determined as the first situation, and if it is after the reference time, it may be determined as the second situation, but is not limited thereto.

Also, recognition information may be used to determine the information provision situation. For example, face recognition information about a person may be used to determine whether biometric index measurement and information provision are necessary, and more specifically, in the case of a registered user as a result of face recognition, it may be determined as the first situation, and registration In the case of a user who has not been determined, the second situation may be determined, but the present invention is not limited thereto.

In addition, the movement direction information of the subject may be used to determine the information provision situation. For example, the movement direction information of the subject detected by sensing or using an external sensor may be used to determine the information provision situation, and more specifically, when the subject moves in the first direction, the first The first situation may be determined, and when the subject moves in the second direction, the second situation may be determined, but is not limited thereto.

In addition, the operating method 7800 of the smart mirror device according to an embodiment includes outputting first information according to a first situation (S7820) and outputting second information according to a second situation (S7830) can, but is not limited thereto.

In this case, when the first situation includes going out of the building and the second situation includes entering the building, the first information and the second information may be different from each other.

For example, the first information may include, but is not limited to, external weather information, today's schedule information, item information according to external weather, and the like, and may be information according to the first situation.

In addition, the second information may include, but is not limited to, internal temperature information, internal humidity information, security information, internal air quality information, and the like, and may be information according to the second situation.

In addition, biometric index information may be included in at least one of the first information and the second information, but is not limited thereto.

Also, the biometric index included in the first information and the biometric index included in the second information may be different from each other. For example, heart rate may be output in a situation where the biometric index of the first subject is provided, and blood pressure may be output in a situation where the biometric index of the second subject is provided, but the present invention is not limited thereto.

65 and 66 are diagrams for explaining an operation of a smart mirror according to an embodiment.

65 and 66 , a smart mirror device 7850 according to an embodiment may include an image sensor 7860 and a mirror display 7870 .

In this case, since the above-described contents may be applied to the image sensor 7860 and the mirror display 7870 , overlapping descriptions will be omitted.

According to an embodiment, FIG. 65 is a diagram illustrating an operation of the smart mirror device 7850 in a first situation, and FIG. 66 may be a diagram illustrating an operation of the smart mirror device 7850 in a second situation.

In addition, the information output in the first situation and the second situation may be different from each other, and since the above-described contents may be applied thereto, the overlapping description will be omitted.

In addition, the external sensor 7861 may be used to determine the first situation and the second situation, but the present invention is not limited thereto, and the above-described examples for determining the first situation and the second situation may be applied.

Also, although not shown in FIGS. 65 and 66 , information output in the first situation and the second situation may at least partially overlap each other. For example, both the biometric index may be included in the information output in the first situation and the second situation, but is not limited thereto.

9.2. Various embodiments of a biometric index measuring device disposed in a vehicle

The above-described biometric index acquisition method and drowsiness detection method may be used in various applications. For example, the method for obtaining the biometric index or the method for detecting drowsiness may be used in a biometric index measuring device disposed in a vehicle.

In addition, the biometric index measuring apparatus may perform an operation corresponding to the biometric index or biometric information obtained according to the method for obtaining the biometric index.

Also, the biometric index measuring apparatus may perform an operation corresponding to the drowsiness information obtained according to the drowsiness detection method.

In addition, although the operation of the biometric index measuring device will be described below, the described content may be performed by other processors such as ECUs mounted on a vehicle, and may also be performed by a processor included in a server.

Also, the device described as the biometric index measuring device may mean an image acquisition device, and in this case, an operation according to the description may be performed by another processor such as an ECU or a processor included in a server.

67 is a view for explaining an apparatus for measuring a biometric index according to an embodiment.

Referring to FIG. 67 , the biometric index measuring apparatus 8010 according to an exemplary embodiment may be disposed in the vehicle 8000 , and may measure the biometric index of a occupant 8020 riding in the vehicle 8000 .

In this case, the biometric index measuring device 8010 may be disposed in various positions of the vehicle 8000 .

For example, the biometric index measuring device 8010 may be disposed on an overhead console, a sun visor, a rearview mirror, a dashboard, an instrument panel, a steering wheel, a Santa facia, a console box, etc. of the vehicle 8000, but this not limited

In addition, the occupant 8020 may be a driver, but is not limited thereto, and may be a person riding in the vehicle, such as a passenger.

Also, when the occupant 8020 is a plurality of people, the biometric index measuring apparatus 8010 may measure the biometric index of the plurality of people.

In addition, the biometric index measuring apparatus 8010 may include an image sensor to obtain a biometric index based on an image frame obtained from the image sensor, but is not limited thereto, and a contact sensor may be used.

In addition, when the biometric index measuring device 8010 includes an image sensor, the image sensor may be provided as a visible light camera for obtaining a visible light image, an infrared camera for obtaining an infrared image, etc. However, the present invention is not limited thereto, and a hybrid type camera for acquiring a visible light image and an infrared image may be provided.

Also, the biometric index measuring apparatus 8010 may operate by changing a mode according to the amount of external light. For example, the first mode may be operated in the daytime when external light is abundant, and the second mode may be operated in the night when external light is low, but is not limited thereto.

Also, for example, the biometric index measuring device 8010 may operate in the first mode for obtaining the biometric index based on the RGB image when the external light is equal to or greater than the reference value, and when the external light is equal to or less than the reference value, the IR image It may operate in the second mode for obtaining the biometric index based on , but is not limited thereto.

68 is a diagram for describing an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 68 , the apparatus for measuring a biometric index according to an embodiment may be disposed at various locations of a vehicle.

For example, as shown in FIG. 68 , the first biometric index measuring device 8011 may be disposed on the dashboard of the driver’s seat, and the second biometrics measuring device 8012 may be disposed on the dashboard of the passenger’s seat. Also, the third biometric index measuring device 8013 may be disposed in the rearview mirror, but is not limited thereto.

Also, at least one bio-index measuring device among the first to third bio-index measuring devices 8011 , 8012 , and 8013 may be disposed inside the vehicle.

Also, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may acquire the driver's biometric index.

For example, at least one of the first to third biometric index measuring devices 8011, 8012, and 8013 may measure the driver's heart rate, blood pressure, oxygen saturation, body temperature, etc. However, the present invention is not limited thereto.

Also, at least one of the first to third biometric index measuring apparatuses 8011 , 8012 , and 8013 may perform an operation according to the obtained driver's biometric index.

For example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 performs an operation such as guiding a nearby hospital when the obtained driver's biometric index is abnormal. can be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may transmit data to another mobile device when the obtained driver's biometric index is abnormal. Information may be transmitted, but is not limited thereto.

In addition, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may be configured by hardware such as sound or vibration when the obtained driver's biometric index is abnormal. An alarm may be activated, but the present invention is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may set the vehicle to the autonomous driving mode when the obtained driver's biometric index is abnormal. An operation for changing may be performed, but is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may acquire the driver's biometric information.

For example, at least one of the first to third biometric index measuring devices 8011, 8012, and 8013 may acquire the driver's biometric information, such as driver's drowsiness information and condition information, However, the present invention is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may perform an operation according to the obtained driver's biometric information.

For example, at least one of the first to third biometric index measuring devices 8011, 8012, and 8013 operates a hardware alarm such as sound vibration when the obtained driver's drowsiness information is abnormal. can be done, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 is configured to guide a nearby drowsiness shelter when the obtained driver's drowsiness information is abnormal. and the like may be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may transmit information about the obtained driver's drowsiness information to another mobile device when there is an abnormality. Information may be transmitted, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 sets the vehicle to the autonomous driving mode when the obtained driver's drowsiness information is abnormal. An operation for changing may be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 performs an operation such as selecting appropriate music according to the acquired condition information of the driver. can be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 performs an operation such as guiding appropriate content according to the obtained driver's condition information. can be performed, but is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may acquire the passenger's biometric index.

For example, at least one of the first to third biometric index measuring devices 8011, 8012, and 8013 may measure the passenger's heart rate, blood pressure, oxygen saturation, body temperature, etc. However, the present invention is not limited thereto.

In addition, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may perform an operation according to the obtained biometric index of the passenger.

For example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 performs an operation such as giving an alarm to the driver when the acquired biometric index of the passenger is abnormal can be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011, 8012, and 8013 may guide a nearby hospital when the obtained biometric index of the passenger is abnormal, etc. may be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may transmit data to another mobile device when the obtained biometric index of the passenger is abnormal. Information may be transmitted, but is not limited thereto.

In addition, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may set the vehicle to the autonomous driving mode when the obtained biometric index of the passenger is abnormal. An operation for changing may be performed, but is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may acquire the passenger's biometric information.

For example, at least one of the first to third biometric index measuring devices 8011, 8012, and 8013 may acquire the passenger's biometric information such as drowsiness information and condition information of the passenger, However, the present invention is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may perform an operation according to the obtained biometric information of the passenger.

For example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may notify the driver of the obtained biometric information when there is an abnormality. and the like may be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8011 , 8012 , and 8013 may transmit information about the obtained biometric information to another mobile device when there is an abnormality. Information may be transmitted, but is not limited thereto.

69 is a diagram for explaining a biometric index measuring device disposed in an autonomous vehicle according to an exemplary embodiment.

Referring to FIG. 69 , the apparatus for measuring a biometric index according to an exemplary embodiment may be disposed in various locations of a vehicle.

For example, as shown in FIG. 69, the first bio-index measuring device 8014 may be disposed on one side of the table, and the second bio-index measuring device 8015 may be disposed on the other side of the table, and the second 3 The biometric index measuring device 8016 may be disposed on the ceiling of the autonomous vehicle, but is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8014 , 8015 , and 8016 may be disposed inside the vehicle.

Also, at least one of the first to third biometric index measuring devices 8014 , 8015 , and 8016 may acquire the passenger's biometric index.

For example, at least one bio-index measuring device among the first to third bio-index measuring devices 8014, 8015, 8016 may measure a bio-indices such as heart rate, blood pressure, oxygen saturation, and body temperature of the passenger, However, the present invention is not limited thereto.

In addition, at least one of the first to third biometric index measuring devices 8014 , 8015 , and 8016 may perform an operation according to the obtained biometric index of the passenger.

For example, when at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 has an abnormality in the obtained biometric index of the passenger, an operation such as guiding a nearby hospital can be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 may drive to a nearby hospital when the obtained biometric index of the passenger is abnormal. may be performed, but is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 may transmit information about the obtained biometric index to another mobile device when there is an abnormality in the obtained biometric index. Information may be transmitted, but is not limited thereto.

In addition, for example, at least one of the first to third biometric index measuring devices 8014 , 8015 , and 8016 may include a sound, vibration, etc. hardware device when the obtained biometric index of the passenger is abnormal. An alarm may be activated, but the present invention is not limited thereto.

Also, for example, at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 may display information on the obtained biometric index when the passenger has an abnormality. However, it is not limited thereto.

Also, at least one of the first to third biometric index measuring devices 8014 , 8015 , and 8016 may perform an operation according to a result of recognition of the passenger.

For example, at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 may recognize the face of the passenger, and when the face is recognized, Information may be displayed, but the present invention is not limited thereto.

In addition, for example, at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 may provide information to the recognized passenger only when the face of the passenger is recognized and the biometric index is obtained. information may be displayed, but the present invention is not limited thereto.

Also, for example, at least one biometric index measuring device among the first to third biometric index measuring devices 8014 , 8015 , and 8016 may acquire the passenger's biometric information.

For example, at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 may acquire biometric information of the passenger, such as drowsiness information and condition information of the passenger, not limited

Also, at least one of the first to third biometric index measuring devices 8014 , 8015 , and 8016 may perform an operation according to the obtained biometric information of the passenger.

For example, when there is an abnormality in the biometric information of the passenger obtained by at least one of the first to third biometric index measuring devices 8014, 8015, and 8016, a hardware alarm such as sound or vibration is operated can be done, but is not limited thereto.

Also, for example, when at least one of the first to third biometric index measuring devices 8014, 8015, and 8016 has an abnormality in the obtained biometric information of the passenger, it transmits the information to another mobile device. Information may be transmitted, but is not limited thereto.

In addition, although not shown in FIGS. 68 and 69 , the biometric index measuring apparatus may be disposed in an ambulance or the like, and may be used to measure a patient's biometric index and obtain biometric information.

9.2.1. Various embodiments of a biometric index measuring device performing an operation according to drowsiness information

70 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 70 , an operating method 8100 of an apparatus for measuring a biometric index according to an embodiment may include obtaining a biometric index ( S8110 ) and determining a drowsiness step ( S8120 ), and the determined drowsiness You can output an alarm according to the steps. For example, as shown in FIG. 70 , the first alarm S8130 , the second alarm S8140 , and the third alarm S8150 may be output, but the present invention is not limited thereto.

In this case, since the above-described method for obtaining the biometric index may be applied to the step of obtaining the biometric index ( S8110 ), the overlapping description will be omitted.

In addition, the above-described method for detecting drowsiness may be applied to the step of determining the drowsiness stage ( S8120 ), and thus the overlapping description will be omitted.

Also, the first alarm S8130 may be an alarm output during the weakest drowsiness stage.

For example, the first alarm S8130 may include a visual alarm, but is not limited thereto.

Also, the second alarm S8140 may be an alarm output in an intermediate step.

For example, the second alarm S8140 may include an audible alarm having a weak intensity or a long repetition period, but is not limited thereto.

Also, the third alarm S8150 may be an alarm output at the highest level of drowsiness.

For example, the third alarm S8150 may include an audible alarm having a strong intensity or a short repetition period, but is not limited thereto.

However, the first to third alarms S8130 , S8140 , and S8150 are not limited to the above-described examples, and it is evident that all examples of outputting different alarms according to the drowsiness stage may be applied.

Also, the operating method 8100 of the apparatus for measuring a biometric index according to an embodiment may include transmitting drowsiness information ( S8160 ).

Also, the step of transmitting the drowsiness information ( S8160 ) may be performed only in at least some stages of drowsiness. For example, in the case of the first drowsiness stage, the step of transmitting the drowsiness information (S8160) may not be performed, and in the case of the second drowsiness stage and the third drowsiness stage, the step of transmitting the drowsiness information (S8160) is performed may be, but is not limited thereto.

In addition, the step of transmitting the drowsiness information ( S8160 ) may include transmitting the drowsiness information to the manager. For example, information on the driver's sleepiness stage may be transmitted to a manager who dispatches the vehicle to be reflected in the dispatch of the vehicle.

Also, the step of transmitting the drowsiness information ( S8160 ) may include transmitting the drowsiness information to the driver's mobile terminal. For example, by transmitting drowsiness information and information on a corresponding driving section to the driver's mobile terminal, the driver may be able to pay attention to the next driving.

Also, the step of transmitting the drowsiness information ( S8160 ) may include transmitting the drowsiness information to the road manager. For example, information on a driver's drowsiness stage and information on a driving section may be transmitted to be used for producing a drowsiness map.

9.2.2. Various embodiments of a driving scheduling method using biometric index and biometric information

71 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 71 , the operating method 8200 of the apparatus for measuring a biometric index according to an embodiment includes the steps of obtaining the biometric index (S8210), obtaining the biometric information (S8220), and calculating the driving scheduling index (S8210) S8230), but is not limited thereto.

In this case, since the above-described method for obtaining the biometric index may be applied to the step of obtaining the biometric index ( S8210 ), the overlapping description will be omitted.

In addition, since the above-described method for obtaining biometric information may be applied to the step of obtaining the biometric information ( S8220 ), the overlapping description will be omitted.

Also, the driving scheduling index may be an index for assisting driving scheduling.

In this case, the driving scheduling index may be obtained based on the biometric index and biometric information. For example, the driving scheduling index may be obtained based on heart rate information and drowsiness information of the driver, but is not limited thereto.

Also, the driving scheduling index may be obtained based on driving information of the vehicle. For example, the driving scheduling index may be obtained based on a driving distance, driving time, number of trips, etc. of the vehicle, but is not limited thereto.

Also, the driving scheduling index may be obtained based on the driver's biometric index, biometric information, and vehicle driving information. For example, the driving scheduling index may be acquired based on driver's drowsiness information, the driving distance of the vehicle, the number of driving, etc., but is not limited thereto.

72 is a view for explaining a method of operating a driving scheduling assistance device using a biometric index measuring device according to an exemplary embodiment.

Referring to FIG. 72 , the driving scheduling assistance device may obtain information from a first biometric index measuring device disposed in a first vehicle and a second biometric index measuring device disposed in a second vehicle.

More specifically, the first biometric index measuring apparatus may acquire the biometric index and biometric information about the first driver who is riding in the first vehicle. For example, the first biometric index measuring apparatus may acquire heart rate and drowsiness information about the first driver, but is not limited thereto.

In addition, the second biometric index measuring apparatus may obtain the biometric index and biometric information about the second driver who is riding in the second vehicle. For example, the second biometric index measuring apparatus may acquire heart rate and drowsiness information about the second driver, but is not limited thereto.

Also, as shown in FIG. 72 , the driving scheduling assistance device may acquire the biometric index and biometric information about the first driver obtained from the first biometric index measuring device, and the second biometric index measuring device The biometric index and biometric information about the second driver obtained from .

Also, as shown in FIG. 72 , the driving scheduling assistance device may obtain first driving information for a first driving of the first vehicle and second driving information for a second driving of the second vehicle.

In this case, the first and second driving information may include, but is not limited to, a driving distance, the number of trips, and a running time.

In addition, the driving scheduling assisting device includes a biometric index and biometric information for the first driver, a biometric index and biometric information for the second driver, first driving information for the first vehicle, and first driving information for the second vehicle. 2 A driving scheduling index may be calculated based on the driving information.

For example, a first driving scheduling index may be calculated based on the first drowsiness information on the first driver and the first driving information on the first vehicle, and the second drowsiness information on the second driver and A second driving scheduling index may be calculated based on the second driving information for the second vehicle, but is not limited thereto.

Also, for example, when the first driving distance included in the first driving information and the second driving distance included in the second driving information are the same, drowsiness according to drowsiness information among the first driver and the second driver A driving scheduling index may be calculated so that a driver having a lower index may perform the third operation.

Also, for example, the first driving scheduling index may be calculated by adding different weights to the first drowsiness information and the first driving information for calculating the first driving scheduling index.

More specifically, the first driving scheduling index may be calculated by adding more weights to the first driving information, but is not limited thereto.

Also, for example, the second driving scheduling index may be calculated by adding different weights to the second drowsiness information and the second driving information for calculating the second driving scheduling index.

More specifically, the second driving scheduling index may be calculated by adding more weights to the second driving information, but is not limited thereto.

Also, the driving scheduling assistance apparatus may perform driving scheduling for the third driving using the calculated driving scheduling index, but is not limited thereto.

Also, the driving scheduling assistance device may display the calculated driving scheduling index, but is not limited thereto.

9.2.3. Various embodiments of a biometric index measuring device used to unlock a vehicle

73 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 73 , the operating method 8300 of the apparatus for measuring a biometric index according to an embodiment includes the steps of recognizing the face of the subject (S8310), obtaining the biometric of the subject (S8320), and the operation of the vehicle It may include at least a part of the step (S8330) of releasing at least a part of the lock, but is not limited thereto.

In this case, in the step of recognizing the subject's face ( S8310 ), not only the above-described content but also a normal face recognition method may be applied, and thus the overlapping description will be omitted.

In addition, since the step (S8320) of obtaining the biometric index of the subject may be applied to the above-described method for obtaining the biometric index, the overlapping description will be omitted.

In addition, the step of unlocking at least a part of the operation of the vehicle ( S8330 ) may be performed when the face of the subject is recognized and a biometric index is obtained.

For example, when the face of the subject is recognized but the biometric index is not obtained, the step of unlocking at least a part of the operation of the vehicle ( S8330 ) may not be performed, but is not limited thereto.

In addition, in this case, it is possible to prevent unlocking by stealing photos, etc., and to further strengthen the security of the vehicle.

In addition, the step of unlocking at least a part of the operation of the vehicle ( S8330 ) may be performed when the recognized face of the subject matches the ID stored in the server and the biometric index is obtained.

For example, when the face of the subject is recognized and the biometric index is obtained, but the recognized face of the subject does not match the ID stored in the server, unlocking at least a part of the operation of the vehicle (S8330) may not be performed, but is not limited thereto.

In addition, the step of unlocking at least a part of the operation of the vehicle ( S8330 ) may be performed when the face of the subject is recognized and the biometric index matches the ID stored in the server.

For example, when the face of the subject is recognized and the biometric index is obtained, but the obtained biometric index of the subject does not match the ID stored in the server, unlocking at least a part of the operation of the vehicle (S8330) ) may not be performed, but is not limited thereto.

In addition, the locking of at least a part of the operation of the vehicle may be a lock for starting the vehicle, but is not limited thereto.

9.2.4. Various embodiments of an operating method of a driving index calculating device using a biometric index measuring device

74 is a flowchart for explaining a method of operating an apparatus for calculating a driving index according to an exemplary embodiment.

Referring to FIG. 74 , the operating method 8350 of the driving index calculation apparatus according to an embodiment includes the steps of obtaining driving information ( S8360 ), obtaining drowsiness information for the subject ( S8370 ), and for the subject It may include at least a part of calculating the driving index ( S8380 ), but is not limited thereto.

In this case, the driving index may be a comprehensive evaluation index for the driver's driving, and may be a numerical value calculated by taking various information into consideration, but is not limited thereto.

In addition, in this case, the driving information may include, but is not limited to, driving distance information, driving frequency information, driving time information, driving section information, road information, etc. for the vehicle being driven.

In addition, since the above-described method for detecting drowsiness may be applied to the step of obtaining drowsiness information on the subject ( S8370 ), a redundant description will be omitted.

In addition, calculating the driving index for the subject ( S8380 ) may include calculating the driving index based on the driving information and the drowsiness information.

For example, the driving index may be calculated by comprehensively considering the number of times drowsiness occurred during the driving time, the degree of drowsiness, a response thereto, and the like, but is not limited thereto.

Also, a reward may be awarded to the driver based on the calculated driving index. For example, a compensation may be awarded to the driver based on a driving index calculated based on driving time information of the driver who has driven for a long time and information on drowsiness in which no drowsiness has occurred, but is not limited thereto.

In addition, in addition to the above-described examples, a driving index for the subject may be calculated according to various methods based on driving time and drowsiness information.

9.3. Various embodiments of a biometric index measuring device disposed in an infant monitoring device

Monitoring infants and young children by using image sensors such as cameras is becoming essential for the health and safety of infants and young children.

In addition, monitoring the health and safety of infants and young children by measuring the biological index of infants and young children is also becoming essential.

However, attaching a separate sensor or disposing a separate contact sensor to monitor the biometric index of infants and young children may cause inconvenience to infants and young children, limiting its role as a comfortable and comfortable shelter.

Therefore, monitoring infants and toddlers using an image sensor such as a camera and monitoring the biometric index of infants and young children simultaneously using an image sensor such as a camera will protect the health and safety of infants and toddlers and provide a comfortable and comfortable shelter for infants and toddlers. can

75 is a diagram for describing an apparatus for monitoring infants and toddlers according to an embodiment.

Referring to FIG. 75 , the infant monitoring apparatus 8400 according to an embodiment may acquire the biometric index of the infant 8410 in a non-contact manner.

More specifically, the infant monitoring apparatus 8400 may include an image sensor, and may obtain a biometric index based on an image frame obtained from the image sensor, but is not limited thereto.

In addition, since the above-mentioned contents may be applied to the infant monitoring apparatus 8400 acquiring the biometric index based on the image frame, the overlapping description will be omitted.

Also, the infant monitoring apparatus 8400 may store or transmit an image acquired through the image sensor.

For example, the infant monitoring apparatus 8400 may acquire an image including at least a portion of the infant 8410 to be measured, and may store the acquired image or transmit it to the user's mobile device.

Also, the infant monitoring apparatus 8400 may detect the operation of the infant 8410 . For example, the infant monitoring apparatus 8400 may detect the overturning of the infant 8410, but is not limited thereto.

Also, the infant monitoring apparatus 8400 may detect a sound generated from the infant 8410 . For example, the infant monitoring apparatus 8400 may detect the cry of the infant 8410, but is not limited thereto.

In addition, the infant monitoring apparatus 8400 may generate a sound. For example, when there is an input for a sound from the input device, the infant monitoring device 8400 may output a sound corresponding to the input, and more specifically, when the voice of the parent of the infant 8410 is input, it output, but is not limited thereto.

In addition, the infant monitoring apparatus 8400 may output lighting. For example, when there is an input for lighting from an input device, the infant monitoring apparatus 8400 may output a lighting corresponding to the input, but is not limited thereto.

Also, the infant monitoring apparatus 8400 may control the operation of the cradle in which the infant 8410 is present. For example, when there is an input for the cradle from the input device, the infant monitoring apparatus 8400 may perform an operation of the cradle corresponding to the input, but is not limited thereto.

Also, the infant monitoring apparatus 8400 may output an alarm. For example, when an event occurs to the infant 8410, an audible alarm may be output, but the present invention is not limited thereto, and a visual and tactile alarm may be output, and information may be transmitted to the user's mobile terminal.

In addition, the infant monitoring apparatus 8400 may perform various operations for monitoring the infant 8410 in addition to the above-described examples.

9.3.1. Various embodiments of an infant monitoring device that performs an operation according to the occurrence of an event

76 is a view for explaining the occurrence of an event for infants and young children.

Referring to FIG. 76 , it can be seen that various events may occur for infants and young children.

Referring to (a) of FIG. 76 , it can be seen that an event in which an infant turns over may occur.

In addition, if the infant turns over, the passage for breathing may be blocked, which may pose a great risk to the infant.

Also, referring to (b) of FIG. 76 , it can be seen that an event in which an infant moves out of the monitoring area may occur.

In this case, the infant may be moved by a nurse at a postpartum care center, etc., but may be moved by an uninvited guest.

In addition, although not shown in FIG. 76 , various events that may threaten the health and safety of infants and young children may occur.

Therefore, an infant monitoring device capable of monitoring and tracking such events may be required.

77 is a flowchart for explaining a method of operating an apparatus for monitoring infants and toddlers according to an embodiment.

Referring to Figure 77, the operation method 8500 of the infant monitoring apparatus according to an embodiment includes the steps of obtaining a biometric index (S8510), determining the occurrence of an event (S8520), and performing an operation corresponding to the event It may include at least a part of (S8530), but is not limited thereto.

In this case, since the above-described method for obtaining the biometric index may be applied to the step of obtaining the biometric index ( S8510 ), the overlapping description will be omitted.

According to an embodiment, the step of determining the occurrence of the event ( S8520 ) may be determined based on the obtained biometric index.

For example, when the biometric index is not obtained through the obtaining of the biometric index ( S8510 ), it may be determined that an event has occurred. As a more specific example, if an infant, a target of obtaining a biometric index, is turned over, the biometric index may not be obtained, and in this case, it may be determined that an event has occurred, but is not limited thereto.

Also, for example, when there is an abnormality in the biometric index obtained through the step of obtaining the biometric index ( S8510 ), it may be determined that an event has occurred. As a more specific example, it may be determined that an event occurs when the heart rate of an infant or toddler, which is a target of obtaining a biometric index, is abnormal, but is not limited thereto.

Also, according to an embodiment, the step of determining the occurrence of the event ( S8520 ) may be determined based on a plurality of biometric indices.

For example, if at least one biometric index is not obtained through the obtaining of the biometric index (S8510), it may be determined that the first event has occurred, and if all the biometric indexes are not obtained, it is determined that the second event has occurred can, but is not limited thereto.

In addition, for example, when at least one biometric index obtained through the step (S8510) of obtaining the biometric index is abnormal, it can be determined that the first event occurs, and when all the biometric indexes are abnormal, the second It may be determined that the event occurs, but is not limited thereto.

Also, according to an embodiment, the step of determining the occurrence of the event ( S8520 ) may be determined based on a change in the biometric index.

For example, when an abrupt change is detected in the biometric index obtained through the step of obtaining the biometric index (S8510), it can be determined as an event occurrence. When changing from a sleep state to a wake-up state, the heart rate of the infant may increase, and this case may be determined as an event occurrence, but is not limited thereto.

Also, according to an embodiment, performing the operation corresponding to the event ( S8530 ) may include performing an operation corresponding to the determined event information.

In this case, the operation corresponding to the event may include a hardware alarm such as a visual, auditory, or tactile alarm. For example, the infant monitoring apparatus may output an audible sound alarm when it is determined that an event occurs, but is not limited thereto.

Also, the operation corresponding to the event may include an operation for recording an image. For example, the infant monitoring apparatus may operate to record an image when it is determined that an event occurs, but is not limited thereto.

Also, the operation corresponding to the event may include an operation of transmitting information about the event. For example, the infant monitoring apparatus may transmit information about the event to the user's mobile terminal when it is determined that the event occurs, but is not limited thereto.

Also, the operation corresponding to the event may include an operation of storing information about an event occurrence time. For example, when it is determined that an event occurs, the infant monitoring apparatus may store information about an event occurrence time, but is not limited thereto.

In addition, the operation corresponding to the event may include various operations corresponding to the event occurring in addition to the above-described examples.

78 is a flowchart for explaining a method of operating an apparatus for monitoring infants and toddlers according to an embodiment.

Referring to FIG. 78 , the operation method 8550 of the device for monitoring infants and toddlers according to an embodiment includes the steps of obtaining a biometric index ( S8560 ), determining the occurrence of an event ( S8570 ), and a first operation corresponding to the first event It may include at least a part of performing ( S8580 ) and performing a second operation corresponding to the second event ( S8590 ), but is not limited thereto.

In this case, since the above-described method for obtaining the biometric index may be applied to the step of obtaining the biometric index ( S8560 ), the overlapping description will be omitted.

According to an embodiment, the step of determining the occurrence of the event ( S8570 ) may be determined based on the obtained biometric index.

For example, when the biometric index is not obtained through the obtaining of the biometric index ( S8560 ), it may be determined that an event has occurred. As a more specific example, if an infant, a target of obtaining a biometric index, is turned over, the biometric index may not be obtained, and in this case, it may be determined that an event has occurred, but is not limited thereto.

Also, for example, when there is an abnormality in the biometric index obtained through the step of obtaining the biometric index ( S8560 ), it may be determined that an event has occurred. As a more specific example, it may be determined that an event occurs when the heart rate of an infant or toddler, which is a target of obtaining a biometric index, is abnormal, but is not limited thereto.

Also, according to an embodiment, the step of determining the occurrence of the event ( S8570 ) may be determined based on a plurality of biometric indices.

For example, if at least one biometric index is not obtained through the obtaining of the biometric index (S8560), it may be determined that the first event has occurred, and if all biometric indexes are not obtained, it is determined that the second event has occurred can, but is not limited thereto.

Also, for example, if there is an abnormality in at least one biometric index obtained through the step of obtaining the biometric index ( S8560 ), it may be determined that the first event occurs, and when all the biometric indexes are abnormal, the second It may be determined that the event occurs, but is not limited thereto.

Also, according to an embodiment, the step of determining the occurrence of the event ( S8570 ) may be determined based on a change in the biometric index.

For example, when an abrupt change is detected in the biometric index obtained through the step (S8560) of acquiring the biometric index, it can be determined as an event occurrence. When changing from a sleep state to a wake-up state, the heart rate of the infant may increase, and this case may be determined as an event occurrence, but is not limited thereto.

In addition, according to an embodiment, performing the first operation corresponding to the first event (S8580) and performing the second operation corresponding to the second event (S8590) are operations corresponding to the determined event information. It may include the step of performing

In this case, the first operation and the second operation may include hardware alarms such as visual, audio, and tactile alarms. For example, the infant monitoring apparatus may output an audible sound alarm when it is determined that an event occurs, but is not limited thereto.

Also, the first operation and the second operation may include an operation for recording an image. For example, the infant monitoring apparatus may operate to record an image when it is determined that an event occurs, but is not limited thereto.

Also, the first operation and the second operation may include an operation of transmitting information on an event. For example, the infant monitoring apparatus may transmit information about the event to the user's mobile terminal when it is determined that the event occurs, but is not limited thereto.

Also, the first operation and the second operation may include an operation of storing information about an event occurrence time. For example, when it is determined that an event occurs, the infant monitoring apparatus may store information about an event occurrence time, but is not limited thereto.

Also, the first operation and the second operation may be different from each other, but are not limited thereto, and may be identical to each other, and may perform at least some identical operations to each other.

In addition, the first operation and the second operation may include various operations corresponding to the first event and the second event in addition to the above-described examples.

9.3.2. Various embodiments of the infant monitoring system

79 is a diagram illustrating a mobile application for implementing a system for monitoring infants and toddlers according to an embodiment.

Referring to FIG. 79 , the mobile application 8600 according to an embodiment includes an image display area 8610, a biometric index display area 8620, an input button display area 8630, an image time display area 8640, and an event time. At least a portion of the recorded image display area 8650 may be included, but is not limited thereto.

In this case, the image displayed on the image display area 8610 may be an image or an image obtained from an infant monitoring apparatus, but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 includes at least one biometric index. For example, as shown in FIG. 79 , the biometric index may include, but is not limited to, heart rate, oxygen saturation, blood pressure, and the like.

In addition, the biometric index displayed in the biometric index display area 8620 may be a biometric index obtained from an infant monitoring device, but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 may be a biometric index obtained from at least one sensor, but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 may be a biometric index obtained from different sensors. For example, the heart rate displayed on the biometric index display area 8620 may be a biometric index obtained from the infant monitoring device, and the blood pressure may be a biometric index obtained from an external blood pressure sensor, but is not limited thereto.

Also, the input button displayed on the input button display area 8630 may include at least one input button.

In addition, the input button may include at least some of various input buttons such as an image recording button, a lighting button, a talk button, a cradle shaking button, and an alarm button, but is not limited thereto.

In addition, the image time displayed in the image time display area 8640 may be time information on a recorded image.

In addition, an event occurrence time may be displayed in the image time displayed in the image time display area 8640 . For example, the first event occurrence time and the second event occurrence time may be displayed in the video time as shown in FIG. 79 , but the present invention is not limited thereto.

Also, a thumbnail of the recorded image may be displayed in the image display area 8650 recorded at the event time.

In addition, information on the recorded image may be displayed in the image display area 8650 recorded at the event time. For example, time information of a recorded image may be displayed as shown in FIG. 79 , but the present invention is not limited thereto.

However, the above-described examples and drawings show only one embodiment and are not limited thereto, and applications for implementing the infant monitoring system may be provided in various types of applications.

9.4. Various embodiments of the biometric index measuring device disposed in the reading room

Measuring biometric index and biometric information in a space for personal work and study, such as a reading room, can enable efficient work or study by monitoring individual concentration while continuously monitoring personal health.

Also, such concentration may be obtained based on the heart rate. For example, concentration information may be obtained based on a change, a size, a change pattern, etc. of the heart rate, but is not limited thereto.

80 is a view for explaining an apparatus for measuring a biometric index disposed in a reading room according to an exemplary embodiment.

Referring to FIG. 80 , the biometric index measuring apparatus 8700 according to an embodiment may acquire the biometric index and biometric information of the subject 8710 .

In this case, the bioindices may include, but are not limited to, biometrics such as heart rate, oxygen saturation, blood pressure, and body temperature.

Also, the biometric information may include biometric information such as drowsiness information, condition information, and concentration information, but is not limited thereto.

In addition, since the above-mentioned contents can be applied to the method for obtaining the biometric index and biometric information, overlapping descriptions will be omitted.

81 is a flowchart illustrating a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 81 , the operating method 8720 of the apparatus for measuring a biometric index according to an embodiment performs the steps of obtaining the biometric index (S8730), obtaining the biometric information (S8740), and an operation corresponding to the biometric information It may include at least a part of the step (S8750).

In this case, since the above-described contents may be applied to the step of obtaining the biometric index (S8730) and the step of obtaining the biometric information (S8740), the overlapping description will be omitted.

In addition, performing the operation corresponding to the biometric information ( S8750 ) may include performing various operations corresponding to the various biometric information.

For example, when the biometric information is drowsiness information, the biometric index measuring apparatus may output an alarm according to a drowsiness stage, but is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the biometric index measurement apparatus may perform an operation of transmitting information to the manager according to the drowsiness stage, but is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the apparatus for measuring the biometric index may perform an operation of transmitting information to the user's terminal according to the drowsiness stage, but is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the apparatus for measuring the biometric index may perform an operation of transmitting information to the scheduling apparatus according to the drowsiness stage, but is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the apparatus for measuring the biometric index may perform an operation of outputting a rest recommendation text according to a drowsiness stage, but is not limited thereto.

Also, for example, when the biometric information is the condition information, the apparatus for measuring the biometric index may perform an operation of outputting an alarm according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric index measurement apparatus may perform an operation of transmitting information to an administrator according to a condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the apparatus for measuring the biometric index may perform an operation of transmitting the information to the user's terminal according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the apparatus for measuring the biometric index may perform an operation of transmitting the information to the scheduling device according to the condition, but is not limited thereto.

Also, for example, when the biometric information is the condition information, the apparatus for measuring the biometric index may perform an operation of outputting a rest recommendation phrase according to the condition, but is not limited thereto.

Also, for example, when the biometric information is concentration information, the apparatus for measuring the biometric index may perform an operation of outputting an alarm according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration information, the biometric index measurement apparatus may perform an operation of transmitting information to the manager according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration information, for example, the biometric index measurement apparatus may perform an operation of transmitting information to the user's terminal according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration level information, for example, the biometric index measurement apparatus may transmit information to the scheduling device according to the concentration level, but is not limited thereto.

In addition, the biometric index measuring apparatus may perform an operation corresponding to the biometric information and the biometric index.

In addition, the bio-index measuring apparatus may perform various operations corresponding to bio-information and bio-indices in addition to the above-described examples.

82 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to Figure 82, the operating method 8760 of the apparatus for measuring a biometric index according to an embodiment includes the steps of obtaining a biometric index (S8770), obtaining biometric information (S8780), and calculating a study or work scheduling index It may include at least a part of the step (S8790).

In this case, since the above-described contents may be applied to the step of obtaining the biometric index (S8770) and the step of obtaining the biometric information (S8780), the overlapping description will be omitted.

In addition, calculating the study or work scheduling index (S8790) may be performed based on the obtained biometric index and / or biometric information.

For example, a study or work scheduling index may be calculated based on the obtained heart rate and drowsiness information, but is not limited thereto.

In addition, calculating the study or work scheduling index (S8790) may be performed based on the obtained biometric index, biometric information and / or information about the study.

For example, a study or work scheduling index may be calculated based on the obtained sleepiness information and study time information, but is not limited thereto.

In addition, by using the study or work scheduling index, it can be scheduled so that the study or work is performed in a time period with high efficiency.

In addition, by using the study or work scheduling index, it can be scheduled to take a break in a time section with low efficiency.

Therefore, if the study or work scheduling index is utilized, it may be possible to schedule the study or work to be performed with maximum efficiency within a limited time.

9.5. Various embodiments of a bio-index measuring device used for cognitive rehabilitation treatment

For patients receiving cognitive rehabilitation, such as dementia patients, the therapeutic effect can be maximized only when intensive treatment is performed during cognitive rehabilitation.

However, the supply and demand of rehabilitation therapists for monitoring individual patient concentration may not meet the demand, so it may be difficult to improve the individual patient's concentration level to provide high-quality treatment.

Therefore, when the biometric index and biometric information are acquired by using the biometric index measuring device to compensate for this, the concentration of each patient can be monitored and high quality treatment can be performed using this.

83 is a view for explaining an apparatus for measuring a biometric index used for cognitive rehabilitation treatment according to an embodiment.

Referring to FIG. 83 , the apparatus for measuring a biometric index according to an exemplary embodiment includes a first biometric index measuring device 8800 , a second biometric index measuring device 8801 , a third biometric index measuring device 8802 , and a fourth biometric index measuring device At least one of the measuring device 8803 and the fifth biometric index measuring device 8804 may be included.

In addition, at least one of the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from at least one of the first to fourth patients 8811, 8812, 8813, 8814. Biometrics can be measured.

In this case, since the above-described content can be applied to the method for measuring the biometric index, overlapping descriptions will be omitted.

In addition, at least one of the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from at least one of the first to fourth patients 8811, 8812, 8813, 8814. Biometric information can be obtained. For example, at least one bio-index measuring device among the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from among the first to fourth patients 8811, 8812, 8813, 8814. At least one concentration information may be acquired, but the present invention is not limited thereto.

In addition, at least one of the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from at least one of the first to fourth patients 8811, 8812, 8813, 8814. When there is an abnormality in the biometric information, an alarm may be output so that the rehabilitation therapist 8810 can recognize it. For example, at least one bio-index measuring device among the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from among the first to fourth patients 8811, 8812, 8813, 8814. When there is an abnormality in at least one biometric information, an audible alarm may be output, and information related to the biometric information may be transmitted to the mobile terminal of the rehabilitation therapist 8810, but is not limited thereto.

In addition, at least one of the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from at least one of the first to fourth patients 8811, 8812, 8813, 8814. If there is an abnormality in the biometric information, an alarm may be output so that the patient with the abnormality in the biometric information can recognize it. For example, at least one bio-index measuring device among the first to fifth bio-index measuring devices 8800, 8801, 8802, 8803, and 8804 is selected from among the first to fourth patients 8811, 8812, 8813, 8814. When there is an abnormality in at least one biometric information, an audible alarm may be output, and information related to the biometric information may be displayed through a display of a corresponding patient.

In addition, as described above, monitoring the patient's bio-index and bio-information in real time to maintain concentration while the treatment is being performed may be a method of maximizing the effect of cognitive rehabilitation treatment.

84 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 84 , the operating method 8850 of the apparatus for measuring a biometric index according to an embodiment includes the steps of obtaining the patient's biometric index (S8860), obtaining the patient's biometric information (S8870), and the patient's biometric information It may include at least a part of the step of performing the operation corresponding to (S8880).

At this time, the steps of obtaining the patient's biometric index (S8860) and obtaining the patient's biometric information (S8870) can be applied to the above-described contents, and thus overlapping descriptions will be omitted.

In addition, performing the operation corresponding to the biometric information of the patient ( S8880 ) may include performing various operations corresponding to the various biometric information.

For example, when the biometric information is concentration level information, the biometric index measuring apparatus may output an alarm according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration information, the biometric index measurement apparatus may perform an operation of transmitting information to be displayed on the patient's display according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration information, for example, the biometric index measurement apparatus may perform an operation of transmitting information to the mobile terminal of the rehabilitation therapist according to the concentration level, but is not limited thereto.

Also, for example, when the biometric information is concentration information, the apparatus for measuring the biometric index may perform an operation of outputting a display for increasing the concentration according to the concentration level, but is not limited thereto.

In addition, the biometric index measuring apparatus may perform an operation corresponding to the biometric information and the biometric index.

In addition, the bio-index measuring apparatus may perform various operations corresponding to bio-information and bio-indices in addition to the above-described examples.

9.6. Various embodiments of a biometric index measuring device used for immigration control

Quarantine at airports where people of various countries and races come and go can play an important role as a gatekeeper in the national quarantine system.

However, it may be difficult to closely manage individuals due to lack of management personnel compared to those entering and leaving the airport.

Accordingly, when an individual easily measures a biometric index at a place where an individual undergoes immigration screening, close management of the individual may be possible more easily.

85 is a view for explaining an apparatus for measuring a biometric index used for immigration control according to an embodiment.

Referring to FIG. 85 , biometric index measuring apparatuses 8901 and 8902 according to an exemplary embodiment may be disposed in an immigration kiosk 8900 and may measure biometric indexes of immigrants 8911 and 8912 .

In this case, the biometric index measuring devices 8901 and 8902 may perform facial recognition used for immigration control, but is not limited thereto.

In addition, the bioindices may include at least one of bioindices such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, since the above-described contents may be applied to the method for the bio-index measuring apparatuses 8901 and 8902 to measure the bio-index, the overlapping description will be omitted.

Also, the biometric index measuring apparatus 8901 and 8902 may perform an operation corresponding to the measured biometric index. For example, as shown in FIG. 85 , when the measured body temperature of the first immigration 8911 is within the normal range, the first biometric index measuring device 8901 displays the first information 8921 indicating access permission. The second biometric index measuring device 8902 operates to display the second information 8922 requiring more accurate body temperature measurement when the measured body temperature of the second immigration 8912 is out of the normal range. can, but is not limited thereto.

In addition, as described above, monitoring the biometric index of immigrants during immigration screening to strengthen quarantine may be a way to maximize the effectiveness of national quarantine.

86 is a flowchart for explaining a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 86 , in a method 8950 of operating an apparatus for measuring a biometric index according to an exemplary embodiment, at least a part of obtaining the biometric index of an entry and exiting person ( S8960 ) and performing an operation corresponding to the biometric index ( S8970 ) may include

In this case, since the above-described contents may be applied to the step (S8960) of obtaining the biometric index of the immigration, the overlapping description will be omitted.

In addition, performing the operation corresponding to the biometric index ( S8970 ) may include performing various operations corresponding to the various biometric index.

For example, when the biometric index is a heart rate and the heart rate is within a normal range, the biometric index measuring apparatus may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, when the bio-index is a heart rate and the heart rate is out of a normal range, the bio-index measuring apparatus may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, the bio-index measuring device may perform an operation corresponding to the bio-index when the oxygen saturation level is within a normal range, but is not limited thereto.

Also, for example, the bio-index measuring apparatus may perform an operation corresponding to the bio-index when the oxygen saturation level is out of a normal range, but is not limited thereto.

Also, for example, when the bio-index is blood pressure and the blood pressure is within a normal range, the bio-index measuring apparatus may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, when the bio-index is blood pressure and the blood pressure is out of a normal range, the bio-index measuring apparatus may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, when the biometric index is the body temperature and the body temperature is within a normal range, the apparatus for measuring the biometric index may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, when the biometric index is the body temperature and the body temperature is out of a normal range, the apparatus for measuring the biometric index may perform an operation corresponding thereto, but is not limited thereto.

Also, for example, when the biometric index is within a normal range, the corresponding operation may include, but is not limited to, displaying information allowing access.

Also, for example, when the biometric index is out of a normal range, an operation corresponding thereto may include, but is not limited to, disallowing access or displaying information indicating a subject for close examination.

In addition, the apparatus for measuring the biometric index may perform various operations corresponding to the biometric in addition to the above-described examples.

9.7. Various embodiments of a biometric index measuring device used in a security device

Security devices through biometric recognition such as face recognition, iris recognition, and fingerprint recognition are being activated.

However, the neutralization of security devices through photo and fingerprint copying is also on the rise.

Accordingly, when a security device using a biometric index is provided in addition to biometric recognition, it is possible to prevent the security device from being incapacitated and to provide a stronger security device.

87 is a view for explaining a biometric index measuring device used in a security device according to an embodiment.

Referring to FIG. 87 , the biometric index measuring apparatus 9010 according to an exemplary embodiment may be disposed in the security device 9000 and may measure the biometric index of the subject 9020 .

More specifically, the security device 9000 may perform biometric recognition such as face recognition, fingerprint recognition, and iris recognition of the subject 9020, but is not limited thereto.

In addition, the bio-index measuring apparatus 9010 may obtain the bio-index of the to-be-measured 9020, but is not limited thereto.

In this case, the bioindices may include at least one of bioindices such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, since the above-described contents may be applied to the method for the bio-index measuring apparatus 9010 to measure the bio-index, the overlapping description will be omitted.

Also, the security device 9000 may release security based on biometric recognition and the measured biometric index. For example, the security may be released by determining whether the user is a designated user primarily through iris recognition and secondarily determining whether the user is a human through whether or not a biometric index is obtained, but is not limited thereto.

Also, the security device 9000 may release security based on a plurality of biometrics and a plurality of biometrics. For example, the security may be released by determining whether the user is a designated user primarily through iris recognition and fingerprint recognition, and secondarily determining whether the user is a human through whether or not to obtain a heart rate and oxygen saturation, but is not limited thereto.

Also, the security device 9000 may release security based on the biometric index. For example, when the acquired heartbeat-related signal matches the designated user's heartbeat-related signal, it is determined that the user is the designated user and security may be released, but the present invention is not limited thereto.

In addition, as described above, enhancing security by additionally using a biometric index measuring device in the security window may be a method of maximizing the effect of security reinforcement.

88 is a flowchart illustrating a method of operating a security device according to an embodiment.

Referring to FIG. 88 , the operating method 9050 of the security device according to an embodiment performs a step of determining whether biometric recognition matches (S9060), a step of determining whether a biometric index is obtained (S9070), and an operation according to the result It may include a step (S9080).

In this case, the biometric recognition may include at least one of biometric recognition such as fingerprint recognition, face recognition, and iris recognition.

In addition, the biometric index may include at least one biometric index such as a heart rate, oxygen saturation, blood pressure, body temperature, and the like.

In addition, the step ( S9060 ) of determining whether the biometrics match may be performed using a conventional biometrics recognition method, but is not limited thereto.

In addition, since the above-mentioned contents may be applied to the method of obtaining the biometric index to determine whether to obtain the biometric index, the overlapping description will be omitted.

In addition, the step of determining whether to acquire the biometric index ( S9070 ) may include determining whether to obtain the biometric index. For example, the step of determining whether to obtain the biometric index ( S9070 ) may include determining whether a heart rate is obtained, but is not limited thereto.

In addition, the step of determining whether to obtain the biometric index ( S9070 ) may include determining whether the biometric signal for obtaining the biometric index matches. For example, the step of determining whether to acquire the biometric index ( S9070 ) may include, but is not limited to, determining whether a heartbeat signal for obtaining a heart rate matches a heartbeat signal of a designated user.

In addition, the step of determining whether the biometric index is obtained ( S9070 ) may include determining whether the biometric index is the same. For example, the step of determining whether to obtain the biometric index (S9070) may include determining whether the currently measured heart rate is included within a reference range based on the previously measured heart rate for the subject. , but is not limited thereto.

In addition, the step of performing the operation according to the result (S9080) may include releasing security when all of the security release conditions are satisfied and maintaining security when the security release conditions are not satisfied, but is limited thereto. doesn't happen

In addition, the security device may perform various operations for enhancing security by using a biometric index in addition to the above-described examples.

9.8. Various embodiments of a biometric index measuring device used in a kiosk

When a bio-index measuring device that can easily measure a bio-index is disposed in a kiosk for displaying information, it is possible to monitor the bio-index in daily life to monitor the individual's health more continuously.

89 is a view for explaining an apparatus for measuring a biometric index used in a kiosk according to an embodiment.

Referring to FIG. 89 , the biometric index measuring apparatus 91110 according to an exemplary embodiment may be disposed in the kiosk 9100 and may measure the biometric index of a subject 9120 .

In this case, the kiosk 9100 may include a display for displaying at least one piece of information. For example, as shown in FIG. 89 , the kiosk 9100 may display date information and weather information, but is not limited thereto.

Also, the biometric index measuring apparatus 91110 may acquire at least one biometric index. For example, the bio-index measuring apparatus 91110 may acquire a heart rate, but is not limited thereto, and may acquire at least one bio-index among bio-indexes such as heart rate, oxygen saturation, blood pressure, and body temperature.

Also, the biometric index measuring apparatus 91110 may acquire at least one piece of biometric information. For example, the biometric index measuring device 91110 may obtain today's condition information, but is not limited thereto, and obtains at least one biometric information among biometric information such as drowsiness information, condition information, concentration information, and health information. can do.

Also, the kiosk 9100 may display the obtained biometric index. For example, as shown in FIG. 89 , the kiosk 9100 may display the acquired heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

Also, the kiosk 9100 may display the obtained biometric information. For example, as shown in FIG. 89 , the kiosk 9100 may display acquired today's condition information, but is not limited thereto.

Also, the kiosk 9100 may transmit the obtained biometric index and biometric information. For example, the kiosk 9100 may transmit the obtained biometric index and biometric information to the mobile terminal of the subject 9120, but is not limited thereto.

Also, the kiosk 9100 may transmit the obtained biometric index and biometric information. For example, the kiosk 9100 may transmit the obtained biometric index and biometric information to a terminal that an administrator can check, but is not limited thereto.

In addition, in this case, it is possible to detect a patient with an abnormal biometric index in the building in which the kiosk 9100 is disposed, thereby enhancing the prevention and security of the building.

In addition, as described above, the additional use of the biometric index measuring device in the kiosk enables continuous monitoring of individual health, and can be a method to strengthen the prevention and security of the building in which the kiosk is disposed.

In addition, the above-mentioned kiosk device can be used for manpower management, for example, it is used in a kiosk for the entrance and exit of construction site workers so that measures can be taken for construction site workers with abnormal biometric index. may be, but is not limited thereto.

90 is a flowchart illustrating a method of operating an apparatus for measuring a biometric index according to an exemplary embodiment.

Referring to FIG. 90 , a method 9150 of operation of an apparatus for measuring a biometric index according to an embodiment includes obtaining a biometric index (S9160), obtaining biometric information (S9170), and outputting the biometric index and biometric information It may include at least a part of the step (S9180).

In this case, since the above-described method for obtaining the biometric index may be applied to the step of obtaining the biometric index ( S9160 ), the overlapping description will be omitted.

In addition, since the above-described method for obtaining biometric information may be applied to the step of obtaining the biometric information ( S9170 ), a redundant description will be omitted.

Also, the step of outputting the biometric index and biometric information ( S9180 ) may include displaying the biometric index and biometric information. For example, the step of outputting the biometric index and biometric information ( S9180 ) may include, but is not limited to, displaying heart rate, oxygen saturation, blood pressure, and today's condition information.

In addition, the step of outputting the biometric index and biometric information (S9180) may include transmitting the biometric index and biometric information. For example, the step of outputting the biometric index and biometric information ( S9180 ) may include, but is not limited to, transmitting heart rate, oxygen saturation, blood pressure, and today's condition information to another mobile terminal.

Also, the step of outputting the biometric index and biometric information (S9180) may include printing the biometric index and biometric information. For example, the step of outputting the biometric index and biometric information ( S9180 ) may include printing heart rate, oxygen saturation, blood pressure, and today's condition information, but is not limited thereto.

In addition, the biometric index acquisition device may perform various operations for acquiring and outputting the biometric index and biometric information in addition to the above-described examples.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (35)

A smart mirror device that displays at least two different types of biometrics among heart rate, oxygen saturation, blood pressure, and body temperature, comprising:
reflective mirror surface;
an image acquisition unit for acquiring a plurality of image frames of the subject;
a display unit disposed behind the reflective mirror surface and configured to display visual information through the reflective mirror surface; and
a control unit for controlling the operation of the image acquisition unit and the display unit and acquiring a biometric index in a non-contact manner; including,
the control unit
a first type biometric index obtained based on a first difference between values of at least two color channels associated with a first region of a first image frame group included in the plurality of image frames is displayed at a first time point; control the display,
a second type biometric index obtained based on a second difference between values of at least two or more color channels associated with a second region different from the first region of a second image frame group included in the plurality of image frames; controlling the display unit to be displayed at a second time point,
The first image frame group and the second image frame group include at least one image frame in common so that the first type biometric index and the second type biometric index are correlated;
The at least one image frame commonly included in the first image frame group and the second image frame group is the first image frame of the subject whose face faces the reflective mirror surface before the first and second time points. 1 containing the image frame acquired in state
smart mirror device.
According to claim 1,
wherein the first and second type biometric indexes include different biometric indexes of at least one of heart rate, oxygen saturation, blood pressure, and body temperature
smart mirror device.
According to claim 1,
The first image frame group and the second image frame group are identical to each other
smart mirror device.
According to claim 1,
The first image frame group and the second image frame group are different from each other
smart mirror device.
According to claim 1,
The first time point and the second time point are the same time point
smart mirror device.
According to claim 1,
The first time point is a time point earlier than the second time point,
The at least one image frame included in common in the first image frame group and the second image frame group is in a second state of the subject in which the face of the subject faces the reflective mirror surface before the first time point. containing the acquired image frame
smart mirror device.
According to claim 1,
The control unit controls the display unit to display a third type biometric index obtained based on a third image frame group included in the plurality of image frames at a third time point,
The first, second, and third image frame groups commonly include at least one image frame so that the first, second, and third type biometric indexes are correlated;
At least one image frame commonly included in the first, second, and third image frame groups includes an image frame acquired in the first state before the first, second, and third time points
smart mirror device.
According to claim 1,
the control unit
obtaining the first and second type biometric indexes based on at least a portion of an image frame obtained during a first time among the plurality of image frames;
Recognizing the subject based on at least one image frame acquired for a second time among the plurality of image frames,
The second time is shorter than the first time, the second time is included in the first time,
Displaying information on the subject recognized before the first and second type biometric indexes are displayed
smart mirror device.
9. The method of claim 8,
The information on the subject is the first and second type biometric indexes of the subject that have been previously measured and stored.
smart mirror device.
According to claim 1,
the control unit
Controls the display unit so that the first information is displayed at a fourth time,
Control the display so that the second information is displayed at a fifth time point,
The first information includes weather information and time information,
The second information includes information about the subject,
The fourth time point is a time point preceding the fifth time point,
The fifth time point is a time point preceding the first and second time points.
smart mirror device.
11. The method of claim 10,
The second information includes at least one of schedule information, medication information, recognition information, messenger information, and interest information of the subject.
smart mirror device.
11. The method of claim 10,
the control unit
controlling the display unit to display the first information at the fourth time point,
controlling the display unit to display the first information and the second information at the fifth time point;
controlling the display unit to display the first information, the second information, the first type biometric index, and the type second biometric index at the second time point;
smart mirror device.
According to claim 1,
the control unit
to obtain the first type biometric index and the second type biometric index based on at least three color channel values for the plurality of image frames;
operating so that both the first type biometric index and the second type biometric index are displayed within 10 seconds from the time of acquiring the first image frame included in the first image frame group
smart mirror device.
According to claim 1,
the control unit
When an image of a plurality of people is included in the image frame acquired through the image acquisition unit, the first and second type biometric indexes are acquired based on the image of one of the plurality of people selected according to priority and
and controlling the display unit so that the obtained first and second type biometric indexes are displayed, and information on a selected one of the plurality of persons is displayed.
smart mirror device.
15. The method of claim 14,
the control unit
When an image frame including the first subject and the second subject is obtained after the image frame including the first subject is obtained, the first and second types based on the image of the first subject to obtain a biometric index,
controlling the display unit to display information on the first subject
smart mirror device.
15. The method of claim 14,
the control unit
Determining a priority for the plurality of people based on the obtained image frame
smart mirror device.
According to claim 1,
the control unit
When images of the first subject and the second subject are included in the image frame acquired through the image acquisition unit
obtaining the first type biometric index for the first subject based on the first image frame group included in the plurality of image frames;
obtaining the second type biometric index for the first subject based on the second image frame group included in the plurality of image frames;
obtaining the first type biometric index for the second subject based on a fourth image frame group included in the plurality of image frames;
to obtain the second type biometric index for the second subject based on a fifth image frame group included in the plurality of image frames,
The fourth and fifth image frame groups include at least one image frame in common.
smart mirror device.
According to claim 1,
The control unit acquires biometric information based on the first type biometric index,
Control the display unit to display the biometric information at a third time point,
In order to display the first type biometric index and the biometric information in real time, the biometric information displayed at the third time point is obtained based on the first type biometric index displayed at the first time point,
The first time point is a time point earlier than the third time point
smart mirror device.
19. The method of claim 18,
The control unit acquires the biometric information based on the first and second type biometric indexes,
In order to improve the accuracy of the biometric information, the biometric information displayed at the third time point is based on the first type biometric index displayed at the first time point and the second type biometric index displayed at the second time point. is obtained,
The third time point is a time point later than the first and second time points.
smart mirror device.
19. The method of claim 18,
The biometric information includes at least one or more biometric information of condition information, concentration information, drowsiness information, and emotional information.
smart mirror device.
According to claim 1,
The control unit acquires biometric information based on the first type biometric index,
Control the display unit to display the biometric information at a third time point,
In order to display accurate biometric information, the biometric information displayed at the third time point is obtained based on a plurality of the first type biometric indexes displayed at a plurality of time points up to the third time point,
The third time point is a time point later than the first time point
smart mirror device.
22. The method of claim 21,
The biometric information displayed at the third time point is obtained based on an average value of the plurality of first type biometric indices displayed from the first time point to the third time point
smart mirror device.
22. The method of claim 21,
The control unit acquires the biometric information based on the first and second type biometric indexes,
The biometric information displayed at the third time point is obtained based on the first type biometric index displayed up to the third time point and the second type biometric index displayed up to the third time point,
The third time point is a time point later than the first and second time points.
smart mirror device.
According to claim 1,
The period in which the first type biometric index is updated and the period in which the second type biometric index is updated are different from each other
smart mirror device.
According to claim 1,
the control unit
Decide on the circumstances of providing information;
In the case of a first situation, the display unit is controlled to display the first information, and in the case of a second situation, the display unit is controlled to display the second information,
The first information includes first and second type biometric indexes, and the second information does not include the first and second type biometric indexes.
smart mirror device.
26. The method of claim 25,
The control unit determines the information provision situation based on the moving direction of the subject
smart mirror device.
27. The method of claim 26,
The smart mirror device further comprises a motion detection sensor for detecting the moving direction of the subject
smart mirror device.
26. The method of claim 25,
The control unit determines the information provision situation based on the obtained plurality of image frames
smart mirror device.
26. The method of claim 25,
The first information includes at least one of external weather information, schedule information of the subject, and time information,
The second information includes at least one of internal temperature information, internal humidity information, internal air information, security information, and active time information
smart mirror device.
26. The method of claim 25,
The first information includes at least two different types of biometric indexes,
The second information includes at least one of weather information, time information, news information, schedule information, and medication information.
smart mirror device. delete delete delete delete delete

Images