WO2021100994A1

WO2021100994A1 - Non-contact method for measuring biological index

Info

Publication number: WO2021100994A1
Application number: PCT/KR2020/006144
Authority: WO
Inventors: 박기범
Original assignee: 주식회사 지비소프트
Priority date: 2019-11-21
Filing date: 2020-05-11
Publication date: 2021-05-27

Abstract

The present invention relates to a method for measuring a biological index and a device for measuring a biological index, and more particularly, to a non-contact method for measuring a biological index, and a device for measuring a biological index using same. The method for measuring a biological index according to the present invention may comprise the steps of: obtaining a plurality of image frames of a subject; obtaining a first color channel value, a second color channel value, and a third color channel value of at least one image frame among the plurality of image frames; and determining a biological index for the subject on the basis of the first color channel value, the second color channel value, and the third color channel value of the at least one image frame among the plurality of image frames.

Description

Non-contact biometric index measurement method

Plethysmography is a technique for measuring and analyzing changes in shape or shape when the volume of human tissues such as human organs or blood vessels changes according to the flow of blood vessels.

The most common technique for measuring photoplethysmography (PPG) using light uses the method of analyzing the amount of transmitted light for the light irradiated to the human body, and the absorbance is proportional to the concentration of the absorbed material and the thickness of the absorbing layer. It is explained by the Beer-Lambert law. According to this law, since the change in transmitted light results in a signal proportional to the change in the volume of the transmitted material, it is possible to grasp the state of the heart by using PPG even if the absorbance of the material is not known.

In recent years, a technology using remote photoplethysmography (rPPG) has emerged one step further from the technology using PPG. As the most popular technology to grasp the signal related to the heartbeat using PPG, it is a technology that obtains PPG by directly contacting the human body with a device with a camera and a light attached to it, such as a smartphone, to irradiate light and immediately measure the transmitted light. If there is, in recent years, a technology related to rPPG (remote photoplethysmography) that grasps changes in the volume of blood vessels from signals acquired from images taken with a camera is continuously researched and developed.

The technology using rPPG can be applied in various ways in devices and places equipped with cameras, such as airport immigration offices and remote medical treatments, in that contact between an object and measurement equipment is not required.

However, the technology for rPPG is a technology that extracts only the signal related to the volume change of the object to be measured from the captured image because the noise generated by the ambient light and the movement of the object has a large effect on the signal in the process of photographing the object with the camera. It can be seen as a core technology among technologies for measuring bio-signals using this rPPG.

The problem to be solved according to an embodiment is to obtain a biometric index in a non-contact manner.

A problem to be solved according to another exemplary embodiment is to reduce noise caused by movement of a subject in order to obtain a biometric index.

Another problem to be solved according to an exemplary embodiment is to reduce noise due to a change in the intensity of external light in order to obtain a biometric index.

The task to be solved according to another embodiment is to obtain various biomarkers at the same time.

Another task to be solved according to an embodiment is to acquire biometric information based on various biometric indexes.

The task to be solved according to another embodiment is to simultaneously acquire various biomarkers to be correlated.

Another task to be solved according to an embodiment is to detect drowsiness based on the heart rate of the subject and the LF/HF of the heart rate signal.

The problem to be solved according to another embodiment relates to a smart mirror device that acquires at least two or more biomarkers that are related.

Another problem to be solved according to an embodiment relates to a method of operating a smart mirror device for acquiring at least two related biomarkers.

Another problem to be solved according to an embodiment relates to a smart mirror device including an opening and closing device.

A method of measuring a biometric index in a non-contact manner according to an embodiment includes the steps of obtaining a plurality of image frames for a subject, a first color channel value and a second value for at least one image frame included in the plurality of image frames. Acquiring a color channel value and a third color channel value, wherein the first color channel value, the second color channel value, and the third color channel value are determined for at least one image frame included in the plurality of image frames. Based on the first difference value and the second difference value. Calculating-the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value for the same image frame Represents a difference value between the value of the third color channel and the value of the third color channel-, the first difference value with respect to at least one image frame included in the first image frame group with respect to the first image frame group acquired during a first preset time And obtaining a first characteristic based on an average value of the first difference value for the first image frame group, a second difference value for at least one image frame included in the first image frame group, and the Obtaining a second characteristic value based on the average value of the second difference value for the first image frame group, and determining a biometric index for the subject based on the first characteristic value and the second characteristic value. Including a step, wherein the first color channel value is an average pixel value of a first color channel for one image frame, and the second color channel value is an average pixel value of a second color channel for one image frame. , The third color channel value may represent an average pixel value of the third color channel for one image frame.

A method of measuring a biometric index using an infrared camera according to an embodiment includes obtaining a plurality of image frames for a subject using an infrared camera, with respect to at least one image frame included in the plurality of image frames. Acquiring a first region value, a second region value, and a third region value, the first region value, the second region value, and the third region with respect to at least one image frame included in the plurality of image frames Calculating a first difference value and a second difference value based on the value, with respect to at least one image frame included in the first image frame group with respect to the first image frame group acquired during a first preset time. Obtaining a first characteristic value based on a first difference value and an average value of the first difference value for the first image frame group, and a first characteristic value for at least one image frame included in the first image frame group. 2 Acquiring a second characteristic value based on a difference value and an average value of the second difference value for the first image frame group, and for the subject based on the first characteristic value and the second characteristic value. Determining a biometric index, wherein the first region value is an average pixel value for a first region of interest in one image frame, and the second region value is an average for a second region of interest in one image frame. Is a pixel value, the third region value is an average pixel value for a third region of interest in one image frame, and the first difference value is a difference value between a first region value and a second region value for the same image frame , The second difference value may be a difference value between a first area value and a third area value for the same image frame.

An apparatus for obtaining a biometric index according to an embodiment includes an image acquisition unit for obtaining an image frame for the subject and a control unit for obtaining a biometric index using the image frame, wherein the control unit includes a plurality of acquired images. A first color channel value, a second color channel value, and a third color channel value are obtained for at least one image frame included in the frame, and the first color channel value is obtained for at least one image frame included in the plurality of image frames. A first difference value and a second difference value are calculated based on a color channel value, the second color channel value, and the third color channel value. A first characteristic value is obtained based on an average value of the first difference value for at least one image frame included in one image frame group and the first difference value for the first image frame group, and the first A second characteristic value is obtained based on a second difference value for at least one image frame included in an image frame group and an average value of the second difference value for the first image frame group, and the first characteristic value And a biometric index for the subject is determined based on a second characteristic value, wherein the first color channel value is an average pixel value for a first color channel in one image frame, and the second color channel value is one Is an average pixel value for a second color channel in an image frame of, the third color channel value is an average pixel value for a third color channel in one image frame, and the first difference value is a second color channel value for the same image frame. A difference value between a first color channel value and a second color channel value, and the second difference value may be a difference value between a first color channel value and a third color channel value for the same image frame.

A method of measuring a non-contact biometric index according to an embodiment includes the steps of obtaining a plurality of image frames for a subject, a first color channel value and a second color channel value for at least one image frame included in the plurality of image frames. And obtaining a third color channel value, based on the first, second, and third color channel values for at least one image frame included in the first image frame group acquired for a first preset time. 1 obtaining a characteristic value, a second characteristic value based on the first, second, and third color channel values for at least one image frame included in a second image frame group acquired during a second preset time And determining a biometric index based on the first characteristic value and the second characteristic value, wherein the first image frame group and the second image frame group overlap at least partially, and the living body To determine the index, a first characteristic value for a first image frame included in the first image frame group and not included in the second image frame group is used, and the first image frame group and the second The first characteristic and the second characteristic value of the second image frame included in the image frame group are used, and the third image frame included in the second image frame group, but not included in the first image frame group, is used. A second characteristic value can be used.

A method of measuring a non-contact biometric index according to an embodiment comprises the steps of obtaining a plurality of image frames including a first image frame group and a second image frame group at least partially overlapping with the first image frame group, the first image frame Obtaining a biometric index based on a group, outputting a biometric index at a first time point, obtaining a first biometric index based on the second image frame group, at a second time point later than the first time point And outputting a biometric index, wherein the biometric index output at the first time point is a biometric index obtained based on the first image frame group, and the biometric index output at the second time point is the first biometric index When the difference between the biometric index output at the first time point and the first time point is less than the reference value, it is the first biometric index, and when the difference between the first biometric index and the biometric index output at the first time point exceeds the reference value, It may be a biometric index corrected from the biometric index output at the time point 1.

A method of measuring a biometric index according to an embodiment includes the steps of obtaining a plurality of image frames for a subject, setting a first region and a second region for at least one image frame included in the plurality of image frames, Determining the oxygen saturation degree of the subject based on a first characteristic obtained based on at least two color channel values of a first color channel value, a second color channel value, and a third color channel value for the first area Determining a heart rate of the subject based on a second characteristic obtained based on a first difference value that is a difference value between the first color channel value and the second color channel value for the first region, the Determining the blood pressure of the subject based on a second difference value that is a difference value between the first color channel value and the second color channel value for a second area, and a third characteristic obtained based on the first difference value Step, outputting the oxygen saturation degree, heart rate, and blood pressure; Including, wherein the first feature is acquired based on a first image frame group acquired during a first time, the second feature is acquired based on a second image frame group acquired during a second time, and the second feature The 3 features are obtained based on the third image frame group acquired during the third time period, and the first image frame group, the second image frame group, and the third image frame are obtained so that the oxygen saturation, heart rate, and blood pressure are acquired in association with each other. The group may include a plurality of image frames in common.

A method of measuring a biometric index according to another embodiment includes the steps of obtaining a plurality of image frames for a subject, a first region, a second region, and a third region for at least one image frame included in the plurality of image frames. Setting, based on a first characteristic obtained based on at least two color channel values among a first color channel value, a second color channel value, and a third color channel value for the first region, Determining an oxygen saturation degree, based on a second characteristic obtained based on a first difference value that is a difference value between the first color channel value and the second color channel value for the first region, the heart rate of the subject Determining, a second difference value that is a difference value between the first color channel value and the second color channel value for the second area, and the first color channel value and the second color channel value for the third area Determining the blood pressure of the subject based on a third characteristic obtained based on a third difference value that is a difference value of, and outputting the oxygen saturation degree, heart rate, and blood pressure, wherein the first characteristic is the first It is acquired based on a first image frame group acquired during a time, the second feature is acquired based on a second image frame group acquired during a second time, and the third feature is obtained during a third time. It is acquired based on an image frame group, and the first image frame group, the second image frame group, and the third image frame group may include a plurality of image frames in common so that the oxygen saturation degree, heart rate, and blood pressure are obtained in a correlation with each other. I can.

In another exemplary embodiment, a method of measuring a biometric index includes obtaining a plurality of image frames for a subject, a first color channel value and a second color channel for at least one image frame included in the plurality of image frames. Obtaining a value and a third color channel value, the blood based on a first characteristic obtained based on at least two color channel values of the first color channel value, the second color channel value, and the third color channel value. Determining a measurer's oxygen saturation, a first difference value that is a difference value between the first color channel value and the second color channel value, and a second difference value that is a difference value between the first color channel value and the third color channel value Determining a heart rate of the subject based on a second characteristic obtained based on, determining a blood pressure of the subject based on a third characteristic obtained based on the first difference value and the second difference value And outputting the oxygen saturation, heart rate, and blood pressure, wherein the first feature is obtained based on a first image frame group acquired during a first time, and the second feature is obtained during a second time. The first image is acquired based on a second image frame group, and the third feature is acquired based on a third image frame group acquired during a third time period, and the oxygen saturation, heart rate, and blood pressure are acquired in association with each other. The frame group, the second image frame group, and the third image frame group may include a plurality of image frames in common.

A method for measuring a biometric index according to another embodiment includes the steps of obtaining a plurality of image frames for a subject, setting at least two or more regions for at least one image frame included in the plurality of image frames, the Obtaining a first color channel value, a second color channel value, a third color channel value, a fourth color channel value, and a fifth color channel value for at least one image frame included in a plurality of image frames, the first Determining an oxygen saturation degree of the subject based on a first characteristic obtained based on at least two color channel values of a first color channel value, a second color channel value, and a third color channel value, the first color channel Determining a heart rate of the subject based on a second characteristic obtained based on at least two color channel values of a value, a second color channel value, and a third color channel value, the first color channel value, and a second Determining a blood pressure of the subject based on a third characteristic obtained based on at least two color channel values of a color channel value and a third color channel value, a fourth obtained based on the fourth color channel value Determining a body temperature of the subject based on a characteristic, and outputting the oxygen saturation degree, heart rate, body temperature, and blood pressure, wherein the first color channel value is a Green channel value, and the second color channel value is A red channel value, the third color channel value may be a blue channel value, the fourth color channel value may be a saturation channel value, and the fifth color channel value may be a Hue channel value.

In another embodiment, a biometric index measurement method includes obtaining a plurality of image frames for a subject, obtaining N preliminary heart rates based on at least one image frame included in the plurality of image frames, Obtaining M preliminary oxygen saturation based on at least one image frame included in the plurality of image frames, obtaining K preliminary blood pressure based on at least one image frame included in the plurality of image frames, Acquiring a heart rate based on the N preliminary heart rates, obtaining an oxygen saturation based on the M preliminary oxygen saturation, acquiring blood pressure based on the K preliminary blood pressure, and the heart rate, oxygen saturation, and Including the step of outputting a blood pressure, wherein when the image frame obtained at the time point in which the subject is in the first state is a first image frame, the first image frame includes the N preliminary heart rates, the M preliminary oxygen saturation, and It may be commonly included in an image frame for obtaining the K blood pressures.

A method of obtaining biometric information according to an embodiment includes obtaining a plurality of image frames for a subject, a first biometric based on a first image frame group including at least one image frame included in the plurality of image frames. Obtaining an index, obtaining a second biometric index based on a second image frame group including at least one image frame included in the plurality of image frames, the first biometric index and the second biometric index Acquiring biometric information based on at least one of and outputting the first biometric index, the second biometric index, and biometric information, wherein the biometric information is obtained by reflecting the specific state of the subject. The first image frame group and the second image frame group may at least partially overlap.

The method of detecting drowsiness based on a heart rate according to an embodiment may be performed by at least one processor, obtaining a heart rate of an object, comparing the heart rate with a reference heart rate to obtain a comparison result, the Comparing a heart rate and a reference heart rate to obtain a comparison result, obtaining a duration for which the heart rate is less than or equal to the reference heart rate based on the comparison result, and whether the duration reaches a reference duration It may include a drowsiness detection step of determining the drowsiness state of the subject on the basis of.

The method of detecting drowsiness based on heart rate and LF/HF according to an embodiment may be performed by at least one processor, obtaining a heart rate for an object, comparing the heart rate with a reference heart rate to obtain a comparison result. Acquiring, obtaining a first drowsiness parameter based on the comparison result, obtaining a low frequency/high frequency (LF/HF) value representing a ratio of the sympathetic nerve activity and the parasympathetic nerve activity of the subject And acquiring a second drowsiness parameter based on the LF/HF value of the subject, and determining a drowsiness state of the subject using at least one of the first drowsiness parameter and the second drowsiness parameter. can do.

The smart mirror device according to an embodiment includes a reflective mirror surface, an image acquisition unit for acquiring a plurality of image frames for a subject, and a display disposed behind the reflective mirror surface and passing through the reflective mirror surface to display visual information. And a control unit for controlling the operation of the unit and the image obtaining unit and the display unit and obtaining a biometric index in a non-contact manner, wherein the control unit is based on a first image frame group included in the plurality of image frames at a first time point. Controlling the display unit to display the obtained first biometric index, and controlling the display unit to display a second biometric index obtained based on a second image frame group included in the plurality of image frames at a second time point, The first image frame group and the second image frame group include at least one image frame in common so that a correlation is given to the first biometric index and the second biometric index, and the first image frame group and the second The at least one image frame commonly included in an image frame group includes an image frame acquired in the first state of the subject observed by the subject through the reflective mirror surface before the first and second viewpoints. can do.

A method of operating a smart mirror device according to an embodiment includes obtaining an on-trigger, obtaining a plurality of image frames for a subject, a first image frame group included in the plurality of image frames Obtaining a first biometric index based on, obtaining a second biometric index based on a second image frame group included in the plurality of image frames, displaying the first and second biometric indexes, Obtaining an off-trigger and stopping the acquisition of a plurality of image frames for the subject, wherein the first image is associated with the first biometric index and the second biometric index. The frame group and the second image frame group commonly include at least one image frame, the on-trigger is obtained from at least one sensor, and the off-trigger is the plurality of It can be obtained from an image sensor to obtain an image frame.

The smart mirror device according to another exemplary embodiment includes a reflective mirror surface, an image acquisition unit for acquiring a plurality of image frames, a display unit disposed behind the reflective mirror surface and passing through the reflective mirror surface to display visual information, and the It is disposed in front of the image acquisition unit, and includes an opening/closing unit for opening and closing the field of view of the image acquisition unit, and a control unit for controlling the operation of the image acquisition unit and the display unit and obtaining a biometric index, wherein the surface of the opening/closing unit is formed of a reflective mirror. When the opening/closing unit is in an open state, the control unit controls the display unit so that the biometric index and at least one visual information are displayed through the display unit, and when the opening/closing unit is in a closed state, the control unit provides at least one visual information to the display unit. The display unit may be controlled to be displayed through the display unit.

According to an embodiment of the present invention, a method of obtaining a biometric index in a non-contact manner may be provided.

According to another embodiment of the present invention, a method of reducing noise caused by a movement of a subject may be provided.

According to another embodiment of the present invention, a method of reducing noise due to a change in intensity of external light may be provided.

According to another embodiment of the present invention, a method of simultaneously acquiring various biomarkers may be provided.

According to another embodiment of the present invention, a method of acquiring biometric information based on various biomarkers may be provided.

According to still another embodiment of the present invention, a method of acquiring various biomarkers to be related at the same time may be provided.

According to another embodiment of the present invention, a smart mirror device may be provided that acquires at least two or more biomarkers that are related.

According to another embodiment of the present invention, a method of operating a smart mirror device for acquiring at least two or more biomarkers that are related may be provided.

A problem to be solved according to another embodiment may be a method and apparatus for detecting drowsiness based on a heart rate of an object and LF/HF of a heart rate signal.

According to another embodiment of the present invention, a smart mirror device including an opening and closing device may be provided.

1 is a diagram of a biometric index and biometric information management system according to an exemplary embodiment.

2 is a diagram of a biometric index and biometric information management system according to another embodiment.

3 is a diagram illustrating an apparatus for obtaining a biometric index according to an exemplary embodiment.

4 is a flowchart illustrating a method of obtaining a biometric index according to an exemplary embodiment.

5 is a diagram illustrating a method of obtaining a biometric index according to an exemplary embodiment.

6 is a flowchart illustrating a method of obtaining biometric information according to an exemplary embodiment.

7 and 8 are diagrams for explaining a method of obtaining a biometric index using a biometric index obtaining model.

9 is a flowchart illustrating a heart rate measurement method according to an exemplary embodiment.

10 is a flowchart illustrating a method of measuring oxygen saturation according to an exemplary embodiment.

11 is a flowchart illustrating a method of measuring oxygen saturation according to another exemplary embodiment.

12 is a flowchart illustrating a blood pressure measurement method according to an exemplary embodiment.

13 is a flowchart illustrating a method of measuring blood pressure according to another exemplary embodiment.

14 is a flowchart illustrating a method of measuring body temperature according to an exemplary embodiment.

15 is a flowchart illustrating a method of obtaining a heart rate according to an exemplary embodiment.

16 is a graph of color channel values according to an exemplary embodiment.

17 is a graph for explaining a noise reduction method according to an exemplary embodiment.

18 is a diagram showing the absorbance of hemoglobin and oxygen hemoglobin in the visible light band.

19 is a diagram for describing a method of obtaining a characteristic value according to an exemplary embodiment.

20 is a diagram for describing a method of obtaining a characteristic value according to another exemplary embodiment.

21 is a diagram for describing a method of using a plurality of characteristic values.

22 is a graph obtained by extracting a frequency component from a graph for characteristic values.

23 is a diagram for describing a method of obtaining a heart rate according to an exemplary embodiment.

24 is a flowchart illustrating a method of correcting an output heart rate according to an exemplary embodiment.

25 is a diagram for describing a method of extracting a heart rate signal according to an exemplary embodiment.

26 is a diagram for describing a method of acquiring a heart rate using infrared rays according to an exemplary embodiment.

27 is a diagram for describing a method of obtaining a heart rate using infrared rays according to an exemplary embodiment.

28 is a flowchart illustrating a method of obtaining a biometric index according to an exemplary embodiment.

29 is a diagram for describing a method of acquiring a plurality of biomarkers and biometric information according to an exemplary embodiment.

30 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

31 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

32 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

33 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

34 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

35 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

36 is a diagram for describing a method of obtaining a plurality of correlated biomarkers according to an exemplary embodiment.

37 is a diagram for describing a method of obtaining a plurality of biomarkers according to an exemplary embodiment.

38 is a block diagram of an apparatus for detecting drowsiness based on heart rate.

39 is a flowchart of a method of detecting drowsiness based on heart rate.

40 is a graph of an average heart rate of a subject based on the heart rate of a subject.

41 is a graph of a heart rate of a test subject for explaining a situation in which the test subject is detected as a drowsy state based on the heart rate.

42 is a graph of heart rate including noise.

43 is a graph for explaining a situation in which a subject has recovered to a normal state.

44 is a flowchart of a method of detecting drowsiness based on LF/HF.

45 is a graph of the LF/HF of the test subject for explaining a situation in which the test subject is detected as drowsy based on the LF/HF.

46 is a graph of the LF/HF of the test subject for indicating a situation in which the test subject recovers from a drowsy state based on the LF/HF.

47 is a flowchart of a method of detecting drowsiness based on heart rate and LF/HF.

48 is a diagram for describing a smart mirror device according to an exemplary embodiment.

49 is a diagram for describing a smart mirror device according to an embodiment.

50 is a diagram for describing a smart mirror device in which a guide area is output according to an exemplary embodiment.

51 is a diagram for describing a smart mirror device in which predetermined information is output according to an exemplary embodiment.

52 is a diagram for describing a smart mirror device according to an embodiment.

53 is a diagram for describing a display device measuring a biometric index in real time according to an exemplary embodiment.

54 is a diagram for describing a smart mirror device in which predetermined information is output according to an exemplary embodiment.

55 is a view for explaining a smart mirror device including an opening and closing device according to an embodiment.

56 is a diagram for describing a smart mirror device disposed in a shoe rack according to an exemplary embodiment.

57 is a flowchart illustrating a method of operating a smart mirror device according to an exemplary embodiment.

58 and 59 are diagrams for describing an operation of a smart mirror device using a trigger signal according to an exemplary embodiment.

60 is a diagram for describing a method of operating a smart mirror device according to an exemplary embodiment.

61 is a diagram for describing an operation of a smart mirror device according to an embodiment.

62 is a diagram for describing a method of operating a smart mirror device according to an exemplary embodiment.

63 is a diagram for describing an operation of a smart mirror device according to an exemplary embodiment.

64 is a diagram for describing a method of operating a smart mirror device according to an exemplary embodiment.

65 and 66 are diagrams for describing an operation of a smart mirror according to an exemplary embodiment.

67 is a diagram for describing an apparatus for measuring a biometric index according to an exemplary embodiment.

68 is a diagram for describing an apparatus for measuring a biometric index according to an exemplary embodiment.

69 is a view for explaining a biometric index measuring apparatus disposed in an autonomous vehicle according to an embodiment.

70 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

71 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

72 is a diagram for describing a method of operating a driving scheduling auxiliary device using a biometric index measuring device according to an exemplary embodiment.

73 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

74 is a flowchart illustrating a method of operating an apparatus for calculating a driving index according to an exemplary embodiment.

75 is a diagram for describing an apparatus for monitoring infants and toddlers according to an embodiment.

76 is a diagram for describing occurrence of an event for infants and toddlers.

77 is a flowchart illustrating a method of operating an infant monitoring apparatus according to an exemplary embodiment.

78 is a flowchart illustrating a method of operating an infant monitoring apparatus according to an embodiment.

79 is a diagram illustrating a mobile application for implementing an infant monitoring system according to an embodiment.

80 is a view for explaining a biometric index measuring device disposed in a reading room according to an exemplary embodiment.

81 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

82 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

83 is a diagram for describing an apparatus for measuring a biometric index used for cognitive rehabilitation treatment according to an exemplary embodiment.

84 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

85 is a diagram for describing an apparatus for measuring a biometric index used for immigration screening, according to an exemplary embodiment.

86 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

87 is a diagram for describing a biometric index measuring device used in a security device according to an exemplary embodiment.

88 is a flowchart illustrating a method of operating a security device according to an embodiment.

89 is a diagram for describing an apparatus for measuring a biometric index used in a kiosk according to an exemplary embodiment.

90 is a flowchart illustrating a method of operating a biometric index measuring apparatus according to an exemplary embodiment.

The embodiments described in the present specification are intended to clearly explain the spirit of the present invention to those of ordinary skill in the technical field to which the present invention belongs, and thus the present invention is not limited to the embodiments described in the present specification. The scope should be construed as including modifications or variations that do not depart from the spirit of the present invention.

The terms used in this specification have been selected as general terms that are currently widely used in consideration of functions in the present invention, but this varies depending on the intention of a person of ordinary skill in the art, precedents, or the emergence of new technologies. I can. However, if a specific term is defined and used in an arbitrary meaning unlike this, the meaning of the term will be separately described. Therefore, the terms used in the present specification should be interpreted based on the actual meaning of the term and the entire contents of the present specification, not a simple name of the term.

The drawings attached to the present specification are for easy explanation of the present invention, and the shapes shown in the drawings may be exaggerated and displayed as necessary to aid understanding of the present invention, so the present invention is not limited by the drawings.

In the present specification, when it is determined that a detailed description of a well-known configuration or function related to the present invention may obscure the subject matter of the present invention, a detailed description thereof will be omitted as necessary.

According to an embodiment, a method of measuring a biometric index in a non-contact manner, comprising: obtaining a plurality of image frames for a subject, a first color channel value for at least one image frame included in the plurality of image frames, Obtaining a second color channel value and a third color channel value, the first color channel value, the second color channel value, and the third color channel for at least one image frame included in the plurality of image frames A first difference value and a second difference value based on the value. Calculating-the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value for the same image frame Represents a difference value between the value of the third color channel and the value of the third color channel-the first difference value for at least one image frame included in the first image frame group with respect to the first image frame group acquired during a first preset time And obtaining a first characteristic based on an average value of the first difference value for the first image frame group, a second difference value for at least one image frame included in the first image frame group, and the Obtaining a second characteristic value based on the average value of the second difference value for the first image frame group, and determining a biometric index for the subject based on the first characteristic value and the second characteristic value. Including a step, wherein the first color channel value is an average pixel value of a first color channel for one image frame, and the second color channel value is an average pixel value of a second color channel for one image frame. , The third color channel value may be provided with a method of measuring a biometric index indicating an average pixel value of a third color channel for one image frame.

Here, the biometric index may include at least one of heart rate and blood pressure.

Here, the first, second, and third color channels may be color channels according to an RGB color space.

Here, in consideration of absorbance of hemoglobin and oxyhemoglobin, in order to reduce noise, the first color channel is set to a green channel, the second color channel is set to a red channel, and the third color channel is a blue channel. Can be set to

Here, the first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group, and the second characteristic value is the first image It is obtained based on a second deviation value of the second difference value for at least one image frame included in a frame group, and the first deviation value is a first difference value and the first difference value for the at least one image frame. It is calculated based on an average value of the first difference value for a group of image frames, and the second deviation value is a second difference value for the at least one image frame and the second difference for the first image frame group It can be calculated based on the average value of the values.

Here, the first characteristic value and the second characteristic value may be normalized values.

Here, the first characteristic value is a value normalized by a first standard deviation value, the second characteristic value is a value normalized by a second standard deviation value, and the first standard deviation value is the first image A standard deviation value of the first difference value for a frame group, and the second standard deviation value may be a standard deviation value of the second difference value for the first image frame group.

Here, the biometric index for the subject may be determined based on a third characteristic value obtained as a sum of the first characteristic value and the second characteristic value.

Here, further comprising the step of outputting a biometric index, wherein the determined biometric index includes a first biometric index and a second biometric index, and the output biometric index is the first biometric index and the second biometric index. It can be determined on the basis of the index.

Here, the first biometric index is determined based on a second image frame group, the second biometric index is determined based on a third image frame group, and the number of image frames included in the first image frame group is the It is smaller than the number of image frames included in the second and third image frame groups, and the first image frame group may be included in the second image frame group.

Here, the number of image frames included in the second image frame group may be the same as the number of image frames included in the third image frame group.

Here, further comprising the step of outputting a biometric index based on the determined biometric index, wherein the determined biometric index includes at least four preliminary biometric indexes, and the output biometric index is the four preliminary biometric indexes It can be determined on the basis of.

Here, further comprising the step of outputting a biometric index based on the determined biometric index, wherein the output biometric index includes a first biometric index and a second biometric index, and the second biometric index is the second biometric index. 1 is a biometric index of the same type as the biometric index, the second biometric index is output after the first biometric index is output, and the difference between the second biometric index and the first biometric index exceeds a reference value The second biometric index may be corrected and output.

According to another embodiment, a method of measuring a biometric index in a non-contact manner using an infrared camera, the step of obtaining a plurality of image frames for a subject using an infrared camera, at least one included in the plurality of image frames Obtaining a first region value, a second region value, and a third region value for an image frame of, the first region value and the second region value for at least one image frame included in the plurality of image frames And calculating a first difference value and a second difference value based on the third area value, at least one included in the first image frame group with respect to the first image frame group acquired during a first preset time. Obtaining a first characteristic value based on the first difference value for the image frame and the average value of the first difference value for the first image frame group, at least one included in the first image frame group Obtaining a second characteristic value based on a second difference value for an image frame and an average value of the second difference value for the first image frame group, and based on the first characteristic value and the second characteristic value Determining a biometric index for the subject, wherein the first region value is an average pixel value for a first region of interest in one image frame, and the second region value is a second region value in one image frame. An average pixel value for a region of interest, the third region value is an average pixel value for a third region of interest in one image frame, and the first difference value is a first region value and a second region for the same image frame. A method for measuring a biometric index may be provided, which is a difference value of values, and the second difference value is a difference value between a first area value and a third area value for the same image frame.

According to another embodiment, a biometric index measurement device for non-contact measurement of a biometric index, comprising: an image acquisition unit for obtaining an image frame for the subject and a control unit for obtaining a biometric index using the image frame Including, wherein the control unit obtains a first color channel value, a second color channel value, and a third color channel value for at least one image frame included in the plurality of obtained image frames, and A first difference value and a second difference value are calculated based on the first color channel value, the second color channel value, and the third color channel value for at least one included image frame, and a first preset time Based on the average value of the first difference value for at least one image frame included in the first image frame group and the first difference value for the first image frame group with respect to the first image frame group obtained during To obtain a first characteristic value, and based on the average value of the second difference value for at least one image frame included in the first image frame group and the second difference value for the first image frame group. 2 A characteristic value is obtained and a biometric index for the subject is determined based on the first characteristic value and the second characteristic value, and the first color channel value is an average pixel for the first color channel in one image frame. Value, the second color channel value is an average pixel value for a second color channel in one image frame, the third color channel value is an average pixel value for a third color channel in one image frame, and the The first difference value is a difference value between a first color channel value and a second color channel value for the same image frame, and the second difference value is a difference between a first color channel value and a third color channel value for the same image frame. A biometric index measuring device that is a value may be provided.

Here, the first, second, and third color channels are color channels according to an RGB color space, and in order to reduce noise, in consideration of absorbance of hemoglobin and oxygen hemoglobin, the first color channel is set to a green channel. , The second color channel may be set as a red channel, and the third color channel may be set as a blue channel.

Here, the controller acquires biometric information based on the determined biometric index, and the biometric information may include at least one of emotion information, sleepiness information, stress information, and excitement level information.

According to another embodiment, a method of measuring a biometric index in a non-contact manner, comprising: acquiring a plurality of image frames for a subject, a first color channel for at least one image frame included in the plurality of image frames Acquiring a value, a second color channel value, and a third color channel value, the first, second, and third image frames included in the first image frame group acquired during a first preset time Acquiring a first characteristic value based on a color channel value, the first, second, and third color channel values for at least one image frame included in a second image frame group acquired for a second preset time Obtaining a second characteristic value based on and determining a biometric index based on the first characteristic value and the second characteristic value, wherein the first image frame group and the second image frame group are At least partially overlapped, and in order to determine the biometric index, a first characteristic value for a first image frame included in the first image frame group and not included in the second image frame group is used, and the first The first characteristic and the second characteristic value of the image frame group and the second image frame included in the second image frame group are used, and are included in the second image frame group, but not included in the first image frame group. A method of measuring a biometric index using a second characteristic value for a third image frame that is not used may be provided.

Here, calculating a first difference value based on at least a portion of the first color channel value, the second color channel value, and the third color channel value for at least one image frame included in the plurality of image frames. Including the step, wherein the first characteristic value is an average value of the first difference value for at least one image frame included in the first image frame group and the first difference value for the first image frame group. It is obtained on a basis, and the second characteristic value is an average value of the first difference value for at least one image frame included in the second image frame group and the second difference value for the second image frame group. The first difference value is obtained based on a difference value between a first color channel value and a second color channel value for the same image frame, and the second difference value is a first color channel value and a second color channel value for the same image frame. It may be a difference value between three color channel values.

Here, the first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group, and the second characteristic value is the second image It is obtained based on a second deviation value of the first difference value for at least one image frame included in the frame group, and the first deviation value is for at least one image frame included in the first image frame group. It is calculated based on a first difference value and an average value of the first difference value for the first image frame group, and the second deviation value is for at least one image frame included in the second image frame group. It may be calculated based on a first difference value and an average value of the first difference value for the second image frame group.

According to another embodiment, a biometric index output method for obtaining and outputting a biometric index in a non-contact manner, comprising: a first image frame group and a second image frame group at least partially overlapping with the first image frame group. Acquiring an image frame, obtaining a biometric index based on the first image frame group, outputting a biometric index at a first time point, obtaining a first biometric index based on the second image frame group Step, comprising the step of outputting a biometric index at a second time point later than the first time point, wherein the biometric index output at the first time point is a biometric index obtained based on the first image frame group, and the second The biometric index output at a time point is the first biometric index when the difference between the first biometric index and the biometric index output at the first time point is less than or equal to a reference value, and the first biometric index and the biometric index output at the first time point When the difference between the index exceeds the reference value, a method for outputting a biometric index, which is a biometric index corrected from the biometric index output at the first time point, may be provided.

Here, the corrected biometric index is a biometric index corrected by adding a preset value to the biometric index output at the first time point when the first biometric index is greater than the biometric index output at the first time point, and the first When the biometric index is smaller than the biometric index output at the first time point, the biometric index may be corrected by subtracting a preset value from the biometric index output at the first time point.

According to an embodiment, as a method of simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, acquiring a plurality of image frames for a subject, included in the plurality of image frames Setting a first area and a second area for at least one image frame, at least two color channel values of a first color channel value, a second color channel value, and a third color channel value for the first area Determining an oxygen saturation degree of the subject based on a first characteristic obtained based on a first characteristic, based on a first difference value that is a difference value between the first color channel value and the second color channel value for the first region Determining a heart rate of the subject based on the acquired second characteristic, a second difference value and the first difference value that are difference values between the first color channel value and the second color channel value for the second area Determining a blood pressure of the subject based on a third characteristic acquired based on the measurement, outputting the oxygen saturation, heart rate, and blood pressure; Including, wherein the first feature is acquired based on a first image frame group acquired during a first time, the second feature is acquired based on a second image frame group acquired during a second time, and the second feature The 3 features are obtained based on the third image frame group acquired during the third time period, and the first image frame group, the second image frame group, and the third image frame are obtained so that the oxygen saturation, heart rate, and blood pressure are acquired in association with each other. The group may provide a method of measuring a biometric index including a plurality of image frames in common.

Here, the first color channel value may be a green channel value, the second color channel value may be a red channel value, and the third color channel value may be a blue channel value.

Here, the first characteristic is that the second color channel value for the second color channel, which is a channel in which the absorbance of oxygen hemoglobin is lower than the absorbance of hemoglobin, and the third color channel in which the absorbance of oxygen hemoglobin is higher than that of hemoglobin. It can be obtained based on the three color channel value.

Here, the second characteristic may be obtained based on a third difference value and the first difference value, which is a difference value between the first color channel value and the second color channel value for the first area.

Here, to reduce noise caused by external light, the first color channel value may be a green channel value, the second color channel value may be a red channel value, and the third color channel value may be a blue channel value.

Here, the second characteristic may include a frequency component value of time series data obtained based on the first difference value.

Here, the third characteristic may include a pulse transit time (PTT) obtained based on the second difference value and the first difference value.

Here, the fourth characteristic is a fourth difference value, the first difference value, the second difference value, and the third difference, which is a difference value between the first color channel value and the third color channel value for the second area. It can be obtained based on the value.

Here, the first region and the second region include a face region of the subject, and a center of the second region and a vertical position of the center of the first region may be different from each other.

According to another embodiment, as a method of simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, obtaining a plurality of image frames for a subject, including in the plurality of image frames Setting a first region, a second region, and a third region for at least one image frame that is configured to, at least two of a first color channel value, a second color channel value, and a third color channel value for the first region Determining an oxygen saturation degree of the subject based on a first characteristic obtained based on the above color channel values, a first difference value between the first color channel value and the second color channel value for the first area Determining a heart rate of the subject based on a second characteristic obtained based on a difference value, a second difference value that is a difference value between the first color channel value and the second color channel value for the second region, and Determining the blood pressure of the subject based on a third characteristic obtained based on a third difference value that is a difference value between the first color channel value and the second color channel value for the third area, and the oxygen saturation degree , Outputting a heart rate and blood pressure, wherein the first feature is obtained based on a first image frame group acquired during a first time, and the second feature is a second image frame group acquired during a second time. Is obtained based on, and the third feature is obtained based on a third image frame group acquired during a third time, and the first image frame group and the second are obtained so that the oxygen saturation, heart rate, and blood pressure are obtained in association with each other. An image frame group and a third image frame group may provide a biometric index measurement method including a plurality of image frames in common.

According to another embodiment, as a method of simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, obtaining a plurality of image frames for a subject, the plurality of image frames Acquiring a first color channel value, a second color channel value, and a third color channel value for at least one included image frame, among the first color channel value, the second color channel value, and the third color channel value Determining an oxygen saturation degree of the subject based on a first characteristic obtained based on at least two color channel values, a first difference value that is a difference value between the first color channel value and the second color channel value, and the Determining a heart rate of the subject based on a second characteristic obtained based on a second difference value that is a difference value between the first color channel value and the third color channel value, the first difference value and the second difference Determining a blood pressure of the subject based on a third characteristic obtained based on a value, and outputting the oxygen saturation, heart rate, and blood pressure, wherein the first characteristic is a first image acquired during a first time Acquired based on a frame group, the second feature is acquired based on a second image frame group acquired during a second time, and the third feature is acquired based on a third image frame group acquired during a third time The first image frame group, the second image frame group, and the third image frame group are provided with a biometric index measurement method including a plurality of image frames in common so that the oxygen saturation degree, heart rate, and blood pressure are obtained in association with each other. I can.

Here, the second characteristic may include a frequency component value of time series data obtained based on the first difference value and the second difference value.

Here, the third feature is a gradient component value, a maximum value, a minimum value, a maximum value, a minimum value, an average value of the local maximum, and an average value of the minimum value of the time series data obtained based on the first difference value and the second difference value. , At least one of a difference value and an average value for the maximum and minimum value average may be included.

According to another embodiment, as a method of simultaneously measuring various physiological parameters including heart rate, oxygen saturation, blood pressure, and body temperature, acquiring a plurality of image frames for a subject, the plurality of images Setting at least two or more regions for at least one image frame included in the frame, a first color channel value, a second color channel value, and a third color for at least one image frame included in the plurality of image frames Obtaining a channel value, a fourth color channel value, and a fifth color channel value, the first color channel value, a second color channel value, and a third color channel value obtained based on at least two color channel values of the third color channel value. 1 determining an oxygen saturation degree of the subject based on a feature, based on a second feature obtained based on at least two color channel values of the first color channel value, the second color channel value, and the third color channel value Determining the heart rate of the subject based on the measurement, the blood based on a third characteristic obtained based on at least two color channel values of the first color channel value, the second color channel value, and the third color channel value. Determining a blood pressure of a measurer, determining a body temperature of the subject based on a fourth characteristic acquired based on the fourth color channel value, and outputting the oxygen saturation, heart rate, body temperature, and blood pressure However, the first color channel value is a green channel value, the second color channel value is a red channel value, the third color channel value is a blue channel value, and the fourth color channel value is a saturation channel value, The fifth color channel value may be a method of measuring a biometric index, which is a Hue channel value.

Here, the first characteristic is that the second color channel value for the second color channel, which is a channel in which the absorbance of oxygen hemoglobin is lower than that of hemoglobin, and the third color channel for the third color channel in which the absorbance of oxygen hemoglobin is higher than that of hemoglobin. It can be obtained based on the three color channel value.

Here, the second characteristic is based on a first difference value that is a difference value between the first color channel value and the second color channel value, and a second difference value that is a difference value between the first color channel value and the third color channel value. It can be obtained by

Here, the third characteristic is based on a first difference value that is a difference value between the first color channel value and the second color channel value, and a second difference value that is a difference value between the first color channel value and the third color channel value. It can be obtained by

Here, the fourth characteristic may be obtained based on the fourth color channel value and the fifth color channel value.

Here, the fourth characteristic may include the skin temperature of the subject.

According to another embodiment, as a method of simultaneously measuring various physiological parameters including heart rate, oxygen saturation and blood pressure, obtaining a plurality of image frames for a subject, the plurality of image frames Acquiring N preliminary heart rates based on at least one included image frame, obtaining M preliminary oxygen saturation based on at least one image frame included in the plurality of image frames, the plurality of image frames Obtaining K preliminary blood pressure based on at least one image frame included in, obtaining a heart rate based on the N preliminary heart rates, obtaining oxygen saturation based on the M preliminary oxygen saturation, the Acquiring blood pressure based on K preliminary blood pressures and outputting the heart rate, oxygen saturation, and blood pressure, wherein the image frame obtained at the time point in which the subject is in the first state is the first image frame, The first image frame may be provided with a method of measuring a biomarker commonly included in an image frame for obtaining the N preliminary heart rates, the M preliminary oxygen saturation and the K blood pressure.

According to another embodiment, a method of acquiring biometric information using various physiological parameters, the step of acquiring a plurality of image frames for a subject, at least one included in the plurality of image frames Obtaining a first biometric index based on a first image frame group including an image frame, a second biometric index based on a second image frame group including at least one image frame included in the plurality of image frames Obtaining, obtaining biometric information based on at least one of the first biometric index and the second biometric index, and outputting the first biometric index, the second biometric index, and the biometric information, , In order to obtain biometric information by reflecting a specific state of the subject, a method for obtaining biometric information in which the first image frame group and the second image frame group overlap at least partially may be provided.

Here, further comprising the step of obtaining personal and statistical data on the subject, wherein the biometric information may be obtained based on the first biometric index, the second biometric index, and the obtained personal and statistical data. .

Here, the first and second biomarkers include at least one of heart rate, oxygen saturation, blood pressure, and body temperature, and the biometric information may include at least one of drowsiness information, stress information, excitement level information, and emotion information. .

The method of detecting drowsiness based on a heart rate according to an embodiment may be performed by at least one processor, obtaining a heart rate of an object, comparing the heart rate with a reference heart rate to obtain a comparison result, the Comparing a heart rate and a reference heart rate to obtain a comparison result, obtaining a duration for which the heart rate is less than or equal to the reference heart rate based on the comparison result, and whether the duration reaches a reference duration Based on the drowsiness detection step of determining the drowsy state of the subject. The reference duration includes a first reference duration, a second reference duration, and a third reference duration, and the second reference duration is It is longer than the first reference duration, the third reference duration is longer than the second reference duration, and the drowsy state of the subject includes a normal state, a first drowsiness state, a second drowsiness state, and a third drowsiness state. And, the first drowsiness state represents a state in which the object is more likely to enter the sleep state than the normal state, and the second drowsiness state is more likely to enter the sleep state than the first drowsiness state. A state, and the third drowsiness state may represent a state in which the object is more likely to enter the sleep state than the second drowsiness state.

Here, the drowsiness detecting step, when the duration is longer than the first reference duration and shorter than the second reference duration, determining that the drowsy state of the subject is the first drowsy state, the continuation When the time is longer than the second reference duration and shorter than the third reference duration, determining that the drowsiness state of the subject is the second drowsiness state, and if the duration is longer than the third reference duration, the drowsiness It may include determining that the state is the second drowsy state.

Here, the drowsiness detection step includes determining a state related to the drowsiness of the subject as the normal state when the duration of the sustained period in which the heart rate of the subject is equal to or less than the reference heart rate is shorter than the first reference duration. It may contain more.

Here, the first drowsiness state may represent a state in which the subject is not aware of drowsiness, but may physically enter a sleep state.

Here, the heart rate of the object may represent an average value of a plurality of heart rates in a predetermined time interval including a time point at which the heart rate is acquired.

Here, when the drowsiness state of the subject is any one of the first drowsiness state to the third drowsiness state, acquiring a recovery duration that continues in a state in which the heart rate is equal to or greater than the reference heart rate, the recovery duration is It may further include a recovery detection step of determining the drowsiness state of the subject based on whether the recovery reference duration is reached.

Here, the recovery reference duration includes a first recovery reference duration, a second recovery reference duration, and a third recovery reference duration, and in the recovery detection step, the drowsy state of the subject is a third drowsy state. If the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the drowsy state of the subject is determined as the second drowsiness state, and the recovery duration is the second recovery If longer than the reference duration and shorter than the third recovery reference duration, the drowsy state of the subject is determined as the first drowsy state, and if the recovery duration is longer than the third recovery reference duration, the drowsiness of the subject If the state is determined as the normal state, and the drowsy state of the subject is a second drowsy state, the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the If the drowsiness state is determined as the first drowsiness state, and the recovery duration is longer than the second recovery standard duration and shorter than the third recovery standard duration, the drowsiness state of the subject is determined as the normal state, and the When the drowsy state of the subject is the first drowsy state, and the recovery duration is longer than the first recovery reference duration and shorter than the second recovery reference duration, the drowsy state of the subject may be determined as the normal state. .

Here, one of the first reference duration, the second reference duration, and the third reference duration is at least one of the first recovery reference duration, the second recovery reference duration, and the third recovery reference duration. Can be the same as

Here, the first reference duration, the second reference duration, and the third reference duration may be different from all of the first recovery reference duration, the second recovery reference duration, and the third recovery reference duration. have.

A method of detecting drowsiness based on a heart rate and LF/HF performed by at least one processor according to an embodiment comprises: acquiring a heart rate for an object, comparing the heart rate and a reference heart rate to obtain a comparison result. Steps, based on the comparison result, acquiring a duration of the heart rate in a changed state, acquiring a first drowsiness parameter (a drowsiness step determined based on the heart rate) based on the duration, the Acquiring the LF/HF (Low Frequency/High Frequency) value of the subject indicating the ratio of the sympathetic nerve activity and the parasympathetic nerve activity of the subject, obtaining a second drowsiness parameter based on the LF/HF value of the subject And a drowsiness detection step of determining a drowsiness state of the object by using at least one of the first drowsiness parameter and the second drowsiness parameter.

Here, in the obtaining of the duration, the duration may be obtained based on a length of a time interval during which the heart rate is less than or equal to the reference heart rate based on the comparison result.

Here, the drowsiness state includes a first drowsiness state and a second drowsiness state, and the second drowsiness state is a state in which the object is more likely to enter a sleep state than the first drowsiness state, and the drowsiness state is the The first drowsiness parameter obtained based on the comparison result of the duration, the first reference duration, and the second reference duration, and the result of comparing the LF/HF of the subject with the first reference value and the second reference value. It is determined according to a second drowsiness parameter, and the drowsiness state is determined as the first drowsiness state when the duration is greater than the first reference duration and LF/HF of the subject is less than the first reference value, and the drowsiness The state may be determined as the second sleepiness state when the duration is greater than the second reference interval or the LF/HF of the object is smaller than the second reference value.

Here, the drowsiness state further includes a third drowsiness state, and the first drowsiness state is a state in which the subject is not aware of drowsiness, but is a state in which there is a possibility of physically entering a sleep state, and the second drowsiness state and the second drowsiness state 3 The drowsiness state is a state in which drowsiness is aware and there is a possibility to enter a sleep state physically, and the third drowsiness state may be a state in which the possibility of entering a sleep state is higher than that of the second drowsiness state.

Here, the drowsiness state includes a normal state, a first phase drowsiness state, a second phase drowsiness state, and a third phase drowsiness state, and the first phase drowsiness state is the likelihood that the subject enters a sleep state rather than the normal state. This indicates a high state, and the second-stage drowsiness state indicates a state in which the object is more likely to enter the sleep state than the first-stage drowsiness state, and the third-stage drowsiness state is more than the second-stage drowsiness state. It represents a state in which the object is likely to enter the sleep state, and the higher the value of the first drowsiness parameter and the second drowsiness parameter is, the higher the probability that the object enters the sleep state is represented, and the first When the value of the drowsiness parameter and the value of the second drowsiness parameter match, a matching value is obtained, and the drowsiness state is determined as a normal state, a first step drowsiness state, a second step drowsiness state, and a second sleep state based on the matched value. It can be determined as one of the three levels of drowsiness.

Here, the drowsiness state includes a normal state, a first phase drowsiness state, a second phase drowsiness state, and a third phase drowsiness state, and the first phase drowsiness state is the likelihood that the subject enters a sleep state rather than the normal state. This indicates a high state, and the second-stage drowsiness state indicates a state in which the object is more likely to enter the sleep state than the first-stage drowsiness state, and the third-stage drowsiness state is more than the second-stage drowsiness state. It represents a state in which the object is likely to enter the sleep state, and the higher the value of the first drowsiness parameter and the second drowsiness parameter is, the higher the probability that the object enters the sleep state is represented, and the first If the value of the drowsiness parameter and the value of the second drowsiness parameter do not coincide, a larger value of the value of the first drowsiness parameter and the value of the second drowsiness parameter is obtained, and the drowsiness is based on the larger value. The state may be determined as one of a normal state, a first-stage drowsiness state, a second-stage drowsiness state, and a third-stage drowsiness state.

Here, the drowsiness state includes a normal state, a first phase drowsiness state, a second phase drowsiness state, and a third phase drowsiness state, and the first phase drowsiness state is the likelihood that the subject enters a sleep state rather than the normal state. This indicates a high state, and the second-stage drowsiness state indicates a state in which the object is more likely to enter the sleep state than the first-stage drowsiness state, and the third-stage drowsiness state is more than the second-stage drowsiness state. It represents a state in which the object is likely to enter the sleep state, and the higher the value of the first drowsiness parameter and the second drowsiness parameter is, the higher the probability that the object enters the sleep state is represented, and the first When the value of the drowsiness parameter and the value of the second drowsiness parameter do not coincide, a smaller value of the value of the first drowsiness parameter and the value of the second drowsiness parameter is obtained, and the drowsiness is based on the smaller value. The state may be determined as one of a normal state, a first-stage drowsiness state, a second-stage drowsiness state, and a third-stage drowsiness state.

Here, the drowsiness state includes a normal state, a first phase drowsiness state, a second phase drowsiness state, and a third phase drowsiness state, and the first phase drowsiness state is the likelihood that the subject enters a sleep state rather than the normal state. This indicates a high state, and the second-stage drowsiness state indicates a state in which the object is more likely to enter the sleep state than the first-stage drowsiness state, and the third-stage drowsiness state is more than the second-stage drowsiness state. It represents a state in which the object is likely to enter the sleep state, and the higher the value of the first drowsiness parameter and the second drowsiness parameter is, the higher the probability that the object enters the sleep state is represented, and the first If the value of the drowsiness parameter and the value of the second drowsiness parameter do not coincide, the average value of the value of the first drowsiness parameter and the value of the second drowsiness parameter is obtained, and the drowsiness state is determined as a normal state based on the average value. , It can be determined as one of the first stage drowsiness state, the second stage drowsiness state, and the third stage drowsiness state.

Here, the method may be performed through a recording medium on which a program performing the method is recorded.

According to another embodiment, as a smart mirror device that simultaneously displays at least two biomarkers of heart rate, oxygen saturation, blood pressure, and body temperature, a reflective mirror surface and a plurality of image frames for a subject are acquired. An image acquisition unit disposed behind the reflective mirror surface, and a display unit that displays visual information through the reflective mirror surface, and a control unit for controlling the operation of the image acquisition unit and the display unit and obtaining a biometric index in a non-contact manner Including, wherein the control unit controls the display unit to display a first biometric index obtained based on a first image frame group included in the plurality of image frames at a first time point, and at a second time point, the plurality of images The display unit controls the display to display a second biometric index obtained based on a second image frame group included in the frame, and the first image frame group and The second image frame group includes at least one image frame in common, and the at least one image frame commonly included in the first image frame group and the second image frame group is before the first and second viewpoints A smart mirror device including an image frame obtained in a first state of the subject observed by the subject through the reflective mirror surface may be provided.

Here, the first and second biometric indexes may include at least one biometric index among heart rate, oxygen saturation, blood pressure, and body temperature.

Here, the first image frame group and the second image frame group may be the same.

Here, the first image frame group and the second image frame group may be different from each other.

Here, the first viewpoint and the second viewpoint may be the same viewpoint.

Here, the first viewpoint is a viewpoint prior to the second viewpoint, and the at least one image frame commonly included in the first image frame group and the second image frame group is a surface of the reflective mirror before the first viewpoint. The image frame obtained in the second state of the subject observed by the subject may be included.

Here, the control unit controls the display unit to display a third biometric index obtained based on a third image frame group included in the plurality of image frames at a third time point, and The first, second, and third image frame groups include at least one image frame in common, and at least one commonly included in the first, second, and third image frame groups so that association is given to the index. The image frame may include an image frame obtained in the first state of the subject observed by the subject through the reflective mirror surface before the first, second and third viewpoints.

Here, the controller obtains the first and second biometric indexes based on at least some of the plurality of image frames during a first time, and at least one image frame of the plurality of image frames during a second time Based on the recognition of the subject, the second time is shorter than the first time, the second time is included in the first time, the recognized before the first and second biometric indexes are displayed Information about the subject can be displayed.

Here, the information on the subject may be the first and second biometric indexes of the subject previously measured and stored.

Here, the control unit controls the display unit to display first information at a fourth time point, and controls the display to display second information at a fifth time point, wherein the first information includes weather information and time information, , The second information may include information on the subject, the fourth time point may be a time point prior to the fifth time point, and the fifth time point may be a time point prior to the first and second time points.

Here, the second information may include at least one of schedule information, medication information, recognition information, messenger information, and interest information of the subject.

Here, the control unit controls the display unit to display the first information at the fourth time point, and controls the display unit to display the first information and the second information at the fifth time point, and at the second time point The display unit may be controlled to display the first information, the second information, the first biometric index, and the second biometric index.

Here, operation to obtain the first biometric index and the second biometric index based on at least three color channel values for the plurality of image frames, and obtain a first image frame included in the first image frame group It is possible to operate to display both the first biometric index and the second biometric index within 10 seconds from a point in time.

Here, when the image frame obtained through the image acquisition unit includes images of a plurality of people, the control unit is based on the first and second images of one of the plurality of people selected according to priority. The display unit may be controlled to obtain a biometric index, display the obtained first and second biometric indexes, and display information on a selected person among the plurality of persons.

Here, when the image frame including the first subject and the second subject is obtained after the image frame including the first subject is acquired, the controller is The display unit may be controlled to obtain the first and second biometric indexes, and to display information on the first subject.

Here, the controller may determine the priority of the plurality of people based on the acquired image frame.

Here, when the image frame obtained through the image acquisition unit includes images of the first subject and the second subject, the control unit includes the first image frame group based on the fourth image frame group included in the plurality of image frames. Obtaining the first biometric index for the subject, obtaining the second biometric index for the first subject based on a fifth image frame group included in the plurality of image frames, and the plurality of image frames Acquiring the first biometric index for the second subject based on a sixth image frame group included in, and for the second subject based on a seventh image frame group included in the plurality of image frames. And the fourth and fifth image frame groups include at least one image frame in common, and the sixth and seventh image frame groups include at least one image frame in common. Can include.

Here, the control unit obtains biometric information based on the first biometric index, and controls the display unit to display the biometric information at a third time point, and displays the first biometric index and the biometric information in real time. The biometric information displayed at the third time point is obtained based on the first biometric index displayed at the first time point, and the first time point may be a time point prior to the third time point.

Here, the control unit acquires the biometric information based on the first and second biometric indexes, and the biometric information displayed at the third time point is displayed at the first time point in order to improve the accuracy of the biometric information. It is obtained based on the first biometric index and the second biometric index displayed at the second time point, and the third time point may be a time later than the first and second time points.

Here, the biometric information may include at least one biometric information of condition information, concentration information, drowsiness information, and emotion information.

Here, the controller obtains biometric information based on the first biometric index, and controls the display unit to display the biometric information at a third time point, and the display unit is displayed at the third time point in order to display accurate biometric information. The biometric information is obtained based on the first biometric index displayed up to the third time point, and the third time point may be a time later than the first time point.

Here, the biometric information displayed at the third time point may be obtained based on an average value of the first biometric index displayed up to the third time point.

Here, the control unit acquires the biometric information based on the first and second biometric indexes, and the biometric information displayed at the third time point is the first biometric index and the third time displayed up to the third time point. It is obtained based on the second biometric index displayed up to the time point, and the third time point may be a time point later than the first and second time points.

Here, a period in which the first bio-index is updated and a period in which the second bio-index is updated may be different from each other.

Here, the control unit determines the information provision situation, controls the display unit to display first information in case of a first situation, and controls the display unit to display second information in case of a second situation, wherein the first The information may include first and second biometric indexes, but the second information may not include the first and second biometric indexes.

Here, the controller may determine the information provision situation based on the moving direction of the subject.

Here, the smart mirror device may further include a motion detection sensor for detecting the moving direction of the subject.

Here, the controller may determine the information provision situation based on the acquired plurality of image frames.

Here, the first information includes at least one of external weather information, schedule information of the subject, and time information, and the second information is internal temperature information, internal humidity information, internal air information, security information, and activity It may include at least one piece of time information.

Here, the first information may include at least two biometric indexes, and the second information may include at least one of weather information, time information, news information, schedule information, and medication information.

Here, the controller obtains at least three color channel values for at least one image frame included in the plurality of image frames, and calculates a first difference value and a second difference value based on the at least three color channel values. And obtain the first biometric index and the second biometric index based on the first difference value and the second difference value.

According to another embodiment, as a method of operating a smart mirror device that simultaneously displays at least two biomarkers of heart rate, oxygen saturation, blood pressure, and body temperature, obtaining an on-trigger , Obtaining a plurality of image frames for a subject, obtaining a first biometric index based on a first image frame group included in the plurality of image frames, a second image included in the plurality of image frames Acquiring a second biometric index based on a frame group, displaying the first and second biometric index, obtaining an off-trigger, and obtaining a plurality of image frames for the subject A step of stopping, wherein the first image frame group and the second image frame group commonly include at least one image frame so that a correlation is given to the first biometric index and the second biometric index, and the on A trigger (on-trigger) is obtained from at least one sensor, the off-trigger (off-trigger) is a method of operating a smart mirror obtained from an image sensor for acquiring the plurality of image frames may be provided.

Here, the on-trigger may be obtained from at least one of a motion detection sensor, a touch sensor, a mouse, and a keyboard.

According to another embodiment, as a smart mirror device for displaying at least one biometric index among heart rate, oxygen saturation, blood pressure, and body temperature, a reflective mirror surface and an image acquisition unit for acquiring a plurality of image frames , A display unit disposed behind the reflective mirror surface and passing through the reflective mirror surface to display visual information, an opening and closing unit disposed in front of the image acquisition unit and for opening and closing the view of the image acquisition unit, and the image acquisition unit and the display A control unit for controlling a negative operation and obtaining a biometric index; Including, wherein the surface of the opening and closing portion is formed with a reflective mirror, and when the opening and closing portion is in an open state, the control unit controls the display unit so that the biometric index and at least one visual information are displayed through the display unit, and the opening and closing unit is closed. In the case of the state, the controller may provide a smart mirror device that controls the display unit so that at least one or more visual information is displayed through the display unit.

Here, the control unit controls the image acquisition unit to acquire an image frame from the image acquisition unit when the opening and closing unit is in the open state from the closed state, and when the opening and closing unit is in the closed state from the open state, the image is acquired. The image acquisition unit may be controlled so that the unit does not acquire an image frame.

0. Definition of terms

The term'measurement' used in the present specification may be understood as a concept including all of measuring by measuring, determining by guessing, and measuring the size of other quantities based on a predetermined quantity.

'Heart rate' as used herein can be understood as a heart rate, which is a result of the heart rate and is a result of the concept of'heart rate' and heart rate, which can mean the number of beats measured near the heart. It can be understood as a concept including all the concept of'pulse' which can mean that the generated vibration has propagated to the peripheral blood vessels.

'Blood pressure' as used herein can be understood as a pressure generated in blood vessels when blood is pushed out of the heart, and this is a value that can be understood as a normal'blood pressure' regardless of the measurement site (e.g. For example, it can be understood as a value that can be inferred as a value measured in the artery of the upper arm).

The'oxygen saturation' used herein may be understood as the degree of saturation of oxygen in the blood, and more specifically, it may be understood as the fraction of oxygen hemoglobin to total hemoglobin in the blood.

'Core Temperature' used in the present specification may be understood as a body temperature of a person or an animal, and may be understood differently from'Skin Temperature' that can be measured on the skin.

'Skin Temperature' as used herein may be understood as the surface temperature of the skin to be measured.

The'image' used in the present specification may be understood as a concept including all of a single image or a plurality of images included in an image.

The'color channel' used in this specification can be understood as each axis constituting a color space. For example, a red channel, a green channel, and a blue channel constitute an RGB color space. It can mean a red axis, a green axis, and a blue axis. These color channels can be composed of two dimensions, three dimensions, or four dimensions.

The'image of the subject' used in the present specification may be understood as an image including the position of the subject to be measured. For example, when the measurement position is the subject's face, the face area of the subject is It can be understood as a containing image.

'Personal, statistical data' as used herein may mean collectable personal and statistical data of a subject, such as age, sex, height, and weight, and observable personal and statistical data such as facial expressions, wrinkles, and face color of the subject. It may mean data, and statistical data calculated for a group including or related to the subject (e.g., average blood pressure in their 20s, average skin color in yellow, average height of men in their 30s, and average weight of men in Korea) It can also mean personal and statistical data that can be quantified, such as).

'Time series data' as used herein may mean data listed along a time axis, but is not limited thereto and may mean data listed along an image frame axis that may correspond to time, and may mean a time axis or an image frame axis. It may mean data that can be sorted according to, and may also be understood as'time series data'.

1. Biometric index and biometric information management system

"Physiological parameter" may mean a result of the physiological activity of the human body that can be measured or estimated, and may include, for example, heart rate, oxygen saturation, blood pressure, body temperature, blood flow, etc. It is not limited and may mean other results according to the physiological activity of the human body that can be measured or estimated.

"Physiological information" may mean information that can be calculated by considering at least some of personal and statistical data such as a result of physiological activity of the human body such as a biometric index and facial expression, posture, and age. For example, , Drowsiness information, stress information, excitement level information, emotion information, and the like, but are not limited thereto.

The biometric index and biometric information management system may mean a management system capable of comprehensively managing the health of individuals using the biometric index and biometric information that can be measured or estimated and stored, shared, or analyzed. have.

1 and 2 are diagrams of a biometric index and biometric information management system according to an embodiment, respectively.

Referring to FIG. 1, a biometric index and biometric information management system 100 according to an exemplary embodiment may include a biometric index acquisition device 10 and a server 20.

In this case, the biometric index acquisition device 10 may measure a physiological parameter of a subject. More specifically, the biometric index acquisition device 10 may invasively measure the biometric index of the subject, non-chip wet but contact method, or non-contact method. have.

For example, the biometric index acquisition device 10 may measure a biometric index such as heart rate, oxygen saturation, and blood pressure of the subject by analyzing an image or image of the subject, but is not limited thereto.

*In addition, the biometric index acquisition device 10 may calculate physiological information based on the measured biometric index. More specifically, the biometric index acquisition device 10 may calculate biometric information based on the measured biometric index, and biometric information by comprehensively considering personal and statistical data such as the measured biometric index, facial expression, and posture age. Can also be calculated.

For example, the biometric index acquisition device 10 may calculate biometric information such as emotion information and sleepiness information of the subject based on the measured biometric index such as heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

In addition, the biometric index acquisition device 10 may store the measured biometric index and the calculated biometric information. For example, the biometric index acquisition device 10 may store the measured biometric index and calculated biometric information of the subject in an internal memory, but is not limited thereto.

In addition, the biometric index acquisition device 10 may display the measured biometric index and the calculated biometric information. For example, the biometric index acquisition device 10 may further include a display, and may display the measured biometric index and calculated biometric information for the subject by using the display, but is not limited thereto. , It is also possible to transmit the corresponding information to be displayed on an external display.

In addition, the biometric index may be displayed once through the display, and the biometric index that changes in real time may be continuously displayed.

In addition, the biometric index acquisition device 10 may transmit the biometric index and the biometric information for the subject to the server 20.

In this case, the biometric index and the biometric information transmitted to the server 20 may be stored in the server 20 as personal data. For example, the biometric index measured from the subject A and the calculated biometric information may be stored in the server 20 as data on the subject A, and the biometric index measured from the subject B and the calculated biometric information May be stored in the server 20 as data on the subject B.

In addition, the server 20 may transmit a biometric index and biometric information for a subject by communicating with an external terminal when necessary. For example, when a person named A, whose biometric index and biometric information data are stored in the server 20, visits a hospital to receive treatment, the doctor for treating the test subject A is the biometric index and the You may need biometric information. At this time, when the server 20 receives a request for transmission of biometric index and biometric information data for subject A from an external terminal disposed in the hospital, the server 20 communicates with the external terminal to provide the biometric index and biometric data for the subject A. Information data can be transmitted.

As such, the biometric index and biometric information management system 100 may serve as a basis for providing a continuous and comprehensive management service for an individual's health, and the biometric index and biometric information management system 100 as well as the above-described example It is obvious that) can serve as a basis for providing various comprehensive management services using biometric indexes and biometric information that are continuously measured, stored and managed.

Further, referring to FIG. 2, the biometric index and biometric information management system 100 according to an embodiment may include an image acquisition device 30 and a server 20.

In this case, the image acquisition device 30 may acquire an image or an image of the subject.

In addition, the server 20 may acquire an image or an image of the subject from the image acquisition device 30.

In addition, the server 20 may obtain personal and statistical data on a subject from an input device, an external terminal, or the like.

In addition, the server 20 may measure the biometric index of the subject based on the acquired image or image. For example, the server 20 may analyze the acquired image or image to measure biomarkers such as heart rate, oxygen saturation, and blood pressure of the subject, but is not limited thereto.

In addition, the server 20 may calculate biometric information based on the measured biometric index. More specifically, the server 20 may calculate biometric information based on the measured biometric index, and calculate biometric information by comprehensively considering personal and statistical data such as the measured biometric index and facial expression, posture, and age. can do.

For example, the server 20 may calculate biometric information such as emotion information and sleepiness information of the subject based on measured biometric indexes such as heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

In addition, the server 20 may store the measured biometric index and the calculated biometric information.

In addition, it is clear that the biometric index and biometric information management system 100 including the image acquisition device 30 and the server 20 can perform the functions of the biometric index and biometric information management system described above in FIG. 1. Therefore, the redundant description will be omitted.

2. Various embodiments of the biometric index acquisition device

Referring to FIG. 3, the apparatus 1000 for obtaining a biometric index according to an embodiment may include an image acquisition unit 1010, a control unit 1020, a storage unit 1030, and a communication unit 1040. However, the biometric index acquisition apparatus 1000 may include only at least some of the image acquisition unit 1010, the control unit 1020, the storage unit 1030, and the communication unit 1040. For example, the biometric index acquisition apparatus 1000 may include only the image acquisition unit 1010 and the control unit 1020, but is not limited thereto and may be implemented in various ways.

In addition, the image acquisition unit 1010 may acquire an image or an image of a subject. More specifically, the image acquisition unit 1010 may include a photographing device, and an image or image of the subject may be acquired using the photographing device, or the biometric index acquisition device 1000 may be placed outside. An image or an image of the subject may be obtained from the photographing device, but the present invention is not limited thereto.

In addition, when the image acquisition unit 1010 acquires an image or image of a subject from a photographing device, the photographing device includes a visible camera for obtaining a visible light image and an infrared camera for obtaining an infrared image. It may be provided as an (IR camera) or the like, but is not limited thereto, and a hybrid type camera for acquiring a visible light image and an infrared image may be provided.

In addition, when the photographing apparatus acquires a visible light image, the acquired visible light image may be acquired with at least one or more color channel values. For example, the obtained visible light image may be obtained as a color channel value of an RGB color space represented by red, green, and blue. It may be obtained as a color channel value of the HSV color space expressed as a value, but is not limited thereto, and may be obtained as a color channel value of various color spaces such as YCrCb and YiQ.

In addition, when the photographing device acquires an infrared image, the photographing device may acquire an infrared image through infrared illumination disposed inside or outside the photographing device. In this case, the infrared illumination may illuminate infrared rays in the near infrared region, which is a wavelength band of 750 nm to 3000 nm, but is not limited thereto, and the middle infrared ray, far infrared ray, and extreme infrared ray Infrared rays of the region (extreme infrared) may be emitted.

In addition, the controller 1020 may obtain a biometric index using an image of a subject obtained from the image acquisition unit 1010.

For example, the controller 1020 may analyze the image of the subject acquired from the image acquisition unit 1010 to obtain a biometric index such as heart rate, oxygen saturation, blood pressure, and body temperature of the subject. It is not limited and various biomarkers can be obtained.

Also, the controller 1020 may calculate biometric information based on the obtained biometric index.

For example, the controller 1020 may calculate biometric information such as emotion information and drowsiness information of the subject based on acquired biometric indexes such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto. Information can be calculated.

In addition, the control unit 1020 may control the operation of at least some of the image acquisition unit 1010, the storage unit 1030, and the communication unit 1040.

In addition, the storage unit 1030 may store the biometric index and biometric information obtained by the control unit 1020. More specifically, the storage unit 1030 may store a biometric index and biometric information for one subject, and may store biometric index and biometric information for a plurality of subjects, respectively.

In addition, the communication unit 1040 may transmit the biometric index and biometric information obtained from the control unit 1020. More specifically, the communication unit 1040 may transmit the biometric index and biometric information obtained from the control unit 1020 to the management server or to the user's terminal.

3. Various embodiments of biometric index and biometric information acquisition method

3.1 How to obtain biometric index

Referring to FIG. 4, a method 1100 for obtaining a biometric index according to an exemplary embodiment may include an operation S1110 of acquiring an image of a subject with respect to the subject.

At this time, in order to acquire an image of a subject, an image can be acquired using various cameras such as a visible light camera and an infrared camera, or an image can be acquired from various cameras, as has been described above, and a detailed description thereof will be omitted. .

In addition, the method 1100 of obtaining a biometric index according to an embodiment may include detecting a skin area (S1120).

In this case, the skin area may mean an area that can be estimated as the skin area of the subject among the images of the subject.

In addition, the skin region may well reflect changes in blood vessels due to the heartbeat, and detecting the skin region as described above may improve the accuracy of obtaining a biometric index.

In addition, according to an embodiment, in order to detect the skin area, areas other than the skin, such as eyes and hair of the subject, may be removed except for a skin area that may reflect a change in color due to the expansion of blood vessels. For example, a color value of an area other than the skin such as eyes and hair of the subject may be substituted with a meaningless value such as black, but is not limited thereto.

Also, according to an embodiment, a specific color space may be used to detect the skin area. For example, in the step of detecting the skin area (S1120), the acquired image of the subject is replaced with a value of the YCrCb color space, and the skin area may be detected based on the image expressed in the YCrCb color space. It is not limited thereto.

In addition, it is obvious that the skin area can be detected using various techniques known in the art to detect the skin area.

In addition, the method 1110 of obtaining a biometric index according to an embodiment may include setting a region of interest (S1130).

In this case, the region of interest may mean an area of interest for processing data among the acquired images of the subject, and may mean an area that can be used for data processing to obtain a biometric index. Not limited.

In addition, according to an embodiment, in order to set the region of interest, a face region of the subject may be set. For example, the face area of the subject may be set to set the region of interest included in the face of the subject.

More specifically, an area having a certain ratio based on the set center of the face area of the subject may be set as the region of interest.

For example, 80% vertically and 60% horizontally based on the center of the face area of the subject may be cropped, but is not limited thereto.

In addition, according to an embodiment, a feature point may be used to set the region of interest. For example, the nose region of the subject may be extracted as a feature point from the acquired image of the subject, and an ROI may be set based on the extracted feature point, but is not limited thereto.

More specifically, an area having a predetermined size around a feature point extracted for a set face area may be set as an ROI, but is not limited thereto.

In addition, according to an embodiment, a plurality of specific points may be used to set the region of interest. For example, from the acquired image of the subject, the eye and nose regions of the subject may be extracted as feature points, and the region of interest may be set based on the extracted feature points.

In addition, when an ROI is set for each of a plurality of images that are successively acquired, an ROI for each of the plurality of images may be independently set or may be set to be related.

For example, in order to set the region of interest for each of the plurality of images to be correlated, a face region of the subject for each of the plurality of images may be set, and the center of the face region set in the first image frame and the first image If the difference between the centers of the face regions set in the second image frame acquired after the frame does not exceed the threshold, the face region of the second image frame may be set to be the same as the face region set in the first image frame. It is not limited.

In addition, when the difference between the center of the face area set in the first image frame and the center of the face area set in the second image frame exceeds the threshold, the face area of the second image frame is It may be set to be different, but is not limited thereto.

In addition, the region of interest according to an exemplary embodiment may be set to include a part of a body part or a part of a face according to a biometric index to be acquired, and at least one region of interest may be set.

For example, the region of interest may be set to include at least a portion of the ball region in order to obtain a heart rate among the biomarkers. More specifically, the region of interest may be set to include a cheek region in which a heart rate can be easily acquired because it can reflect the degree of expansion of a blood vessel according to blood flow, but is not limited thereto.

In addition, for example, the region of interest may be set to at least two or more to obtain a blood pressure among the biomarkers, and more specifically, the region of interest includes an upper region of the face to reflect the flow of blood flow. It may be set as a region of interest including the region and the lower region of the face, but is not limited thereto.

In addition, for example, in order to obtain a blood pressure among the biomarkers, the ROI may be set to at least two or more, and more specifically, the ROI may be set to two or more ROI regions having different distances from the heart. . For example, the ROI may be set as an ROI including a hand region and an ROI including a face region, but is not limited thereto.

In addition, the method 1110 of obtaining a biometric index according to an exemplary embodiment may include processing data on an ROI (S1140).

In addition, a color channel value for the ROI may be extracted to process data on the ROI. In this case, the color channel value may be an average value of color channel pixel values of pixels included in the region of interest, and may also be referred to as an average pixel value.

For example, when a color channel value according to an RGB color space is extracted, a red channel pixel value, a green channel pixel value, and a blue channel pixel value of each pixel included in the region of interest may be extracted. The red channel value, which is the average value of the included red channel pixel values, the blue channel value that is the average value of the blue channel pixel values, and the green channel value, which is the average value of the green channel pixel values, may be extracted, but is not limited thereto, and the HSV, YCrCb color space Color channel values according to various color spaces, such as, can be extracted.

Also, a color channel value extracted according to a specific color space may be converted to another color space. For example, a color channel value extracted according to the RGB color space may be converted into a color channel value according to various color spaces such as HSV and YCrCb color space.

In addition, a color channel value extracted to process data for the ROI may be a color channel value combined by applying a weight to at least a part of the extracted color channel values according to various color spaces.

In addition, the color channel value may be extracted for each of a plurality of image frames that are successively acquired, or may be extracted for at least some image frames.

In addition, color channel values extracted from one image frame may be processed through an operation or the like. More specifically, a plurality of channel values obtained from one image frame can be processed through an operation such as adding or subtracting each other.For example, the green channel value and the red channel value obtained from one image frame are calculated as difference calculations. Although it may be processed through, it is not limited thereto, and various channel values may be processed through various calculations.

In addition, color channel values or processed values extracted from each of a plurality of image frames may be processed through an operation or the like. More specifically, the color channel values or processed values of the color channel values extracted from each of a plurality of image frames may be processed through an operation such as obtaining an average of a certain section or obtaining a deviation, but is not limited thereto. It is also possible to obtain the difference between the maximum and minimum values during the period, and can be processed through various operations.

In addition, color channel values extracted from each of a plurality of image frames and processed values of the color channel values may be processed to obtain at least one time series data.

Also, a feature value for obtaining a biometric index may be extracted based on at least some of color channel values, processed values, and time series data. For example, a frequency component for obtaining a heart rate may be extracted based on the frequency component of time series data, but is not limited thereto.

In addition, the method 1110 of obtaining a biometric index according to an exemplary embodiment may include obtaining a biometric index (S1150).

In addition, a feature value extracted from the region of interest may be used to obtain the biometric index. For example, a heart rate may be obtained based on a frequency component of time series data extracted from the region of interest, but is not limited thereto.

In addition, since the step of processing data on the region of interest (S1140) and the step of acquiring the biometric index (S1150) may be different according to each biometric index, more detailed information will be described in detail in the corresponding part.

Referring to FIG. 5, an image 1161 of a subject may include a face region, a skin region (1162), and an interest region (1163). A detailed description of this has been described above, and thus, redundant descriptions will be omitted.

Further, referring to FIG. 5, a color channel value for an ROI may be extracted (1164 ), the extracted color channel value may be processed (1165 ), and a biometric index may be obtained based on this (1166 ).

However, this will be described in detail in the relevant part below.

3.2 How to obtain biometric information

Referring to FIG. 6, a method of obtaining biometric information according to an embodiment may include obtaining a biometric index (S1210).

In this case, the biometric index may be obtained by the aforementioned biometric index acquisition method, but is not limited thereto and may be obtained by an external sensor such as an ECG sensor.

In addition, redundant description of the biometric index will be omitted.

In addition, a method of obtaining biometric information according to an embodiment may include obtaining personal and statistical data (S1220).

In this case, the personal and statistical data may mean collectable personal and statistical data of the subject such as age and gender, and may mean observable personal and statistical data such as facial expressions and wrinkles of the subject, but are not limited thereto. It can be a variety of personal and statistical data, except for the biometric index for obtaining biometric information.

In addition, a method of obtaining biometric information according to an embodiment may include obtaining biometric information (S1230).

In this case, the biometric information may be sleep information, emotion information, etc. of the subject, but is not limited thereto.

In addition, the biometric index may be used to obtain the biometric information. For example, the heart rate of the biometric index may be used to obtain the drowsiness information. More specifically, when the heart rate of the subject is less than or equal to the reference heart rate, the subject may be considered to be in a drowsy state, and the degree of drowsiness of the subject may be obtained according to a time when the heart rate of the subject is less than or equal to the reference heart rate.

In addition, for example, in order to obtain the emotion information, a heart rate and a blood pressure among the biomarkers may be used. More specifically, when the heart rate and blood pressure of the subject are equal to or higher than the reference heart rate and blood pressure, the subject may be considered to be in an excited state.

In addition, the biometric index and the personal and statistical data may be used to obtain the biometric information. For example, personal and statistical data such as heart rate and age and sex of a subject may be used to obtain the drowsiness information.

In addition, for example, personal and statistical data such as heart rate, blood pressure, and facial expression of a subject, age, and gender may be used to obtain the emotion information.

In addition, in order to obtain the biometric information, a weight may be assigned to the biometric index and the personal and statistical data. For example, in order to obtain the biometric information, different weights may be assigned to the biometric index and the personal and statistical data, and different weights may be assigned according to the subject.

3.3 Bio-index acquisition method using a bio-index acquisition model

7A illustrates a method 1300 for obtaining a biometric index using the model 1302 for obtaining a biometric index according to an exemplary embodiment.

In this case, the biometric index acquisition model 1302 according to an embodiment may be implemented using a machine learning method. For example, the biometric index acquisition model 1302 may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, and reinforcement learning.

In addition, the biometric index acquisition model 1302 according to an embodiment may be implemented as an artificial neural network (ANN). For example, the biometric index acquisition model 1302 may be implemented as a feedforward neural network, a radial basis function network, or a kohonen self-organizing network. Not limited.

In addition, the biometric index acquisition model 1302 according to an embodiment may be implemented as a deep neural network (DNN). For example, the biometric index acquisition model 1302 is a convolutional neural network (CNN), a recurrent neural network (RNN), a long short term memory network (LSTM), or a gated recurrent units (GRUs). It may be implemented as, but is not limited thereto.

In addition, the image 1301 input to the biometric index acquisition model 1302 may be acquired image data itself.

In addition, the image 1302 input to the biometric index acquisition model 1302 may be preprocessed image data. For example, the image 1302 may be eulerian video magnification, but is not limited thereto, and various pre-processing such as obtaining an average value of the obtained RGB values may be performed.

In addition, the obtained biometric index 1303 may be heart rate, oxygen saturation, blood pressure, body temperature, and the like.

In addition, the obtained biometric index 1303 may be one, and a plurality of biometric indexes may be obtained at the same time. For example, a heart rate may be obtained as a result of the biometric index acquisition model 1302, but is not limited thereto, and a heart rate and blood pressure may be simultaneously acquired as a result of the biometric index acquisition model 1302.

In addition, FIG. 7B shows a method 1350 for obtaining a biometric index using the model 1354 for obtaining a biometric index according to another exemplary embodiment.

In this case, the biometric index acquisition model 1354 may acquire features 1352 extracted from the image 1351 and personal and statistical data 1352 as input values. For example, a feature called time series data about a color channel value may be extracted from the image 1351, and the biometric index acquisition model 1354 inputs time series data and personal and statistical data about the color channel value. It can be obtained as a result of the biometric index 1355.

In addition, the personal and statistical data may refer to collectable personal and statistical data of the subject such as age, sex, height, and weight, and may refer to observable personal and statistical data such as facial expressions, wrinkles, and face color of the subject. Alternatively, it may mean personal and statistical data that can be quantified, such as average blood pressure, average color, average height, and average weight.

In addition, since the contents of the biometric index acquisition model 1302 described above may be applied to the biometric index acquisition model 1354, a redundant description will be omitted.

In addition, FIG. 8 illustrates a method 1400 for obtaining a biometric index using the model 1405 for obtaining a biometric index according to another embodiment.

In this case, the biometric index acquisition method 1400 may include a feature extraction model 1402, and the feature extraction model 1402 according to an embodiment may be implemented using a machine learning method. For example, the feature extraction model 1402 may be a model implemented through supervised learning, but is not limited thereto, and may be a model implemented through unsupervised learning, semi-supervised learning, and reinforcement learning.

In addition, the feature extraction model 1402 according to an embodiment may be implemented as an artificial neural network (ANN). For example, the biometric index acquisition model 1302 may be implemented as a feedforward neural network, a radial basis function network, or a kohonen self-organizing network. Not limited.

In addition, the feature extraction model 1402 according to an embodiment may be implemented as a deep neural network (DNN). For example, the biometric index acquisition model 1302 is a convolutional neural network (CNN), a recurrent neural network (RNN), a long short term memory network (LSTM), or a gated recurrent units (GRUs). It may be implemented as, but is not limited thereto.

In addition, the biometric index acquisition model 1405 may acquire features 1403 and personal and statistical data 1404 extracted from the feature extraction model 1402 as input values, and based on this, the biometric index 1406 ) Can be calculated as the result.

In addition, since the contents of the biometric index acquisition model 1302 described above may be applied to the biometric index acquisition model 1405, a redundant description will be omitted.

Machine learning, an artificial neural network, or a deep neural network model may be used to obtain a biometric index as in the example described with reference to FIGS. 7 and 8.

4. Various Examples of Biometric Index Acquisition

4.1 Various embodiments of heart rate measurement method

When the heart beats in the body of a living organism, blood can be carried throughout the body by the beat of the heart. At this time, blood flows through the blood vessel, the volume of the blood vessel may change over time, and the amount of blood contained in the blood vessel may change.

Therefore, when measuring a change in the volume of a blood vessel or a change in the amount of blood, the heart rate can be obtained. For example, when the amount of blood contained in a blood vessel changes, the amount of hemoglobin and oxyhemoglobin contained in the blood may change, and accordingly, the amount of light reflected by the blood may change. Therefore, when measuring the change in the amount of light reflected by the blood in this way, it is possible to obtain a heart rate.

In addition, it is obvious that in addition to the above-described exemplary principles, various principles for measuring heart rate with light can be applied.

Referring to FIG. 9, the method 1500 for measuring heart rate according to an embodiment includes obtaining an image (S1510) and detecting a skin area (S1520) with respect to at least one image frame among a plurality of acquired image frames. ), detecting the region of interest (S1530) and processing data on the region of interest (S1540), but the present invention is not limited thereto.

In addition, since the step of acquiring the image (S1510), the step of detecting the skin area (S1520), and the step of detecting the region of interest (S1530) are described above, redundant descriptions will be omitted.

In addition, the step of processing the data on the ROI (S1540) may be performed on at least one image frame among a plurality of acquired image frames.

In addition, a color channel value for the ROI may be extracted for at least one image frame among a plurality of image frames obtained to process data on the ROI. In this case, the color channel value may be an average value of color channel values of pixels included in the ROI, and may be referred to as an average pixel value.

Further, the detailed description of the step of processing the data on the ROI (S1540) has been described above, and thus, a duplicate description will be omitted.

In addition, the heart rate measurement method 1500 according to an embodiment includes at least some of the steps of extracting time series data (S1550) and the step of acquiring a heart rate (S1560) for at least some image frame groups among a plurality of acquired image frames. It may include, but is not limited thereto.

In addition, the step of extracting the time series data (S1550) may be performed on at least some image frame groups among the acquired plurality of image frames.

In this case, the image frame group may mean a plurality of continuous or discontinuous image frame groups. For example, the image frame group may mean a group of consecutive image frames from the first image frame to the 180th image frame, but is not limited thereto, and at least some of the images from the first image frame to the 180th image frame It may also mean a group of frames.

In addition, the step of acquiring the heart rate (S1560) may be performed on at least some image frame groups among the acquired plurality of image frames.

In this case, a frequency component of the acquired time series data may be extracted to obtain the heart rate. For example, in order to obtain the heart rate, the time series data may be transformed according to a Fourier transform (FT) to extract a frequency component, but is not limited thereto, and the time series data is a Fast Fourier transform. , FFT), discrete Fourier transform (DFT), short time fourier transfom (STFT), and the like, but are not limited thereto, and various processes for extracting frequency components may be performed.

In addition, the heart rate may be acquired based on one heart rate, and may be acquired based on at least two or more heart rates. For example, one heart rate may be obtained based on one time series data, another heart rate may be obtained based on another time series data, and a final The heart rate may be obtained, but is not limited thereto.

4.2 Various Examples of Method for Measuring Oxygen Saturation

Oxygen saturation (SPO2) refers to the amount of oxygen bound to hemoglobin, and can be expressed as the fraction of oxygen hemoglobin to total hemoglobin in the blood.

In addition, hemoglobin and oxyhemoglobin may have the same or different absorption rates for light having one wavelength. For example, hemoglobin and oxyhemoglobin may have different absorbances for light in the 700 nm band, different absorbances for light in the 1000 nm band, and similar absorbance for light in the 800 nm band. .

In addition, when the amount of blood contained in the blood vessel changes, the amount of hemoglobin and oxygen hemoglobin contained in the blood may change.

Therefore, the extinction coefficient of hemoglobin for light in the first wavelength band, the extinction coefficient of oxyhemoglobin, the extinction coefficient of hemoglobin for the light in the second wavelength band, the extinction coefficient of oxyhemoglobin, the first wavelength according to the change in the amount of blood Oxygen saturation can be obtained by using the degree of change of light in the second wavelength band according to the change in the amount of light and the amount of blood in the band.

For example, the absorbance of oxyhemoglobin for light in the first wavelength band

, The absorbance of hemoglobin for light in the first wavelength band

, The absorbance of oxygen haemoglobin for light in the second wavelength band

, The absorbance of hemoglobin for light in the second wavelength band

, When the ratio of oxygen hemoglobin is S, the following equation 1 can be established,

Oxygen saturation may be expressed as S*100 (%), but is not limited thereto.

In addition, it is obvious that in addition to the above-described exemplary principles, various principles for measuring oxygen saturation with light can be applied.

Referring to FIG. 10, in the method 1600 for measuring oxygen saturation according to an embodiment, for at least one image frame among a plurality of acquired image frames, acquiring an image (S1610) and detecting a skin area ( S1620), detecting the region of interest (S1630), and processing at least two color channel values for the region of interest (S1640) may include at least some, but the present invention is not limited thereto.

In addition, since the step of acquiring the image (S1610), the step of detecting the skin area (S1620), and the step of detecting the region of interest (S1630) are described above, redundant descriptions will be omitted.

Further, in the step of processing at least two color channel values for the region of interest (S1640), operations of the step of processing the data for the region of interest described above may be performed, so a redundant description will be omitted.

In addition, the two color channels may be selected in consideration of the absorbance of hemoglobin and the absorbance of oxygen hemoglobin. For example, in the case of using the RGB color space, a Red channel in which the absorbance of hemoglobin is higher than that of oxygen-hemoglobin and a Blue channel in which the absorbance of oxygen-hemoglobin is higher than that of hemoglobin may be selected, but are not limited thereto.

In addition, the oxygen saturation measurement method 1600 according to an embodiment includes the step of extracting time series data for at least two color channel values for at least some image frame groups among a plurality of acquired image frames (S1650) and oxygen saturation. It may include at least some of the step of obtaining (S1660), but is not limited thereto.

In addition, the step of extracting time series data for the at least two color channel values (S1650) may be performed on at least some image frame groups among the plurality of acquired image frames.

In addition, the step of obtaining the oxygen saturation (S1660) may be performed on at least some of the image frame groups among the obtained plurality of image frames.

In this case, the AC component and the DC component of the obtained at least two time series data may be used to obtain the oxygen saturation degree. In this case, the AC component may mean the difference between the maximum value and the minimum value of the time series data, and may mean a difference between the average of the maximum values and the average of the minimum values, but is not limited thereto, and will be understood as an ordinary AC component. I can. Further, the DC component may be understood as an average value of time series data, but is not limited thereto, and may be understood as a conventional DC component.

Further, in order to obtain the oxygen saturation degree, an equation may be used.

For example, the absorbance of oxyhemoglobin for the red channel

, The absorbance of hemoglobin for the red channel

, The absorbance of oxygen hemoglobin for the blue channel

, The absorbance of hemoglobin for the blue channel

, When the ratio of oxygen hemoglobin is S, Equation 2 as follows may be established.

In this case, the oxygen saturation degree may be expressed as S*100 (%), but is not limited thereto.

In addition, it is obvious that in addition to the above-described exemplary equations, various equations for obtaining oxygen saturation using at least two color channel values may be used.

In addition, the oxygen saturation degree may be obtained based on one oxygen saturation degree, and may be obtained based on at least two oxygen saturation degrees. For example, one oxygen saturation may be obtained based on at least two time series data, another oxygen saturation may be obtained based on another at least two time series data, and the obtained at least two oxygen saturation levels may be obtained. The final oxygen saturation may be obtained based on the saturation, but the present invention is not limited thereto.

In addition, in order to obtain a more accurate oxygen saturation degree, the finally obtained oxygen saturation degree and an oxymeter may be used.

Referring to FIG. 11, the method 1700 for measuring oxygen saturation according to an embodiment includes acquiring an image (S1710) and detecting a skin area with respect to at least one image frame among a plurality of acquired image frames ( S1720), detecting a region of interest (S1730), and processing at least one color channel value for the region of interest (S1740), but are not limited thereto.

In addition, the oxygen saturation measurement method 1700 according to an embodiment may acquire a plurality of IR (Infrared) image frames, and acquires an IR image for at least one IR image frame among a plurality of acquired IR image frames. It may include at least some of the step of (S1711), the step of detecting the skin area (S1721), the step of detecting the region of interest (S1731), and the step of processing IR data for the region of interest (S1741), but limited thereto. It doesn't work.

In addition, the steps of acquiring the image (S1710 and S1711), the steps of detecting the skin area (S1720 and S1721), and the steps of detecting the region of interest (S1730 and S1731) are described above. do.

In addition, the processing of the at least one color channel value for the ROI (S1740) and the processing of IR data for the ROI (S1741) include the operations of processing the data for the ROI. Since it can be performed, the redundant description will be omitted.

In addition, the at least one color channel and the wavelength band of the IR may be selected in consideration of absorbance of hemoglobin and absorbance of oxygen hemoglobin. For example, in the case of using the RGB color space, a red channel in which the absorbance of hemoglobin is higher than that of oxygen hemoglobin may be selected, and the IR wavelength band of the 880 nm band in which the absorbance of oxygen-hemoglobin is higher than the absorbance of hemoglobin may be selected. , But is not limited thereto.

In addition, the oxygen saturation measurement method 1700 according to an embodiment includes the step of extracting time series data for color channel values for at least some image frame groups among a plurality of acquired image frames (S1750), and time series for IR data. It may include at least some of the step of extracting data (S1751) and the step of obtaining oxygen saturation (S1760), but is not limited thereto.

In addition, the step of extracting the time series data for the color channel value (S1750) and the step of extracting the time series data for the IR data (S1751) may include the operations of the step of extracting the time series data described above. The description will be omitted.

In addition, the step of obtaining the oxygen saturation (S1760) may be performed for at least some of the acquired image frame groups and at least some of the acquired IR image frames.

In this case, at least some image frame groups among the plurality of acquired image frames and at least some IR image frame groups among the acquired plurality of IR image frames may be the same or different from each other.

For example, when the image frame and the IR image frame are acquired in different sequences, the image frame group and the IR image frame group may be different from each other, and the image frame and the IR image frame are acquired in the same sequence. In this case, the image frame group and the IR image frame group may be the same, but are not limited thereto.

For example, the absorbance of oxyhemoglobin for the red channel

, The absorbance of hemoglobin for the red channel

, The absorbance of oxygen hemoglobin for 880 nm

, The absorbance of hemoglobin for 880nm

, When the ratio of oxygen hemoglobin is S, Equation 3 below can be established.

In addition, it is obvious that in addition to the above-described exemplary equations, various equations for obtaining oxygen saturation using at least one color channel value and IR data may be used.

4.3 Various embodiments of blood pressure measurement method

Blood pressure may refer to a pressure applied by blood flowing along a blood vessel to a wall of a blood vessel.

Accordingly, blood pressure may be affected by the speed of blood flow, the thickness of the blood vessel wall, and waste products accumulated in the blood vessel.

In addition, when blood flows along a blood vessel, the volume of the blood vessel may change over time, and the amount of blood contained in the blood vessel may change.

Therefore, when measuring the rate of change in the volume of blood vessels and the rate of change in the amount of blood, blood pressure can be obtained.

For example, when measuring changes in blood vessels at two points with different distances from the heart, blood pressure can be obtained based on the difference in changes in blood vessels at two points, and features that can represent changes in blood vessels that change over time Blood pressure may be obtained by extraction, but is not limited thereto.

In addition, it is obvious that in addition to the above-described exemplary principles, various principles for measuring blood pressure with light can be applied.

Referring to FIG. 12, in the blood pressure measurement method 1800 according to an embodiment, obtaining an image (S1810) and detecting a skin area (S1820) with respect to at least one image frame among a plurality of acquired image frames (S1820). ), detecting a first region of interest (S1830), detecting a second region of interest (S1831), processing data for a first region of interest (S1840), and processing data for a second region of interest At least a part of the step S1841 may be included, but the present disclosure is not limited thereto.

In addition, since the step of acquiring the image (S1810) and the step of detecting the skin area (S1820) are described above, redundant descriptions will be omitted.

Further, details of the steps of detecting the first region of interest and the second region of interest (S1830 and S1831) have been described above, and thus redundant descriptions will be omitted.

In this case, the first region of interest and the second region of interest may be set to two regions having different distances from the heart of the subject. For example, the first region of interest may be set as an upper region of the subject's face, and the second region of interest may be set as a lower region of the subject's face, but is not limited thereto, and the first region of interest is It may be set as the face area of and the second ROI may be set as the back area of the subject's hand.

In addition, the processing of the data on the first ROI (S1840) and the processing of the data on the second ROI (S1841) may perform the operations of processing the data on the ROI. As it may be, duplicate descriptions will be omitted.

In addition, the blood pressure measurement method 1800 according to an embodiment includes extracting time series data for first and second ROIs for at least some image frame groups among a plurality of acquired image frames (S1850). It may include at least a part of calculating a pulse transit time (PTT) based on time series data (S1860) and obtaining a blood pressure (S1870), but is not limited thereto.

In addition, in the step of extracting time series data for the first and second ROI (S1850), operations of the step of extracting time series data described above may be performed, and therefore, a redundant description will be omitted.

In addition, calculating PTT based on the acquired time series data (S1860) may be performed on at least some image frame groups among the acquired plurality of image frames.

In this case, the PTT may be calculated based on an extreme value of time series data for the first region of interest and time series data for the second region of interest. For example, the PTT may be calculated based on a time difference between a maximum value of time series data for the first region of interest and a maximum value of time series data for the second region of interest, but is not limited thereto, and the first region of interest The PTT may be calculated based on a time difference between the minimum value of the time series data for and the minimum value of the time series data for the second ROI.

In addition, the PTT may be calculated based on an inflection point of time series data for the first region of interest and time series data for the second region of interest. For example, the PTT may be calculated based on a time difference between an inflection point of time series data for the first region of interest and an inflection point of time series data for the second region of interest, but is not limited thereto.

In addition, the PTT may be calculated based on various points of time series data for each region of interest in addition to the above-described extreme values and inflection points.

In addition, a time difference between the time series data for the first region of interest and the time series data for the second region of interest may be calculated based on frames obtained at points such as extreme values and inflection points. For example, when a maximum value is obtained in a 10th frame from time series data for a first region of interest, and a maximum value is obtained in a twelfth frame from time series data for a second region of interest, the first region of interest and the second region of interest The time difference between the time series data for the region may be a time for acquiring two frames, and a PTT may be calculated based on this.

In addition, the step of obtaining the blood pressure (S1870) may be performed on at least some image frame groups among the plurality of acquired image frames.

In addition, PTT may be used to obtain the blood pressure. For example, a function for PTT may be used to obtain the blood pressure. More specifically, a function such as Equation 4 may be used, and various functions such as a linear function, a quadratic function, a log function, and an exponential function may be used, but the function is not limited thereto. .

In addition, PTT and personal and statistical data can be used to obtain the blood pressure. For example, in order to obtain the blood pressure, a function of PTT and a function of personal and statistical data such as age, weight, and height may be used. More specifically, Equation 5 as follows may be used, and various functions such as PTT, weight, height, and age as variables, linear functions, quadratic functions, logarithmic functions, and exponential functions may be used. It is not limited thereto.

In addition, a regression analysis method using the above-described function may be used to obtain the blood pressure, but is not limited thereto.

In addition, a machine learning method using the above-described function may be used to obtain the blood pressure, but is not limited thereto.

In addition, it is obvious that in addition to the above-described exemplary equations, various equations for obtaining blood pressure using PTT may be used.

Also, the blood pressure may be obtained based on one blood pressure, and may be obtained based on at least two or more blood pressures. For example, one blood pressure may be obtained based on a first PTT calculated based on time series data for the first and second regions of interest, and based on other time series data for the first and second regions of interest. Another blood pressure may be obtained based on the calculated second PTT, and a final blood pressure may be obtained based on at least two or more obtained blood pressures, but the present invention is not limited thereto.

Referring to FIG. 13, in the blood pressure measurement method 1900 according to an embodiment, obtaining an image (S1910) and detecting a skin area (S1920) with respect to at least one image frame among a plurality of acquired image frames. ), detecting the region of interest (S1930) and processing data on the region of interest (S1940), but are not limited thereto.

At this time, the steps for acquiring the image (S1910), detecting the skin area (S1920), detecting the ROI (S1930), and processing data on the ROI (S1940) As detailed information has been described above, redundant descriptions will be omitted.

In addition, in the blood pressure measurement method 1900 according to an embodiment, the step of extracting time series data for at least some image frame groups among a plurality of acquired image frames (S1950), and extracting a feature based on the acquired time series data. At least a part of the step S1960 and the step S1970 of obtaining a blood pressure may be included, but the present invention is not limited thereto.

Further, in the step of extracting time series data (S1950), since the operations of the step of extracting time series data described above may be performed, a redundant description will be omitted.

Also, the step of extracting a feature based on the acquired time series data (S1960) may be performed on at least some image frame groups among the plurality of acquired image frames.

In this case, the feature may mean a mathematical or physical feature of the acquired time series data. For example, the feature is the maximum value of the acquired time series data, the average of the local maximum value, the minimum value, the average of the minimum value, the difference between the local maximum value and the minimum value, average, inflection point, first derivative data, second derivative data, slope at a specific point in time. It may mean a mathematical characteristic such as, and may mean a physical characteristic such as a change amount of blood, a change rate of blood, an amount of change of blood vessels, and a rate of change of blood vessels, but is not limited thereto.

In addition, it is obvious that the above characteristics may be various characteristics for obtaining blood pressure in addition to the above-described exemplary characteristics.

In addition, the step of obtaining the blood pressure (S1970) may be performed on at least some image frame groups among the plurality of acquired image frames.

Also, the feature can be used to obtain the blood pressure. For example, a function for the feature can be used to obtain the blood pressure. More specifically, a function such as Equation 6 may be used, and various functions such as a linear function, a quadratic function, a log function, and an exponential function may be used, but the function is not limited thereto.

In addition, the characteristics and personal and statistical data can be used to obtain the blood pressure. For example, to obtain the blood pressure, a function of the characteristic and a function of personal and statistical data such as age, weight, and height may be used. More specifically, Equation 7 as follows may be used, and various functions such as a linear function, a quadratic function, a log function, an exponential function, etc. may be used as the function, It is not limited thereto.

In addition, it is obvious that, in addition to the above-described exemplary equation, an equation for obtaining a blood pressure using a feature may be used.

Also, the blood pressure may be obtained based on one blood pressure, and may be obtained based on at least two or more blood pressures. For example, one blood pressure may be obtained based on a first feature calculated based on time series data, and another blood pressure may be obtained based on a second feature calculated based on other time series data, The final blood pressure may be obtained based on the obtained at least two or more blood pressures, but the present invention is not limited thereto.

4.4 Various Examples of Method for Measuring Body Temperature

Referring to FIG. 14, the method 2000 for measuring body temperature according to an exemplary embodiment may include at least some of acquiring a skin temperature (S2010) and acquiring a body temperature (S2020 ), but is not limited thereto.

In addition, the step of obtaining the skin temperature (S2010) may be performed in a non-contact manner.

For example, an image sensor such as a camera may be used to obtain the skin temperature. More specifically, at least one color channel value of an image obtained from an image sensor such as a camera may be used to obtain the skin temperature. For example, when the HSV color space is used, the skin temperature may be obtained using a brightness (S, Saturation) value, but is not limited thereto.

In addition, for example, a sensor such as a thermal imaging camera may be used to obtain the skin temperature, and an image sensor such as an infrared camera may be used, but the present invention is not limited thereto.

In addition, the step of obtaining the body temperature (S2020) may be performed in a non-contact manner.

For example, skin temperature may be used to obtain the body temperature. More specifically, the body temperature may be obtained based on the skin temperature by using the relationship between the body temperature and the area where the skin temperature is measured.

In this case, the skin temperature may be obtained from the step of acquiring the skin temperature (S2010), or may be obtained by another external sensor.

Also, for example, an image sensor such as a camera may be used to obtain the body temperature. More specifically, an image obtained from an image sensor such as a camera may be used to obtain the skin temperature. For example, a body temperature may be obtained by using an image acquired from an image sensor such as the camera as data in a body temperature measurement machine learning model, but is not limited thereto.

Hereinafter, a method of acquiring skin temperature and a method of acquiring body temperature will be described in detail.

First, a method of acquiring skin temperature using an image sensor such as a camera will be described.

According to an embodiment, the brightness value of the acquired image may be used to obtain the skin temperature. For example, a function for a lightness value may be used to obtain the skin temperature. More specifically, Equation 8 as follows may be used, and as the function, various functions such as a linear function, a quadratic function, a log function, and an exponential function may be used, but the function is not limited thereto. .

In addition, according to an embodiment, the brightness value and personal and statistical data of the acquired image may be used to obtain the skin temperature. For example, in order to obtain the skin temperature, a function of a lightness value and a function of personal and statistical data such as age, race, and gender may be used. More specifically, Equation 9 as follows may be used, and various functions such as a linear function, a quadratic function, a log function, an exponential function, etc. may be used as the function. , But is not limited thereto.

In addition, according to an embodiment, a brightness value and a color value of an image obtained may be used to obtain a skin temperature. For example, a function for a lightness value and a function for a color value may be used to obtain the skin temperature. More specifically, Equation 10 as follows may be used, and various functions such as a linear function, a quadratic function, a log function, and an exponential function may be used, but are limited thereto. It doesn't work.

In addition, according to an embodiment, a brightness value, a color value, and personal and statistical data of the acquired image may be used to obtain the skin temperature. For example, in order to obtain the skin temperature, a function for a lightness value, a function for a color value, and a function of personal and statistical data such as age, race, and gender may be used. More specifically, the following Equation 11 may be used, and various functions such as a linear function, a quadratic function, a log function, an exponential function, etc. are used as the function with a lightness value, a color value, age, race, and gender as variables. It may be, but is not limited thereto.

In addition, according to an embodiment, in order to obtain the skin temperature, a brightness value, a color value, and a change value of the brightness value of the acquired image may be used. For example, in order to obtain the skin temperature, a function for a brightness value, a function for a color value, and a function for a change value of the brightness value may be used. More specifically, the following Equation 12 may be used, and various functions such as a linear function, a quadratic function, a log function, an exponential function, etc. are used as a variable of a brightness value, a color value, and a change value of the brightness value. It may be, but is not limited thereto. In addition, when the change value is used as described above, measurement noise due to an external environment can be reduced.

In addition, according to an embodiment, a brightness value, a color value, a change value of the brightness value, and personal and statistical data of the acquired image may be used to obtain the skin temperature. For example, in order to obtain the skin temperature, a function of a lightness value, a function of a color value, a function of a change value of a brightness value, and a function of personal and statistical data such as age, race, and gender may be used. More specifically, Equation 13 as follows may be used, and the function is a linear function, a quadratic function, a log function, and an exponent with a brightness value, a color value, a change value of the brightness value, age, race, and sex as variables. Various functions such as functions may be used, but are not limited thereto.

In addition, according to an embodiment, a brightness value, a color value, a change value of the brightness value, and a change value of the color value of the acquired image may be used to obtain the skin temperature. For example, in order to obtain the skin temperature, a function for a lightness value, a function for a color value, a function for a change value of a brightness value, and a function for a change value of a color value may be used. More specifically, the following Equation 14 may be used, and the function is a linear function, a quadratic function, a log function, and an exponent using a brightness value, a color value, a change value of the brightness value, and a change value of the color value as variables. Various functions such as functions may be used, but are not limited thereto. In addition, when the change value is used as described above, measurement noise due to an external environment can be reduced.

In addition, according to an embodiment, a brightness value, a color value, a change value of the brightness value, a change value of the color value, and personal and statistical data of the acquired image may be used to obtain the skin temperature. For example, in order to obtain the skin temperature, a function for a lightness value, a function for a color value, a function for a change value of a brightness value, a function for a change value of a color value, and personal, statistical Data functions can be used. More specifically, Equation 15 as follows may be used, and the function is a lightness value, a color value, a change value of a brightness value, a change value of a color value, a linear function with age, race, and gender as variables, and a second order. Various functions such as functions, log functions, and exponential functions may be used, but are not limited thereto.

In addition, a regression analysis method using the above-described function may be used to obtain the skin temperature, but is not limited thereto.

In addition, a machine learning method or a deep learning method using the above-described function may be used to obtain the skin temperature, but is not limited thereto.

In addition, it is obvious that in addition to the above-described exemplary equations, various equations may be used. For example, an equation using at least some of personal and statistical data such as the above-described brightness value, color value, change value of brightness value, change value of color value, age, race, gender, etc. may be used. The used equation may be used.

Next, a method of acquiring body temperature using an image sensor such as a camera will be described.

According to an embodiment, the acquired skin temperature may be used to obtain body temperature. For example, a function of skin temperature can be used to obtain the body temperature. More specifically, the following Equation 16 may be used, and the function may be a variety of functions such as a linear function, a quadratic function, a log function, and an exponential function using skin temperature as a variable, but is not limited thereto.

In addition, according to an embodiment, when the body temperature is obtained indoors, the obtained skin temperature and the room temperature may be used to obtain the body temperature. For example, a function of skin temperature and a function of room temperature may be used to obtain the body temperature. More specifically, the following Equation 17 may be used, and the function may be a variety of functions such as a linear function, a quadratic function, a log function, an exponential function, etc. using skin temperature and room temperature as variables, but is not limited thereto. Does not.

In addition, according to an embodiment, when body temperature is obtained indoors, the acquired skin temperature, room temperature, and heart rate may be used to obtain body temperature. For example, a function of skin temperature, a function of room temperature, and a function of heart rate may be used to obtain the body temperature. More specifically, the following Equation 18 may be used, and the function may be a variety of functions such as a linear function, a quadratic function, a log function, an exponential function, etc. using skin temperature, room temperature, and heart rate as variables, but limited thereto. It doesn't work.

Further, according to an embodiment, various skin temperatures may be used at the same time. For example, when body temperature is obtained indoors, a first skin temperature for a first body part, a second skin temperature for a second body part, room temperature, and heart rate obtained to obtain body temperature may be used. For example, in order to obtain the body temperature, a function for a first skin temperature, a function for a second skin temperature, a function for a room temperature, and a function for a heart rate may be used. More specifically, Equation 19 as follows may be used, and the functions include various functions such as a first skin temperature, a second skin temperature, a room temperature, and a heart rate as variables: a linear function, a quadratic function, a log function, an exponential function, etc. May be, but is not limited thereto.

In addition, a regression analysis method using the above-described function may be used to obtain the body temperature, but is not limited thereto.

Further, in order to obtain the body temperature, a machine learning method or a deep learning method using the above-described function may be used, but is not limited thereto.

In addition, it is obvious that in addition to the above-described exemplary equations, various equations may be used. For example, an equation using at least some of the above-described first skin temperature, second skin temperature, room temperature, and heart rate may be used, and in addition, mathematics using personal and statistical data such as age, gender, race, height, weight, etc. Equations can also be used.

5. Various embodiments of the heart rate measurement method

Referring to FIG. 15, the method 2100 of obtaining a heart rate according to an embodiment includes the steps of acquiring an image (S2110), detecting a skin area (S2120), detecting a region of interest (S2130), and It may include at least some of the step of processing data for (S2140), the step of acquiring a characteristic value (S2150), and the step of acquiring a heart rate (S2160), but is not limited thereto.

In this case, since the step of acquiring the image (S2110), the step of detecting the skin area (S2120), and the step of detecting the region of interest (S2130) are described above, redundant descriptions will be omitted.

In addition, the processing of the data on the ROI (S2140) may be performed on at least one image frame among a plurality of acquired image frames.

In addition, the processing of the data on the ROI (S2140) may be performed to reduce motion artifacts and noises caused by external light.

In addition, the step of obtaining the characteristic value (S2150) may be performed on at least some image frame groups among the plurality of acquired image frames.

In addition, the step of acquiring the characteristic value (S2150) may be performed to reduce noise due to motion, noise due to external light, and the like.

In addition, the step of acquiring the heart rate (S2160) may be performed on at least some image frame groups among a plurality of acquired image frames.

In this case, the image frame group for acquiring the heart rate and the image frame group for acquiring the characteristic value may be the same or different from each other. For example, the image frame group for acquiring the characteristic value may include 18 image frames, and the image frame group for acquiring the heart rate may include 180 image frames, but are not limited thereto.

Hereinafter, the step of processing data for the region of interest (S2140), the step of acquiring the characteristic value (S2150), and the step of acquiring the heart rate (S2160) will be described in more detail.

16 is a graph of color channel values according to an exemplary embodiment.

According to an embodiment, a color channel value for the ROI may be extracted to process data on the ROI. In this case, the color channel value may be an average value of color channel values of pixels included in the region of interest, and may also be referred to as an average pixel value.

Referring to FIG. 16, in order to process data on the ROI, a color channel value according to an RGB color space for the ROI may be extracted. More specifically, a red channel value that is an average value of red channel pixel values, a blue channel value that is an average value of blue channel pixel values, and a green channel value that is an average value of green channel pixel values may be extracted.

Hereinafter, a description will be made based on a color channel value extracted according to an RGB color space, but it is obvious that various color channel values may be applied without being limited thereto.

(A) of FIG. 16 is a graph showing the value of the red channel extracted according to the RGB color space, (b) is a graph showing the value of the green channel extracted according to the RGB color space, and (c) is a graph showing the value of the green channel extracted according to the RGB color space. This is a graph showing the extracted blue channel value.

As illustrated in FIG. 16, each color channel value may fluctuate according to the heartbeat.

However, the value of each color channel may fluctuate according to the heartbeat, and at the same time, fluctuations in values may occur according to a movement of a subject or a change in intensity of external light.

Therefore, in order to obtain the heart rate using the extracted color channel value, it is necessary to reduce the fluctuation of the value according to the movement of the subject or the change in the intensity of external light, and to maximize the fluctuation of the value according to the heart rate. I can.

5.1 Various embodiments of a method of processing data for a region of interest

It is obvious that the above-described contents can be applied to a method of processing data for an ROI, and descriptions overlapping with the above-described contents will be omitted.

FIG. 17A is a graph showing the value of the red channel extracted according to the RGB color space, and (b) is a graph showing the value of the Green channel extracted according to the RGB color space.

Referring to FIGS. 17A and 17B, it can be seen that the extracted color channel value fluctuates over time.

In this case, the extracted color channel value may fluctuate according to the beat of the heart, but fluctuation may occur according to a movement of the subject or a change in the intensity of external light.

More specifically, the large and slow fluctuation of the color channel value is more influenced by the movement of the subject or the change in the intensity of external light, and the fluctuation occurs, and the small and rapid fluctuation occurs in the heart of the subject. It may be that the fluctuation occurs because it is more influenced by the beat.

Therefore, since the value according to the subject's movement or the change in the intensity of external light is greater than the change according to the heartbeat, the relative difference between the values of at least two color channels may be used to reduce this.

For example, in order to reduce noise, a difference value between a green channel value and a red channel value may be used. More specifically, the green channel value and the red channel value acquired in the same image frame can reflect the same motion and the same intensity of external light, and the difference between the green channel value and the red channel value in the same frame is the motion of the subject and the external light. Noise due to a change in light intensity may be reduced, but the present invention is not limited thereto, and noise may be reduced by using a relative difference between values of at least two color channels.

17C is a graph showing a difference value between the Green channel value and the Red channel value.

As shown in (c) of FIG. 17, the difference between the green channel value and the red channel value may reduce noise due to a movement of a subject and a change in intensity of external light.

In addition, the above-described noise reduction method may be performed on at least one image frame among a plurality of acquired image frames, or may be performed on each of a plurality of consecutive image frames.

In addition, although not shown in FIG. 17C, noise may be reduced by using a difference value between the green channel value and the blue channel value, and the difference value between the red channel value and the blue channel value is used. Noise can also be reduced.

In addition, as described above, at least two color channel values may be selected to obtain a difference value in order to reduce noise by using a relative difference between the at least two color channel values.

In this case, the values of the at least two color channels may be selected in consideration of absorbance of blood.

According to an embodiment, the Red channel may be a channel including at least a portion of the 620nm to 750nm wavelength band, the Green channel may be a channel including at least a portion of the 495nm to 570nm wavelength band, and the Blue channel may be 450nm to 495nm The channel may include at least a portion of the wavelength band, but the present invention is not limited thereto, and each of the red channel, the green channel, and the blue channel may be a commonly understood color channel.

Referring to FIG. 18, absorbance of hemoglobin and oxygen hemoglobin according to the wavelength band of light can be seen. For example, as shown in FIG. 18, the absorbance of hemoglobin and oxygen hemoglobin for light in the 550 nm wavelength band included in the Green channel is higher than the absorbance of hemoglobin and oxygen hemoglobin in the light in the 650 nm wavelength band included in the Red channel. I can.

In addition, for example, as shown in FIG. 18, the absorbance of hemoglobin and oxygen hemoglobin for light in the 550 nm wavelength band included in the Green channel is the absorbance of hemoglobin and oxygen hemoglobin for light in the 470 nm wavelength band included in the Blue channel. Can be higher.

In addition, when blood is transported throughout the body by the heartbeat, the volume of the blood vessel may change due to the flow of blood, or the amount of blood contained in the blood vessel may change.

Accordingly, a color channel value including a wavelength band of light that is relatively absorbed by hemoglobin and oxygen hemoglobin contained in the blood may be relatively largely fluctuated by a change in the amount of blood due to the heartbeat.

On the other hand, a color channel value including a wavelength band of light that is relatively less absorbed by hemoglobin and oxygen hemoglobin contained in the blood may be changed relatively little by a change in the amount of blood due to the heartbeat.

Accordingly, according to an embodiment, at least two color channels for reducing noise may be selected in consideration of absorbance of hemoglobin and oxygen hemoglobin.

For example, according to an embodiment, in order to reduce noise, a difference between a value of a green channel, which is relatively absorbed by hemoglobin and oxygen hemoglobin, and a red channel value, which is relatively less absorbed by hemoglobin and oxygen hemoglobin, is Can be used.

In addition, for example, according to an embodiment, in order to reduce noise, a value of a Green channel value that is relatively absorbed by hemoglobin and oxygen hemoglobin and a Blue channel value that is relatively absorbed by hemoglobin and oxygen hemoglobin are reduced. Difference values can be used.

In addition, for example, according to an embodiment, in order to reduce noise, a difference between a blue channel value that is relatively absorbed by hemoglobin and oxygen hemoglobin and a red channel value that is relatively absorbed by hemoglobin and oxygen hemoglobin Values can be used.

Further, for example, according to an embodiment, a difference value between the Green channel value and the Red channel value, and a difference value between the Green channel value and the Blue channel value may be used simultaneously.

In addition, the above examples have been described based on the difference value, but in order to reduce noise due to the motion of the subject and noise due to external light, in addition to the difference value, a processed value processed by using a weight for each channel value may be used. have.

For example, as shown in Equation 20 below, a processed value processed by using a weight for each channel value may be used.

In addition, at this time, each a, b, c value can be determined so that a+b+c=0 in order to efficiently remove noise caused by the motion of the subject and noise caused by external light, which means that each channel value is In one image frame, noise due to movement of a similar degree and noise due to external light may be included, and thus it may be advantageous to effectively reduce noise.

5.2 Various embodiments of property value acquisition

A characteristic value may be obtained in order to reduce noise caused by movement of a subject and noise due to an intensity of external light.

In this case, the characteristic value may be obtained for at least some of the image frame groups among the obtained plurality of image frames.

In addition, the characteristic value may be a value indicating a characteristic of an acquired color channel value or a processed value. For example, the characteristic value may mean a color channel value or an average value of a processed value included in an image frame group, a deviation value, a standard deviation value, and the like, but is not limited thereto.

FIG. 19A is a graph showing a color channel value obtained according to an embodiment, and more specifically, a graph showing a difference value between a green channel value and a red channel value. However, for convenience of explanation, this is only specified as a difference value between a green channel value and a red channel value, and is not limited thereto, and may be various color channel values, difference values, and processed values.

Referring to (a) of FIG. 19, it can be seen that the difference between the green channel value and the red channel value (hereinafter referred to as'GR value') may not have a constant magnitude of change over time. .

In this case, the G-R value may not be constant due to the movement of the subject. For example, when the subject's movement is small, the change in the G-R value may be small, and when the subject’s movement is large, the change in the G-R value may be large, but is not limited thereto.

Also, the G-R value may not be constant according to the intensity of external light. For example, when the intensity of external light is weak, the change of the G-R value may be small, and when the intensity of the external light is strong, the change of the G-R value may be large, but the present invention is not limited thereto.

Accordingly, a characteristic value may be extracted to reduce noise due to the movement of the subject or the intensity of external light as described above.

In addition, a window for the characteristic value may be set to extract the characteristic value.

In this case, the window for the characteristic value may refer to a preset time interval, and may refer to a preset number of frames, but is not limited thereto, and at least some frame groups among a plurality of frames to obtain the characteristic value It may mean a window for setting.

FIG. 19B is a schematic diagram for explaining a window for a characteristic value, and more specifically, a schematic diagram for explaining a window for a characteristic value set as 18 image frames obtained by dividing 180 image frames into 10 equal parts. However, for convenience of explanation, this is only showing a window for a characteristic value set as 18 image frames obtained by dividing 180 image frames into 10 equal parts, but is not limited thereto, and a window for the characteristic value may be set in various ways and numbers. .

Referring to FIG. 19B, a plurality of acquired image frames may be set as a group by a window for a characteristic value. For example, as shown in (b) of FIG. 19, 180 image frames may be set as a group including 18 image frames by a window for a characteristic value. More specifically, the first image frame to the 18th image frame may be included in the first image frame group 2210, and the 19th image frame to the 36th image frame may be included in the second image frame group 2220. Not limited.

In this case, the characteristic value may be obtained for an image frame group set by a window for the characteristic value. For example, the characteristic value may be obtained for color channel values for the first image frame group 2210 and for color channel values for the second image frame group 2220.

Also, for example, when the characteristic value is an average value, an average value of color channel values for an image frame group may be obtained. More specifically, an average value of the GR values for the first to eighteenth image frames included in the first image frame group 2210 may be obtained, and the 19th to the nineteenth image frames included in the second image frame group 2220 may be obtained. An average value of GR values for 36 image frames may be obtained, but is not limited thereto.

Also, for example, when the characteristic value is a standard deviation value, a standard deviation value of a color channel value for an image frame group may be obtained. More specifically, a standard deviation value of GR values for the first to 18th image frames included in the first image frame group 2210 may be obtained, and the 19th to the 19th to the second image frame group 2220 may be A standard deviation value of GR values for the 36th image frames may be obtained, but is not limited thereto.

However, it is not limited to the above-described examples, and various characteristic values may be obtained for the image frame group.

In addition, the characteristic value may be obtained for at least some image frames included in an image frame group divided by a window for the characteristic value. For example, the characteristic value may be obtained for a color channel value for at least some of the 18 image frames included in the first image frame group 2210, and the second image frame group 2220 Color channel values for at least some of the included 18 image frames may be obtained.

Also, for example, when the characteristic value is a deviation value, a deviation value of a color channel value for at least some image frames included in the image frame group may be obtained. More specifically, a deviation value of the GR value of the first image frame included in the first image frame group with respect to the average of the GR value of the first image frame group 2210 may be obtained, and the second image frame group 2220 A deviation value of the GR value of the 19th image frame included in the second image frame group with respect to the average of the GR value of may be obtained, but is not limited thereto.

Also, for example, when the characteristic value is a deviation value, a deviation value of a color channel value for at least some image frames included in the image frame group may be obtained. More specifically, a deviation value of the GR value of the first image frame included in the first image frame group with respect to the average of the GR value of the first image frame group 2210 may be obtained, and the first image frame group 2210 A deviation value of the GR value of the second image frame included in may be obtained, but the present invention is not limited thereto.

Also, the obtained feature values can be normalized.

For example, when the characteristic value is a deviation value, the deviation value may be normalized by a standard deviation value. More specifically, when a deviation value of the GR value of the first image frame included in the first image frame group 2210 to the average of the GR value of the first image frame group 2210 is obtained, the GR of the first image frame The deviation value of the value may be normalized by the standard deviation value of the first image frame group 2210, but is not limited thereto and may be normalized in various ways.

In addition, when normalized in this way, the magnitude of the change is normalized to better reflect the change in the value due to the heartbeat, and noise due to the movement of the subject and the noise due to the change in the intensity of external light can be effectively reduced.

FIG. 19C is a graph showing a characteristic value obtained according to an exemplary embodiment, and more specifically, a graph showing a deviation value obtained based on a G-R value. However, for convenience of explanation, this is only a specific representation of the deviation value obtained based on the G-R value, and is not limited thereto, and may be various characteristic values obtained based on various color channel values, difference values, and processed values.

Referring to (c) of FIG. 19, it can be seen that the magnitude of the change of the value is more constant compared to (a) of FIG. 19.

Therefore, acquiring the characteristic values as described above reduces noise due to the movement of the subject and noise due to changes in the intensity of external light, and better reflects the change in values according to the heartbeat.

FIG. 20A is a graph showing a color channel value obtained according to an exemplary embodiment, and more specifically, a graph showing a difference value between a green channel value and a red channel value. However, for convenience of explanation, this is only specified as a difference value between a green channel value and a red channel value, and is not limited thereto, and may be various color channel values, difference values, and processed values.

Referring to (a) of FIG. 20, it can be seen that the difference between the green channel value and the red channel value (hereinafter referred to as'GR value') may not have a constant magnitude of change over time. .

In this case, the G-R value may not be constant due to the movement of the subject. For example, the overall GR value may be small in the time section 2301 in which the subject is positioned in the first state, and the overall GR value in the time section 2302 in which the subject is positioned in a second state different from the first state. The value may be large, but is not limited thereto.

Also, the G-R value may not be constant according to the intensity of external light. For example, the intensity of the external light in the time section 2301 in which the subject is positioned in the first state and the intensity of the external light in the time section 2302 in which the subject is positioned in the second state may be different from each other. In this case, a difference in the overall GR value may occur, but is not limited thereto.

In addition, at least some of the frame groups set by the window may at least partially overlap.

FIG. 20B is a schematic diagram for explaining a window for a characteristic value, and more specifically, a schematic diagram for explaining a window for a characteristic value set to a size obtained by dividing 180 image frames into 8 equal parts. However, for convenience of explanation, this is only showing a window for a characteristic value set to a size of 180 image frames divided into 8 equal parts, but is not limited thereto, and a window for the characteristic value may be set in various methods and sizes.

Further, referring to (b) of FIG. 20, a plurality of acquired image frames may be set as a group by a window for a characteristic value. For example, as shown in (b) of FIG. 20, 180 image frames may be set as a group including 22 or 23 image frames by a window for a characteristic value. More specifically, the first image frame to the 22nd image frame may be included in the first image frame group 2310.

In addition, referring to FIG. 20B, an image frame group set by a window for a characteristic value may at least partially overlap. For example, as shown in (b) of FIG. 20, the first image frame to the 22nd image frame may be included in the first image frame group 2310, and the sixth image frame to the 28th image frame 2 may be included in the image frame group 2320, the twelfth image frame to the 33rd image frame may be included in the third image frame group 2330, the 17th image frame to the 39th image frame group 2340 However, it is not limited thereto.

In addition, referring to FIG. 20B, the image frame group set by the window for the characteristic value may not overlap. For example, as shown in (b) of FIG. 20, although the first image frame to the 22nd image frame may be included in the first image frame group 2310, the fifth image frame from the 23rd image frame to the 45th image frame It may be included in the image frame group 2350, but is not limited thereto.

In this case, the characteristic value may be obtained for an image frame group set by a window for the characteristic value, and may also be obtained for at least some image frames included in the image frame group. Since the above-described contents may be applied, a duplicate description will be omitted.

However, processing of characteristic values obtained for image frames included in at least partially overlapping image frame groups will be described in detail below.

When the characteristic value is obtained, a plurality of characteristic values may be obtained for image frames included in an area where at least two image frame groups overlap.

For example, at least two characteristic values may be obtained for sixth to twenty-second image frames in which the first image frame group 2310 and the second image frame group 2320 overlap.

More specifically, for the sixth image frame, a first deviation value that is a deviation value of the GR value of the sixth image frame with respect to the average of the GR value of the first image frame group 2310 may be obtained, and the second image frame group A second deviation value, which is a deviation value of the GR value of the sixth image frame with respect to the average of the GR value of 2320 may be obtained, but is not limited thereto.

In addition, a plurality of obtained characteristic values may be obtained as one characteristic value through an operation. For example, a deviation value of the sixth image frame may be obtained by adding the first deviation value and the second deviation value, but is not limited thereto.

In addition, the above-described operations may be applied to obtain a characteristic value for an image frame included in an area where a plurality of image frame groups, such as three or four, overlap.

In addition, in addition to the above-described operation, a sliding window method that can be commonly understood may be applied.

FIG. 20C is a graph showing a characteristic value obtained according to an embodiment, and more specifically, a graph showing a deviation value obtained based on a G-R value. However, for convenience of explanation, this is only a specific representation of a deviation value obtained based on a G-R value, and is not limited thereto, and may be various characteristic values obtained based on various color channel values, difference values, and processed values.

Referring to (c) of FIG. 20, it can be seen that the magnitude of the overall value is constant compared to (a) of FIG. 20.

More specifically, an overall characteristic value in the time interval 2301 in which the subject is located in the first state and the overall characteristic value in the time interval 2302 in which the subject is located in the second state may be similar to each other.

Therefore, as described above, acquiring the characteristic value reduces noise caused by the movement of the subject and noise due to changes in the intensity of external light, and better reflects the change of the value according to the heartbeat.

5.3 Various embodiments of a method of using a plurality of feature values

The characteristic values obtained according to the above-described methods may be affected by the underlying color channel value, the difference value, and the processed value. Therefore, obtaining a plurality of characteristic values based on various color channel values, difference values, and processed values and using the plurality of characteristic values may enable more accurate acquisition of a biometric index.

FIG. 21A is a graph showing two characteristic values obtained according to an embodiment, and more specifically, a first characteristic value obtained based on a GR value and a second characteristic value obtained based on a GB value. It is a graph showing. However, this is only specifically indicated for convenience of description, and is not limited thereto, and may be a characteristic value obtained based on various color channel values, difference values, and processed values.

In this case, the first characteristic value obtained based on the G-R value may be affected by the G-R value. For example, when the external light is light close to the blue channel, the G-R value may not reflect changes in blood according to the heartbeat well.

Or, for example, the change in blood according to the heartbeat may be reflected by the difference between the absorbance of the green channel and the absorbance of the red channel.

Also, the second characteristic value obtained based on the G-B value may be affected by the G-B value. For example, when the external light is light close to the red channel, the G-B value may not reflect changes in blood according to the heartbeat well.

Alternatively, for example, the change in blood according to the heartbeat may be reflected by the difference between the absorbance of the green channel and the absorbance of the blue channel.

In addition, referring to FIG. 21A, the first characteristic value and the second characteristic value may have a complementary relationship. For example, in a section in which the first characteristic value does not reflect the change according to the heartbeat well, the second characteristic value may well reflect the change according to the heartbeat, and vice versa.

Accordingly, the first characteristic value and the second characteristic value may be used to reduce noise according to a change in a wavelength of external light or to better reflect a change in blood according to a heartbeat.

(B) of FIG. 21 is a graph showing a third characteristic value obtained using the first characteristic value and the second characteristic value, and more specifically, obtained by summing the first characteristic value and the second characteristic value. It is a graph showing the third characteristic value. However, this is only shown specifically for convenience of description, and is not limited thereto.

In addition, the third characteristic value may be obtained based on an operation of the first characteristic value and the second characteristic value. For example, the third characteristic value is the first characteristic value and the second characteristic value. It may be obtained based on a sum operation, but is not limited thereto, and may be obtained based on various operations such as a difference operation and a multiplication operation.

In addition, the third characteristic value may be obtained by assigning various weights to the first characteristic value and the second characteristic value. For example, it may be obtained based on Equation 21 below, but is not limited thereto.

In addition, referring to FIGS. 21A and 21B, the third characteristic value can better reflect the change of blood according to the heartbeat than the first characteristic value and the second characteristic value, and the external light Noise due to wavelength change can be reduced.

5.4 Various embodiments of heart rate acquisition method

In order to obtain a heart rate from data obtained from the above-described methods, it may be necessary to detect a periodic change according to the heart rate. For example, in order to obtain a heart rate from the obtained characteristic value, it is necessary to obtain a wavelength or frequency component most frequently included in the characteristic value.

22 is a graph obtained by extracting a frequency component from the graph of the characteristic value shown in FIG. 21(b). More specifically, FIG. 22 is a graph obtained by converting a graph of a characteristic value into a frequency domain by performing a fast Fourier transform. However, this is a fast Fourier transform specified and shown for convenience of explanation, but is not limited thereto, and a graph of the characteristic value includes a fast Fourier transform (FFT), a discrete Fourier transform (DFT), It can be transformed according to short time fourier transfom (STFT).

In addition, the graph of the characteristic value may be converted to the frequency domain, as shown in FIG. 22.

In this case, a frequency index having the highest intensity may be obtained, and the heart rate may be obtained by Equation 22 below.

For example, as illustrated in FIG. 22, when the frequency index having the highest intensity is 1.2 Hz, the heart rate may be 72 bpm.

In addition, although the graph of the characteristic value is not shown in FIG. 22, it may be converted into a frequency*measurement time domain.

In this case, an index having the highest intensity may be obtained, and the heart rate may be obtained by Equation 23 below.

For example, although not shown in FIG. 22, the index having the highest intensity is 8, and when the measurement time is 6.6 seconds, the heart rate may be 8/6.6*60, which may be 72 bpm.

In addition, in addition to the above-described example, various equations for obtaining the heart rate may be used.

Also, a preliminary heart rate may be acquired to acquire a heart rate. In this case, the preliminary heart rate may be a calculated heart rate that is a basis for obtaining a heart rate.

Prior to the description, the'preliminary heart rate' described below may mean a heart rate acquired according to a heart rate acquisition method, and may mean a heart rate that is a basis for acquiring one heart rate. For example, the at least two heart rates obtained according to the above-described heart rate acquisition method may be a first preliminary heart rate and a second preliminary heart rate, which are the basis for obtaining one final heart rate, but are not limited thereto.

In addition, the preliminary heart rate may itself be the final heart rate, and the final heart rate may be obtained based on a plurality of preliminary heart rates.

23A is a graph showing values obtained as time series data. For example, (a) of FIG. 23 may mean a color channel value obtained as time series data, but is not limited thereto, and may mean a difference value or a processed value obtained as time series data, or obtained as time series data. It can mean the value of the property.

Since the above description can be applied to a method of converting a graph representing a value obtained as time series data as shown in FIG. 23A to a wavelength domain or a frequency domain, a redundant description will be omitted.

FIG. 23B is a schematic diagram for explaining a window for a preliminary heart rate, and more specifically, a schematic diagram for explaining a window for a preliminary heart rate set to a size of 6 seconds. However, this is a specific one for convenience of description and is not limited thereto, and a window may be set in various sizes.

In this case, the window for the preliminary heart rate may mean a preset time interval, and may mean a preset number of frames, but is not limited thereto, and at least some frame groups among a plurality of frames are selected to obtain a preliminary heart rate. It can mean a window for setting.

Also, referring to (b) of FIG. 23, a plurality of acquired image frames may be set as a group by a window for a preliminary heart rate. For example, as shown in (b) of FIG. 20, an image frame acquired between 0 seconds and 6 seconds may be included in the first image frame group 2410, but is not limited thereto.

In addition, referring to FIG. 23B, the image frame group set by the window for the preliminary heart rate may at least partially overlap. For example, as shown in (b) of FIG. 23, an image frame acquired between 0 seconds and 6 seconds may be included in the first image frame group 2410, and acquired between 0.5 seconds and 6.5 seconds. The image frame may be included in the second image frame group 2420, and the image frame acquired between 1 second and 7 seconds may be included in the third image frame group 2430, and acquired between 1.5 and 7.5 seconds. The image frame may be included in the fourth image frame group 2440, but is not limited thereto.

In this case, the preliminary heart rate may be obtained for an image frame group set by a window for the preliminary heart rate. For example, the first preliminary heart rate may be obtained based on characteristic values obtained from image frames included in the first image frame group 2410.

Also, in order to obtain a preliminary heart rate, a value obtained as time series data in each image frame group may be converted into a wavelength domain or a frequency domain. For example, a value obtained as time series data in a first image frame group may be converted into first frequency data 2460, and a value obtained as time series data in a second image frame group is second frequency data 2470 The value obtained as time series data in the third image frame group can be converted into third frequency data 2480, and the value obtained as time series data in the fourth image frame group is the fourth frequency data ( 2490), but is not limited thereto.

In addition, since the above-described heart rate acquisition method may be applied to acquire the first to fourth preliminary heart rates from the first to

fourth frequency data

2460, 2470, 2480, and 2490, redundant descriptions will be omitted.

Also, a heart rate may be obtained based on a plurality of preliminary heart rates. For example, a heart rate may be obtained by performing an operation on a plurality of preliminary heart rates, and more specifically, an operation to extract the average, maximum, and minimum values of the first to fourth preliminary heart rates is performed so that the heart rate is It may be obtained, but is not limited thereto.

Also, a heart rate may be obtained based on a plurality of preliminary heart rates. For example, the heart rate may be obtained by performing an operation on the remaining heart rate except for the preliminary heart rate in which 10 digits are different among the acquired plurality of preliminary heart rates.

More specifically, when the first reserve heart rate is 72 bpm, the second reserve heart rate is 80 bpm, the third reserve heart rate is 75 bpm, and the fourth reserve heart rate is 73 bpm, except for the second reserve heart rate with a different digit of 10, the first, The heart rate may be obtained by performing calculations on the third and fourth preliminary heart rates. In this case, the operation may be an operation for extracting an average, a maximum value, a minimum value, and the like.

Also, a heart rate may be obtained based on a plurality of preliminary heart rates. For example, when the 10 digits of the acquired 4 reserve heart rates are paired and different, the calculation for the remaining heart rate excluding the preliminary heart rate pair where 10 digits of the acquired 4 reserve heart rates are different from the previously acquired heart rate. This is done so that the heart rate can be obtained.

More specifically, when the first reserve heart rate is 72 bpm, the second reserve heart rate is 80 bpm, the third reserve heart rate is 85 bpm, the fourth reserve heart rate is 73 bpm, and the previously acquired heart rate is 75 bpm, the second and third reserve heart rates are Except, the heart rate may be obtained by performing calculations on the first and fourth preliminary heart rates.

In addition, as described above, when a heart rate is acquired using a plurality of preliminary heart rates, a more robust and accurate heart rate against noise may be obtained.

5.5 Various embodiments of heart rate output

The heart rate obtained by the above-described methods may be output through a display or the like, or may be transmitted to a terminal or a server using a communication unit. Hereinafter, for convenience of explanation, such an operation will be described as an output of a heart rate.

When the heart rate is continuously measured in real time, correction may be performed on the output heart rate in order to impart stability and reliability to the measured heart rate.

In addition, when correction is performed on the output heart rate, stability and reliability may be given to the acquired and output heart rate even if a bad image is acquired from some of the acquired plurality of image frames.

Referring to FIG. 24, the method 2500 of correcting an output heart rate according to an embodiment includes obtaining a first heart rate (S2510), comparing a difference between the first heart rate and the first heart rate with a reference value (S2520). ) May be included, but is not limited thereto.

In the step of acquiring the first heart rate (S2510), since the above-described heart rate acquisition methods may be applied, a redundant description will be omitted.

In the step of comparing the difference between the first heart rate and the first time heart rate with a reference value (S2520), the first time heart rate may mean a heart rate that is acquired or output before the first heart rate is acquired. For example, when the first heart rate is acquired at 6.5 seconds, the first time heart rate may be the heart rate acquired at 6 seconds, and may be the heart rate output at 6 seconds, but is not limited thereto.

In addition, the reference value may be determined as a certain value or may be determined as a certain ratio. For example, the reference value may be set to 10, and in this case, it may be determined whether the difference between the first heart rate and the first point heart rate exceeds 10, but is not limited thereto.

In addition, when the difference between the first heart rate and the first time point heart rate is less than or equal to a reference value, the step of outputting the first heart rate as a second time point heart rate (S2531) may be performed. In this case, the second time point is the second time point. It may be later than point 1.

For example, when the first heart rate is 72 bpm, the first heart rate is 75 bpm, and the reference value is 10, the heart rate output at the second time may be 75 bpm, but is not limited thereto.

In addition, when the difference between the first heart rate and the first time point heart rate exceeds a reference value, the step of outputting the corrected heart rate from the first time point heart rate as a second time point heart rate (S2532) may be performed. In this case, The second time point may be a time later than the first time point.

In addition, in order to obtain the corrected heart rate from the first time point heart rate, an operation on the first time point heart rate may be performed. For example, an operation such as adding or subtracting a predetermined value to the heart rate at the first time point may be performed, but is not limited thereto.

For example, when the first heart rate is 72 bpm, the first heart rate is 85 bpm, and the reference value is 10, the heart rate output at the second time point may be 75 bpm obtained by adding +3 bpm to the heart rate at the first time point. , Is not limited thereto.

In addition, for example, when the first heart rate is 72 bpm, the first heart rate is 61 bpm, and the reference value is 10, the heart rate output at the second time point is 69 bpm in which -3 bpm is added to the first time heart rate. However, it is not limited thereto.

5.6 Various embodiments of heart rate signal extraction

The heart rate signal may refer to a signal that may fluctuate according to the heart rate, and may refer to a signal that may be estimated to fluctuate according to the heart rate.

In addition, a heartbeat signal may be extracted based on a plurality of acquired image frames.

First, (a) of FIG. 25 is a graph showing values obtained as time series data. For example, (a) of FIG. 25 may mean a color channel value obtained as time series data, but is not limited thereto, and may mean a difference value or a processed value obtained as time series data, or obtained as time series data. It can mean the value of the property.

In this case, the value obtained as the time series data may be extracted as a heart rate signal through a band pass filter. More specifically, the value obtained as the time series data may be extracted as a heart rate signal through a band pass filter of a frequency band or a wavelength band corresponding to the heart rate.

In addition, the frequency band or wavelength band corresponding to the heart rate may be a frequency band or wavelength band that can be generally understood, but is not limited thereto, and a frequency band or wavelength determined based on the heart rate obtained by the above-described methods. It can be a band.

For example, a typical heart rate may be 60 to 100 bpm, and a corresponding frequency band may be 1 Hz to 1.67 Hz, and a corresponding bandpass filter may be used, but is not limited thereto.

In addition, for example, when the acquired heart rate is 72 bpm, the corresponding frequency is 1.2 Hz, and a frequency band may be set based on this. More specifically, when setting the frequency band in the 0.5Hz range, the frequency band may be set in the range of 0.95Hz to 1.45Hz, and a corresponding band pass filter may be used, but is not limited thereto.

FIG. 25B is a graph of a heart rate signal obtained by extracting a value obtained from the time series data shown in FIG. 25A as a heart rate signal through a band pass filter.

Referring to FIG. 25B, it can be seen that a value obtained as time series data can be extracted as a heart rate signal through a band pass filter.

5.7 Various Examples of Heart Rate Measurement Method Using Infrared

Basically, the above-described heart rate measurement methods may be used for a heart rate measurement method using infrared rays.

In this case, the infrared ray may be an infrared ray in the near infrared region, which is a wavelength band of 750 nm to 3000 nm, but is not limited thereto. Infrared rays of the region (extreme infrared) may be used.

Referring to FIG. 26, the method 2600 for measuring heart rate using infrared rays according to an embodiment includes obtaining an image (S2610) for at least one image frame among a plurality of acquired image frames, and detecting a skin area. It may include at least some of the step S2620, the step of detecting the region of interest (S2630), and the step of processing data on the region of interest (S2640), but the present invention is not limited thereto.

In addition, since the above-described contents may be applied to the step of acquiring the image (S2610) and the step of detecting the skin area (S2620), redundant descriptions will be omitted.

In addition, since the above-described methods of detecting the ROI may be applied to the detailed operations of detecting the ROI (S2630), a redundant description will be omitted.

In addition, detecting the region of interest (S2630) may include an operation of detecting the first, second, and third regions of interest, and may be performed on at least one image frame among a plurality of acquired image frames. .

In addition, the first, second, and third regions of interest may at least partially overlap each other. For example, the first region of interest may be set to be included in the second and third regions of interest, and the second region of interest may be set to be included in the third region of interest, but is not limited thereto. The first to third regions of interest may be set to at least partially overlap each other.

In addition, the first, second, and third regions of interest may be set so as not to overlap each other. For example, the first region of interest and the second region of interest may be vertically positioned on the left cheek of the subject, and the third region of interest may be positioned on the right cheek of the subject, but are not limited thereto. The first to third regions of interest may be set so as not to overlap each other.

In addition, the detailed operations of the step of processing data for the first, second, and third ROI (S2640) are omitted because the above-described methods of processing data for the ROI may be applied. .

In addition, the processing of data on the ROI (S2640) may be performed on at least one image frame among a plurality of acquired image frames.

In addition, the step of processing data on the ROI (S2640) may be performed on the above-described first, second, and third ROIs.

In addition, IR intensity values for the first to third regions of interest may be extracted for at least one image frame among a plurality of image frames obtained to process data for the first to third regions of interest. In this case, the IR intensity value may be an average value of the IR intensity values of pixels included in the first to third ROI, and may be referred to as an average pixel value.

In addition, when the above-described method of processing data for an ROI is applied, an IR intensity value for each ROI may correspond to the above-described color channel value. For example, the IR intensity value of the first ROI may correspond to the red channel value, the IR intensity value of the second ROI may correspond to the Green channel value, and the IR intensity value of the third ROI is Blue. It may correspond to the channel value, but is not limited thereto.

Also, for example, the above-described GR value may correspond to a difference value between the IR intensity value of the second ROI and the IR intensity value of the first ROI, and the GB value described above is the IR intensity value of the second ROI. It may correspond to a difference value between the IR intensity values of the and the third ROI.

Accordingly, data may be processed based on the IR intensity value for each region of interest, and a detailed operation may follow the above-described data processing method for the region of interest.

In addition, the method 2600 for measuring heart rate using infrared rays according to an embodiment includes the steps of extracting time series data for at least some of the image frame groups among a plurality of acquired image frames (S2650) and obtaining the heart rate (S2660). At least some of them may be included, but the present invention is not limited thereto.

In this case, the step of extracting the time series data (S2650) and the step of acquiring the heart rate (S2660) may be applied to the above description, so a duplicate description will be omitted.

Referring to FIG. 27, at least two regions of interest may be set in a face region of a subject. More specifically, a first region of interest 2710, a second region of interest 2720, and a third region of interest 2730 may be set in the face region of the subject, but the present invention is not limited thereto.

In addition, IR intensity values for the first to third regions of

interest

2710, 2720, and 2730 may be extracted.

For example, an IR intensity value for the first ROI 2710 (FIG. 27A) may be extracted, and an IR intensity value for the second ROI 2720 (B )) may be extracted, and an IR intensity value (FIG. 27(c)) of the third ROI 2730 may be extracted, but is not limited thereto.

In addition, data on the first to third regions of

interest

2710, 2720, and 2730 may be processed based on the IR intensity values extracted for the first to third regions of

interest

2710, 2720, and 2730. . However, detailed operations for this have been described above, and therefore, redundant descriptions will be omitted.

Also, a characteristic value may be obtained based on data processed for the first to

third ROIs

2710, 2720, and 2730. However, detailed operations for this have been described above, and therefore, redundant descriptions will be omitted.

In addition, a heart rate may be obtained by using a characteristic value obtained based on an IR intensity value extracted for the first to

third ROIs

6. A method of obtaining a biometric index according to an embodiment.

Referring to FIG. 28, a method of obtaining a biometric index according to an embodiment includes obtaining a plurality of image frames for a subject (S2810), and obtaining first, second, and third color channel values (S2820). , Calculating a first difference value and a second difference value (S2830), obtaining a first characteristic value and a second characteristic value (S2840), and the body of the subject based on the first and second characteristic values It may include at least some of the step of determining the index (S2850).

In this case, the plurality of images may be acquired by a camera. For example, the plurality of images may be acquired by a camera such as a visible light camera or an IR camera.

In addition, the plurality of images may be acquired from a camera disposed outside. For example, the plurality of images may be images obtained from a camera such as a visible light camera or an IR camera disposed outside.

In addition, the step of obtaining the first, second, and third color channel values (S2820) may be performed on at least one image frame among a plurality of acquired image frames.

In this case, the first color channel value may mean an average pixel value for the first color channel of the image frame in which the step is performed, and the second color channel value is an average value for the second color channel of the image frame. It may mean a pixel value, and the third color channel value may mean an average pixel value for a third color channel of the image frame.

For example, the first color channel value may be a green channel value, the second color channel value may be a red channel value, and the third color channel value may be a blue channel value, but is not limited thereto. .

In addition, calculating the first difference value and the second difference value (S2830) may be performed on at least one image frame among a plurality of acquired image frames.

In this case, the first difference value may mean a difference value between a first color channel value and a second color channel value. For example, the first difference value is a first color channel value for the same image frame. This may mean a difference value between the second color channel values, but is not limited thereto.

More specifically, when the first color channel value is a green channel value and the second color channel value is a red channel value, the first difference value may be a G-R value, but is not limited thereto.

In addition, the second difference value may mean a difference value between a first color channel value and a third color channel value. For example, the second difference value is a first color channel value and a second color channel value for the same image frame. It may mean a difference value between three color channel values, but is not limited thereto.

More specifically, when the first color channel value is a green channel value and the second color channel value is a blue channel value, the second difference value may be a G-B value, but is not limited thereto.

In addition, the step of obtaining the first characteristic value and the second characteristic value (S2840) may be performed on at least some image frame groups among the plurality of obtained image frames.

For example, the first characteristic value may be obtained for a first image frame group, and the first image frame group may mean an image frame group acquired during a preset time.

Also, for example, the second characteristic value may be obtained for a second image frame group, and the second image frame group may mean an image frame group acquired during a preset time.

In addition, the first characteristic value may be obtained based on an average value of the first difference value for the first image frame group and a first difference value for image frames included in the first image frame group.

For example, the first characteristic value may be a deviation value obtained based on an average value of GR values for the first image frame group and a GR value for image frames included in the first image frame group. Not limited.

In addition, the second characteristic value may be obtained based on an average value of the first difference value for the second image frame group and a second difference value for image frames included in the second image frame group.

For example, the second characteristic value may be a deviation value obtained based on an average value of GB values for the second image frame group and a GB value for image frames included in the second image frame group. Not limited.

In addition, the first image frame group and the second image frame group may be the same, but are not limited thereto and may be different from each other.

In addition, the first characteristic value may be an average value, a standard deviation value, or the like for the first image frame group, but is not limited thereto.

In addition, the first characteristic value may be a deviation value for at least some image frames included in the first image frame group, but is not limited thereto.

In addition, the second characteristic value may be an average value, a standard deviation value, or the like for the second image frame group, but is not limited thereto.

In addition, the second characteristic value may be a deviation value for at least some image frames included in the second image frame group, but is not limited thereto.

In addition, the first and second characteristic values may be normalized. For example, the first characteristic value may be normalized using a standard deviation value for the first image frame group, but is not limited thereto.

Also, for example, the second characteristic value may be normalized using a standard deviation value for the second image frame group, but is not limited thereto.

In addition, determining the biometric index of the subject based on the first and second characteristic values (S2850) may be performed on at least some image frame groups among the plurality of acquired image frames.

In this case, the biometric index may be heart rate, blood pressure, oxygen saturation, body temperature, etc., but is not limited thereto.

In addition, the biometric index may be obtained based on the first characteristic value and the second characteristic value.

For example, the biometric index may be obtained based on a third characteristic value obtained by adding the first characteristic value and the second characteristic value, but is not limited thereto.

7. Various embodiments of a method of acquiring a plurality of biomarkers and biometric information

Referring to FIG. 29, in the method of obtaining a plurality of biomarkers and biometric information according to an exemplary embodiment, a plurality of biomarkers may be obtained based on the acquired image 3000. More specifically, as shown in FIG. 29, the first biometric index 3010, the second biometric index 3020, and the third biometric index 3030 may be obtained based on the acquired image 3000. Without limitation, at least two or more biomarkers may be obtained, for example, four biomarkers may be obtained.

In this case, the acquired image 3000 may be a visible light image acquired using a visible light camera or obtained from a visible light camera disposed outside.

Also, the acquired image 3000 may be an infrared image acquired using an infrared camera or an infrared camera disposed outside.

In addition, the first to

third biomarkers

3010, 3020, and 3030 may include at least one of heart rate, oxygen saturation, blood pressure, and body temperature, but are not limited thereto.

In addition, the first to

third biomarkers

3010, 3020, and 3030 may be obtained so as to be correlated with each other. For example, the first to

third biomarkers

3010, 3020, and 3030 may be biomarkers obtained from the same or similar state of the subject, but are not limited thereto.

In addition, the first to third

biometric indexes

3010, 3020, and 3030 may be obtained so as to be independent of each other. For example, the first to

third biomarkers

3010, 3020, and 3030 may be biomarkers obtained from different states of the subject, but are not limited thereto.

In addition, the first to third

biometric indexes

3010, 3020, and 3030 may be output at the same time. For example, when the first to third

biometric indices

3010, 3020, and 3030 are measured in real time, they may be output at the same time, but the present invention is not limited thereto.

In addition, the first to third

biometric indexes

3010, 3020, and 3030 may be output at different times. For example, after the first biometric index is output, the second biometric index may be output, and after the second biometric index is output, the third biometric index may be output.

Also, referring to FIG. 29, in the method of obtaining a plurality of biomarkers and biometric information according to an exemplary embodiment, a plurality of biometric information may be obtained based on the obtained plurality of biomarkers.

More specifically, based on the first to third

biometric indexes

3010, 3020, and 3030 obtained as shown in FIG. 29, the first biometric information 3040, the second biometric information 3050, and the third biometric The information 3060 may be obtained, but is not limited thereto.

In this case, the first to third

biometric information

3040, 3050, and 3060 may include at least one of drowsiness information, stress information, excitement level information, and emotion information, but is not limited thereto.

In addition, the first to third

biometric information

3040, 3050, 3060 may be obtained based on at least one of the first to third

biometric indexes

3010, 3020, and 3030, but is not limited thereto. Does not.

In addition, the first to third biometric information (3040, 3050, 3060) may be obtained in consideration of personal and statistical data in addition to the first to third biometric indexes (3010, 3020, 3030). The personal and statistical data may refer to personal and statistical data that can be collected by the subject such as age, sex, height, and weight, and may refer to observable personal and statistical data such as facial expressions, wrinkles, and facial colors of the subject. , Statistical data (e.g., average blood pressure in 20s, average skin color for yellow people, average height for men in 30s, average weight of men in Korea, etc.) It can also mean personal and statistical data that can be used.

In addition, the first to third

biometric information

3040, 3050, and 3060 may be obtained based on independent biometric indexes. For example, the first to third

biometric information

3040, 3050, and 3060 may be obtained based on the first to third

biometric indexes

3010, 3020, and 3030 obtained so as to be independent.

In addition, the first to third

biometric information

3040, 3050, and 3060 may be obtained based on relevant biometric indexes. For example, the first to third

biometric information

3040, 3050, and 3060 may be obtained based on the first to third

biometric indexes

3010, 3020, and 3030 obtained to be related.

In addition, when the first to third

biometric information

3040, 3050, and 3060 are obtained based on correlated biometric indexes, the accuracy of the biometric information may be improved. For example, in order to obtain hypertension information that may be included in the first to third biometric information (3040, 3050, 3060), a heart rate that may be included in the first to third biometric indexes (3010, 3020, 3030) And when the blood pressure is used, the hypertension information may be more accurate when the heart rate and the blood pressure are correlated with each other.

More specifically, if the subject measures blood pressure while exercising or is excited, and the subject measures the heart rate in a stable state and obtains hypertension information based on this, the possibility that the subject will be detected as hypertension even if the subject is not hypertensive. There is this. However, when the subject measures blood pressure and heart rate while exercising or in an excited state and acquires hypertension information based on this, the hypertension information may be more accurately obtained.

Hereinafter, a method of obtaining a plurality of biomarkers and biometric information will be described in more detail.

Referring to FIG. 30, an image frame 3110 may be obtained to acquire a plurality of biomarkers according to an embodiment.

In this case, the image frame 3110 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

In addition, the image frame 3110 may include a plurality of image frames, but is not limited thereto.

Further, referring to FIG. 30, at least one pixel value 3120 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In this case, the pixel value 3120 may refer to an intensity value of at least one pixel constituting the image frame, and more specifically, may refer to an intensity value of at least one pixel for at least one color channel. have. For example, when the image frame is a 640*480 image, the image frame may include 640*480 pixels, and a red channel pixel value, a green channel pixel value, and a blue channel pixel value for each pixel are obtained. However, the present invention is not limited thereto, and pixel values for color channels according to various color spaces may be obtained.

Also, the pixel value 3120 may be obtained only for at least some of the plurality of pixels constituting the image frame.

Further, referring to FIG. 30, at least one or more color channel values 3130 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In this case, the color channel value 3130 may mean an average value of color channel pixel values. For example, the red channel value can mean the average value of the pixel values of the red channel, the green channel value can mean the average value of the pixel values of the green channel, and the blue channel value represents the average value of the pixel values of the blue channel. It may mean, but is not limited thereto.

Although the color channel value and the processed value have been separately described throughout the specification, in order to explain a method of obtaining a plurality of biomarkers, the processed value may be described below as a concept included in the color channel value. .

In addition, the color channel value 3130 may be described including a processed value. For example, the color channel value may be described by including a G-R value that is a difference between a red channel value and a green channel value, but is not limited thereto.

Also, the color channel value 3130 may be obtained for at least one image frame among a plurality of acquired image frames.

Also, the color channel value 3130 may be obtained based on one color channel pixel value, and may be obtained based on a plurality of color channel pixel values. For example, the red channel value may be obtained based on the red channel pixel value, and the G-R value may be obtained based on the green channel pixel value and the red channel pixel value, but is not limited thereto.

Further, referring to FIG. 30, at least one or more time series data may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In this case, the time series data 3140 may be obtained based on the obtained at least one or more color channel values 3130, and may be obtained for at least some image frame groups among a plurality of acquired image frames.

More specifically, when the time series data 3140 is obtained based on at least one color channel value 3130 obtained for a first image frame, the time series data is an image frame group including the first image frame Can be obtained against.

Also, the time series data 3140 may be time series data of the color channel value 3130. For example, the time series data 3140 is time series data of a red channel value, time series data of a green channel value, time series data of a blue channel value, time series data of a Hue channel value, time series data of a GR value, or time series data of a GB value. It may include, but is not limited thereto.

In addition, it may be time series data 3140 of a characteristic value obtained based on the color channel value 3130. For example, the time series data may include a deviation value, a standard deviation value, or an average value of a color channel value for at least some image frames included in the image frame group, but is not limited thereto.

For example, the time series data 3140 may be time series data of a Red channel value, and may be time series data of a G-R value, but is not limited thereto.

Also, for example, the time series data 3140 may be time series data about a deviation value of a color channel value for at least some image frames included in an image frame group, but is not limited thereto.

In addition, the time series data 3140 may be time series data obtained based on a plurality of time series data. For example, the time series data 3140 is based on first time series data for a first image frame group including a first image frame and second time series data for a second image frame group including a second image frame. It may be time series data obtained as, but is not limited thereto.

For a more specific example, when the time series data 3140 is time series data for an image frame group including 180 image frames, the time series data 3140 is a first image including first to 18th image frames First time series data obtained for the frame group, second time series data obtained for a second image frame group including 19th to 36th image frames, and a third image frame group including 37th to 54th image frames With respect to the third time series data obtained with respect to, the fourth time series data obtained with respect to the fourth image frame group including the 55th to 72th image frames, and the fifth image frame group including the 73rd through 90th image frames The obtained fifth time series data, the sixth time series data obtained for the sixth image frame group including the 91st to the 108th image frames, and the seventh image frame group including the 109th to 126th image frames The eighth time series data obtained for the eighth image frame group including the seventh time series data, 127th to 144th image frames, and the ninth time series data obtained for the ninth image frame group including the 145th to 162th image frames It may be obtained based on the time series data and the tenth time series data obtained for the tenth image frame group including the 163th to 180th image frames.

Further, referring to FIG. 30, a feature 3150 may be obtained to obtain a plurality of biomarkers according to an embodiment.

In this case, the feature 3150 may mean a mathematical and physical feature of the acquired time series data. For example, the characteristic is the frequency component of the acquired time series data, the maximum value, the average of the maximum value, the minimum value, the average of the minimum value, the difference between the maximum value and the minimum value, the difference between the maximum value average and the minimum value average, AC value, DC Value, average value, inflection point, first derivative data, second derivative data, and slope at a specific point in time, and other mathematical features, and physical characteristics such as blood change, blood change rate, blood vessel change amount, blood vessel change rate, etc. It may mean a feature, but is not limited thereto.

In addition, the feature 3150 may mean a mathematical and physical feature between a plurality of time series data. For example, the feature may mean a mathematical feature such as a time difference between a plurality of acquired time series data, a time difference between a maximum value, a time difference between a minimum value, a time difference between inflection points, and so on, and PTT (Pulse Transit Time) , It may mean physical characteristics, such as a difference in the rate of change of blood and a difference in time of change of blood vessels by blood, but is not limited thereto.

In addition, the feature 3150 may be obtained for at least some image frame groups among a plurality of acquired image frames.

Also, the feature 3150 may be obtained based on at least one time series data 3140. For example, the feature 3150 may be obtained based on one time series data, and may be obtained based on at least two time series data, but is not limited thereto.

Further, referring to FIG. 30, a biometric index 3160 may be obtained to obtain a plurality of biometric indexes according to an embodiment.

In this case, the biometric index 3160 may include, but is not limited to, heart rate, oxygen saturation, blood pressure, and body temperature.

In addition, the biometric index 3160 may be obtained based on different characteristics. For example, heart rate may be obtained based on a frequency component of time series data, blood pressure may be obtained based on a PTT value between two time series data, and oxygen saturation is an AC value and DC value of each of the two time series data. It may be obtained based on the value, but is not limited thereto.

In addition, a plurality of the biometric indexes 3160 may be obtained. For example, heart rate, oxygen saturation, and blood pressure may be obtained, but the present invention is not limited thereto.

In addition, the biometric index 3160 may be obtained for at least some image frame groups among a plurality of acquired image frames.

Also, the image frame groups that are the basis for obtaining the plurality of biometric indexes 3160 may be identical to each other. For example, heart rate, oxygen saturation, and blood pressure may be obtained based on an image frame group including first to 180th image frames, but are not limited thereto.

Also, image frame groups that are the basis for obtaining the plurality of biometric indexes 3160 may be different from each other. For example, the heart rate may be obtained based on a first image frame group including the first to 90th image frames, and the oxygen saturation is based on a second image frame group including the 91st to 180th image frames. The blood pressure may be acquired, and the blood pressure may be acquired based on the third image frame group including the 181-th image frames, but is not limited thereto.

In addition, the image frame groups that are the basis for obtaining the plurality of biometric indexes 3160 may at least partially overlap each other. For example, the heart rate may be obtained based on a first image frame group including first to 180th image frames, and the oxygen saturation is based on a second image frame group including 30th to 100th image frames. The shale may be acquired, and the shale may be acquired based on the third image frame group including the 10th to 180th image frames, but is not limited thereto.

In addition, the number of image frames included in the image frame group as a basis for obtaining the plurality of biometric indexes 3160 may be the same or may be different.

7.1 Method for obtaining a plurality of biomarkers according to an embodiment

Referring to FIG. 31, an image frame 3210 may be obtained to acquire a plurality of biomarkers according to an embodiment.

In this case, the image frame 3210 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

In addition, the image frame 3210 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 31, according to an embodiment, at least one region of interest 3220 may be set to obtain a plurality of biomarkers. More specifically, a first region of interest, a second region of interest, a third region of interest, and a fourth region of interest may be set.

In this case, the first region of interest may be a region of interest for obtaining oxygen saturation, the second region of interest may be a region of interest for obtaining a heart rate, and the third region of interest is a region of interest for obtaining blood pressure. The fourth region of interest may be a region, and the fourth region of interest may be a region of interest for acquiring body temperature, but is not limited thereto.

In addition, the first to fourth regions of interest may be the same or different from each other.

In addition, the first to fourth regions of interest may at least partially overlap each other.

Also, the size and area of the first to fourth regions of interest may be set based on a biometric index to be acquired. For example, the second region of interest for acquiring a heart rate may be largely set to include a cheek region of a subject in order to better detect a change in blood due to a heart rate, and the third region of interest for acquiring blood pressure The region may be set vertically and horizontally to include the cheek region of the subject in order to better detect the minute blood flow velocity, but is not limited thereto, and the first to fourth regions of interest may be set to various sizes and various regions. I can.

Also, referring to FIG. 31, in order to obtain a plurality of biomarkers, at least one pixel value 3230 may be obtained according to an embodiment.

Also, the pixel value 3230 may be obtained for at least one image frame among a plurality of acquired image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained for the first region of interest, and the second region of interest may be obtained in an RGB color space. At least part of a red channel pixel value, a green channel pixel value, and a blue channel pixel value may be obtained, and a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space for the third ROI At least some of them may be obtained, and a Hue channel pixel value, a saturation channel pixel value, and a Value channel pixel value according to the HSV color space may be obtained for the fourth ROI, but is not limited thereto.

Also, referring to FIG. 31, according to an embodiment, at least one or more color channel values 3240 may be obtained in order to obtain a plurality of biomarkers.

In this case, the color channel value 3240 may mean an average value of the color channel pixel values or a processed value, but a detailed description thereof will be omitted as described above.

In addition, the color channel value 3240 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least two color channel values may be obtained for the first ROI for obtaining oxygen saturation.

For example, for the first region of interest for obtaining oxygen saturation, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to an HSV color space, a saturation channel value, and a value channel value. At least two color channel values may be obtained among color channel values such as, but are not limited thereto.

In addition, for example, for the first region of interest for obtaining oxygen saturation, a GR value that is a difference between a red channel value and a green channel value, a GB value that is a difference between a green channel value and a blue channel value, a Hue channel value and a Value channel At least two color channel values among color channel values such as an HV value, which is a value difference value, may be obtained, but the present invention is not limited thereto.

In addition, the at least two color channel values may be selected in consideration of absorbance of hemoglobin and oxyhemoglobin.

For example, a Blue channel in which the absorbance of oxyhemoglobin is higher than that of hemoglobin and a Red channel in which the absorbance of oxyhemoglobin is lower than that of hemoglobin may be selected. It may be selected as a blue channel value, but is not limited thereto.

In addition, at least two color channel values may be obtained for the second ROI for obtaining a heart rate.

For example, for the second region of interest for obtaining a heart rate, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to the HSV color space, a saturation channel value, a Value channel value, etc. At least two color channel values among the color channel values of may be obtained, but are not limited thereto.

In addition, for example, a GR value that is a difference between a red channel value and a green channel value, a GB value that is a difference between a green channel value and a blue channel value, and a Hue channel value and a Value channel value for the second ROI for obtaining a heart rate. At least two color channel values among color channel values such as an HV value that is a difference value of may be obtained, but are not limited thereto.

In addition, the at least two color channel values may be selected to reduce noise due to motion and noise due to external light.

For example, in order to reduce noise, the GR value, which is the difference between the green channel value, which is relatively absorbed by hemoglobin and oxygen hemoglobin, and the red channel value, which is relatively less absorbed by hemoglobin and oxygen hemoglobin, and relatively hemoglobin, A value of GB, which is a difference value between a value of a green channel absorbed by oxygen hemoglobin and a blue channel value absorbed relatively less by hemoglobin and oxygen hemoglobin, may be selected, but is not limited thereto.

In addition, for example, in order to reduce noise, a GR value that is a difference between a green channel value that reflects a relatively large change due to heartbeat and a red channel value and a blue channel value that reflects relatively little change due to heartbeat, and The GB value may be selected, but is not limited thereto.

In addition, at least two color channel values may be obtained for the third ROI for obtaining blood pressure.

For example, for the third region of interest for obtaining blood pressure, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to the HSV color space, a saturation channel value, a value channel value, etc. At least two color channel values among the color channel values of may be obtained, but are not limited thereto.

In addition, for example, for the third region of interest for obtaining blood pressure, a GR value that is a difference between a red channel value and a green channel value, a GB value that is a difference between a green channel value and a blue channel value, and a Hue channel value and a Value channel value. At least two color channel values among color channel values such as an HV value that is a difference value of may be obtained, but are not limited thereto.

For example, to reduce noise, the GR value, which is the difference between the green channel value, which is relatively absorbed by hemoglobin and oxyhemoglobin, and the red channel value, which is relatively less absorbed by hemoglobin and oxyhemoglobin, and relatively hemoglobin, A value of GB, which is a difference between a value of a green channel absorbed by oxygen hemoglobin and a blue channel value absorbed relatively less by hemoglobin and oxygen hemoglobin, may be selected, but is not limited thereto.

In addition, at least two color channel values may be obtained for the fourth ROI for obtaining body temperature.

For example, for the fourth ROI for obtaining body temperature, a Red channel value, a Green channel value, a Blue channel value according to an RGB color space, a Hue channel value according to an HSV color space, a saturation channel value, a Value channel value, etc. At least two color channel values among the color channel values of may be obtained, but are not limited thereto.

In addition, for example, for the fourth ROI for acquiring body temperature, a GR value that is a difference between a red channel value and a green channel value, a GB value that is a difference between a green channel value and a blue channel value, and a Hue channel value and a Value channel value. At least two color channel values among color channel values such as an HV value that is a difference value of may be obtained, but are not limited thereto.

In addition, the values of the at least two color channels may be selected in consideration of the characteristics of body temperature and the skin color of the subject.

For example, a saturation (brightness) channel value associated with the subject's body temperature and a Hue (color) channel value correlated with the subject's skin color may be selected, but are not limited thereto.

Also, referring to FIG. 31, at least one or more time series data 3250 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In addition, the time series data 3250 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least two time series data may be obtained for the first region of interest for obtaining oxygen saturation. For example, first time series data and second time series data may be obtained for the first ROI for obtaining oxygen saturation.

In this case, the first time series data may be obtained based on a color channel value obtained for the first ROI. For example, when a color channel value obtained for the first ROI for obtaining an oxygen saturation is a red channel value, the first time series data may be obtained based on a red channel value, but is not limited thereto.

In addition, the first time series data may be obtained for at least some image frame groups among a plurality of acquired image frames.

For example, when the first time series data is obtained based on a Red channel value obtained for a first image frame, the first time series data is obtained for a first image frame group including the first image frame. However, it is not limited thereto.

Also, the second time series data may be obtained based on a color channel value obtained for the first ROI. For example, when a color channel value obtained for the first ROI for obtaining an oxygen saturation is a blue channel value, the second time series data may be obtained based on a blue channel value, but is not limited thereto.

Also, the second time series data may be obtained for at least some image frame groups among a plurality of acquired image frames.

For example, when the second time series data is obtained based on a blue channel value obtained for a second image frame, the second time series data is obtained for a second image frame group including the second image frame. However, it is not limited thereto.

In addition, the first and second image frame groups may be identical to each other, may be different from each other, and may at least partially overlap each other.

Also, at least one time series data may be obtained for the second ROI for obtaining a heart rate. For example, third time series data may be obtained for the second ROI for obtaining a heart rate.

In this case, the third time series data may be obtained based on a color channel value obtained for the second ROI. For example, when a color channel value obtained for the second ROI for obtaining a heart rate is a GR value and a GB value, the third time series data may be obtained based on a GR value and a GB value, but is limited thereto. It doesn't work.

In addition, the third time series data may be obtained for at least some image frame groups among a plurality of acquired image frames.

For example, when the third time series data is obtained based on a GR value and a GB value obtained for a third image frame, the third time series data is for a third image frame group including the third image frame. It may be obtained, but is not limited thereto.

In addition, the third time series data may be obtained based on a characteristic value obtained for the second ROI. For example, the third time series data may be obtained based on a characteristic value obtained based on a GR value obtained for the second ROI and a characteristic value obtained based on a GB value, but is not limited thereto. .

Also, at least one time series data may be obtained for the third ROI for obtaining blood pressure. For example, fourth time series data may be obtained for the third ROI for obtaining blood pressure.

In this case, the fourth time series data may be obtained based on a color channel value obtained for the third ROI. For example, when a color channel value obtained for a third ROI for obtaining a blood pressure is a GR value and a GB value, the fourth time series data may be obtained based on the GR value and the GB value, but is not limited thereto. Does not.

In addition, the fourth time series data may be obtained for at least some image frame groups among a plurality of acquired image frames.

For example, when the fourth time series data is obtained based on a GR value and a GB value obtained for a fourth image frame, the fourth time series data is for a fourth image frame group including the fourth image frame. It may be obtained, but is not limited thereto.

In addition, the fourth time series data may be obtained based on a characteristic value obtained for the third region of interest. For example, the fourth time series data may be obtained based on a characteristic value obtained based on a GR value obtained for the third ROI and a characteristic value obtained based on a GB value, but is not limited thereto. .

In addition, the first, second, third and fourth image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may at least partially overlap each other.

Also, referring to FIG. 31, at least one biometric index acquisition model 3251 may be used to obtain a plurality of biometric indexes according to an embodiment.

However, with respect to the biometric index acquisition model 3251, the above-described redundant description will be omitted.

In addition, the biometric index acquisition model 3251 may be used to acquire body temperature.

More specifically, the biometric index acquisition model 3251 may use a color channel value acquired for the fourth ROI for acquiring body temperature as an input value. For example, the biometric index acquisition model 3251 may use a Hue channel value and a saturation channel value acquired for the fourth ROI as input values, but are not limited thereto.

Also, the input value of the biometric index acquisition model 3251 may be a color channel value, a characteristic value, an average of color channel values for an image frame group, etc., but is not limited thereto.

Also, referring to FIG. 31, at least one feature 3260 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In addition, at least one or more of the features 3260 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least one feature may be obtained for the first region of interest for obtaining oxygen saturation. For example, a first feature may be obtained for the first region of interest for obtaining oxygen saturation.

In this case, the first characteristic may be obtained based on a color channel value or time series data obtained for the first ROI. For example, when a color channel value obtained for the first ROI is a red channel value and a blue channel value, the first characteristic may be obtained based on the red channel value and the blue channel value, but is limited thereto. It doesn't work.

Also, for example, the first feature may be obtained based on first time series data and second time series data acquired for the first ROI, but is not limited thereto.

In addition, the first characteristic may be a characteristic for obtaining oxygen saturation. For example, the first characteristic may be an AC value and a DC value obtained based on the first time series data, but are not limited thereto.

In addition, for example, the first characteristic is a difference between a local maximum value average and a minimum value average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data. It may include at least one or more of a difference between the obtained maximum value average and the minimum value average and an average value obtained based on the second time series data, but is not limited thereto.

In addition, the first feature may include a plurality of features. For example, the first feature is a difference between a local maximum value average and a minimum value average obtained based on the first time series data, an average value obtained based on the first time series data, and the second time series data. It may include at least two or more of the difference between the maximum value average and the minimum value average and the average value obtained based on the second time series data, but is not limited thereto.

In addition, at least one feature may be acquired for the second ROI for acquiring a heart rate. For example, a second characteristic may be obtained for the second region of interest for obtaining a heart rate.

In this case, the second characteristic may be obtained based on a color channel value or time series data obtained for the second ROI. For example, when a color channel value obtained for the second ROI is a G-R value and a G-B value, the second characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the second feature may be obtained based on third time series data obtained for the second ROI, but is not limited thereto.

In addition, the second characteristic may be a characteristic for acquiring a heart rate. For example, the second characteristic may be a frequency value obtained based on the third time series data, a wavelength value, and a value for a number of repeated periods during a measurement time, but is not limited thereto.

In addition, at least one or more features may be obtained for the third region of interest for obtaining blood pressure. For example, a third feature may be obtained for the third region of interest for obtaining blood pressure.

In this case, the third characteristic may be obtained based on a color channel value or time series data obtained for the third ROI. For example, when a color channel value obtained for the third ROI is a G-R value and a G-B value, the third characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the third feature may be obtained based on fourth time series data obtained for the third ROI, but is not limited thereto.

In addition, the third characteristic may be a characteristic for obtaining blood pressure. For example, the third characteristic may be a slope value, a maximum value, a minimum value, an average of a maximum value, an average of a minimum value, an average of the maximum value, and a difference value between the average of the minimum value, etc., obtained based on the fourth time series data. , Is not limited thereto.

In addition, the third feature may include a plurality of features. For example, the third characteristic includes at least two of the difference between the slope value, the maximum value, the minimum value, the average of the local maximum, the average of the minimum value, the average of the local maximum, and the difference between the average of the minimum value obtained based on the fourth time series data. It may include, but is not limited thereto.

Further, at least one or more features may be acquired for the fourth ROI for acquiring body temperature. For example, a fourth characteristic may be obtained for the fourth region of interest for obtaining body temperature.

In this case, the fourth characteristic may be obtained based on a color channel value obtained for the fourth ROI or an output value of a biometric index acquisition model. For example, when a color channel value obtained for the fourth ROI is a Hue channel value and a saturation channel value, the fourth characteristic may be obtained based on the Hue channel value and the Saturation value, but is not limited thereto. Does not.

Also, for example, the fourth feature may be acquired based on an output value of the biometric index acquisition model, but is not limited thereto.

In addition, the fourth characteristic may be a characteristic for acquiring body temperature. For example, the fourth characteristic may be a skin region, a skin temperature, and the like, but is not limited thereto.

Also, referring to FIG. 31, at least one biometric index 3270 may be obtained in order to obtain a plurality of biometric indexes according to an embodiment.

More specifically, each biometric index may be obtained based on a corresponding feature. For example, oxygen saturation may be obtained based on the first characteristic, heart rate may be obtained based on the second characteristic, blood pressure may be obtained based on the third characteristic, and body temperature It may be acquired based on the fourth feature, but is not limited thereto.

In this case, the above-described equations may be used to obtain oxygen saturation based on the first characteristic, and redundant descriptions will be omitted.

In addition, the above-described equations may be used to obtain the heart rate based on the second characteristic, and redundant descriptions will be omitted.

In addition, the above-described equations may be used to obtain blood pressure based on the third feature, and redundant descriptions will be omitted.

In addition, the above-described equations may be used to obtain body temperature based on the fourth characteristic, and redundant descriptions will be omitted.

In addition, when there are a plurality of biometric indexes 3270, each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be obtained 6 seconds after measurement, and blood pressure may be obtained 8 seconds after measurement, but are not limited thereto.

In addition, when there are a plurality of the biometric indexes 3270, each biometric index may be obtained based on each image frame group. For example, the oxygen saturation degree may be obtained based on the fifth image frame group, the heart rate may be obtained based on the sixth image frame group, and the blood pressure may be obtained based on the seventh image frame group.

In addition, each image frame group for obtaining the biometric index 3270 may be the same, but is not limited thereto, may be different from each other, and may at least partially overlap with each other. For example, the fifth, sixth, and seventh image frame groups may be identical to each other, may be different from each other, and may at least partially overlap each other.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3270. For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

However, since the above description may be applied to a method of obtaining a heart rate or a biometric index using the preliminary heart rate or the preliminary biometric index, a duplicate description will be omitted.

In addition, the number of preliminary biomarkers for obtaining the biometric index 3270 may be the same for each biometric index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the oxygen saturation level and the preliminary oxygen saturation level and the preliminary blood pressure for acquiring the blood pressure may be at least two, but are not limited thereto.

7.2 Method for obtaining a plurality of biomarkers according to an embodiment

Referring to FIG. 32, an image frame 3310 may be obtained to obtain a plurality of biomarkers according to an embodiment.

In this case, the image frame 3310 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

In addition, the image frame 3310 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 32, according to an embodiment, at least one region of interest 3320 may be set to obtain a plurality of biomarkers. More specifically, a first region of interest, a second region of interest, and a third region of interest may be set.

In this case, the first region of interest may be a region of interest for acquiring oxygen saturation and heart rate, and the second region of interest and the third region of interest may be regions of interest for obtaining blood pressure.

In addition, the first to third regions of interest may be the same or different from each other.

In addition, the first to third regions of interest may at least partially overlap each other.

In addition, the first to third regions of interest may be set in size and region based on a biometric index to be acquired. For example, in order to obtain heart rate and oxygen saturation, the first region of interest may be sized to include a cheek region of the subject in order to better detect changes in blood due to heart rate, and to obtain blood pressure. The second and third regions of interest may be set along a direction of blood flow in order to reflect the flow of blood well, but the present invention is not limited thereto, and the first to third regions of interest may be set to various sizes and various regions. I can.

Also, referring to FIG. 32, at least one pixel value 3330 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

Also, the pixel value 3330 may be obtained for at least one image frame among a plurality of acquired image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained for the first region of interest, and the second region of interest may be obtained in an RGB color space. At least part of a red channel pixel value, a green channel pixel value, and a blue channel pixel value may be obtained, and a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space for the third ROI At least some of them may be obtained, but the present invention is not limited thereto.

* Also, referring to FIG. 32, at least one color channel value 3340 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In this case, the color channel value 3340 may mean an average value of the color channel pixel values or a processed value, but a detailed description thereof will be omitted as described above.

In addition, the color channel value 3340 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least two or more color channel values may be obtained for the first ROI for obtaining oxygen saturation and heart rate.

For example, for the first region of interest, among color channel values such as a red channel value, a green channel value, a blue channel value according to an RGB color space, a hue channel value according to an HSV color space, a saturation channel value, and a value channel value. At least two or more color channel values may be obtained, but the present invention is not limited thereto.

In addition, for example, the GR value that is the difference between the red channel value and the green channel value, the GB value that is the difference between the green channel value and the blue channel value, and the HV value that is the difference between the Hue channel value and the Value channel value for the first region of interest. At least two or more color channel values among color channel values such as, etc. may be obtained, but the present invention is not limited thereto.

In addition, in order to obtain oxygen saturation, the at least two color channel values may be selected in consideration of absorbance of hemoglobin and oxygen hemoglobin.

In addition, in order to obtain a heart rate, the at least two color channel values may be selected to reduce noise due to movement and noise due to external light.

Also, at least two color channel values may be obtained for the second and third ROIs for obtaining blood pressure.

For example, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to the HSV color space, a saturation channel value, and a value for the second and third regions of interest for obtaining blood pressure. At least two color channel values among color channel values such as a channel value may be obtained, but are not limited thereto.

In addition, for example, for the second and third regions of interest for acquiring blood pressure, a GR value that is a difference between a red channel value and a green channel value, a GB value that is a difference between a green channel value and a blue channel value, a Hue channel value, and At least two color channel values among color channel values, such as an HV value, which is a difference value between the Value channel values, may be obtained, but are not limited thereto.

Also, referring to FIG. 32, at least one or more time series data 3350 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In addition, the time series data 3350 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least two or more time series data may be obtained for the first region of interest for obtaining oxygen saturation and heart rate. For example, first time series data and second time series data may be obtained to obtain an oxygen saturation degree for the first region of interest, and third time series data may be obtained to obtain a heart rate.

In this case, the first time series data may be obtained based on a color channel value obtained for the first ROI. For example, when a color channel value obtained for the first ROI is a red channel value, the first time series data may be obtained based on a red channel value, but is not limited thereto.

Also, the second time series data may be obtained based on a color channel value obtained for the first ROI. For example, when a color channel value obtained for the first ROI is a blue channel value, the second time series data may be obtained based on a blue channel value, but is not limited thereto.

Also, the third time series data may be obtained based on a color channel value obtained for the first ROI. For example, when a color channel value obtained for the first ROI is a G-R value and a G-B value, the third time series data may be obtained based on a G-R value and a G-B value, but is not limited thereto.

For example, when the third time series data is obtained based on a GR value and a GB value obtained for a third image frame, the third time series data is included in a third image frame group including the third image frame. It may be obtained for, but is not limited thereto.

In addition, the third time series data may be obtained based on a characteristic value obtained for the first ROI. For example, the third time series data may be obtained based on a characteristic value obtained based on a GR value obtained for the first ROI and a characteristic value obtained based on a GB value, but is not limited thereto. .

In addition, the first, second, and third image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may at least partially overlap each other.

In addition, in order to obtain a blood pressure, at least one time series data may be obtained for each of the second region of interest and the third region of interest. For example, fourth time series data may be obtained for the second region of interest, and fifth time series data may be obtained for the third region of interest, but the present invention is not limited thereto.

In this case, the fourth time series data may be obtained based on a color channel value obtained for the second ROI. For example, when a color channel value obtained for the second ROI is a G-R value and a G-B value, the fourth time series data may be obtained based on a G-R value and a G-B value, but is not limited thereto.

Also, the fourth time series data may be obtained based on a characteristic value obtained for the second ROI. For example, the fourth time series data may be obtained based on a feature value obtained based on a GR value obtained for the second ROI and a feature value obtained based on a GB value, but is not limited thereto. .

Also, the fifth time series data may be obtained based on a color channel value obtained for the third ROI. For example, when a color channel value obtained for the third ROI is a G-R value and a G-B value, the fifth time series data may be obtained based on a G-R value and a G-B value, but is not limited thereto.

In addition, the fifth time series data may be obtained for at least some image frame groups among a plurality of acquired image frames.

For example, when the fifth time series data is obtained based on a GR value and a GB value obtained for a fifth image frame, the fifth time series data is for a fifth image frame group including the fifth image frame. It may be obtained, but is not limited thereto.

In addition, the fifth time series data may be obtained based on a characteristic value obtained for the third ROI. For example, the fifth time series data may be obtained based on a characteristic value obtained based on a GR value obtained for the third ROI and a characteristic value obtained based on a GB value, but is not limited thereto. .

In addition, the first, second, third, fourth, and fifth image frame groups may be identical to each other, but are not limited thereto, may be different from each other, and may at least partially overlap each other.

Also, referring to FIG. 32, at least one feature 3360 may be obtained to obtain a plurality of biomarkers according to an embodiment.

In addition, at least one or more of the features 3360 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least one feature may be obtained for the first region of interest for obtaining oxygen saturation and heart rate. For example, a first characteristic of the first region of interest may be obtained to obtain an oxygen saturation degree, and a second characteristic of the first region of interest may be obtained to obtain a heart rate.

Also, the second characteristic may be obtained based on a color channel value or time series data obtained for the first ROI. For example, when a color channel value obtained for the first ROI is a G-R value and a G-B value, the second characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the second feature may be obtained based on third time series data obtained for the first ROI, but is not limited thereto.

In addition, at least one feature may be obtained for the second and third ROIs for obtaining blood pressure. For example, in order to obtain a blood pressure, a third characteristic may be obtained for the second and third ROIs.

In this case, the third characteristic may be obtained based on color channel values or time series data obtained for the second and third ROI. For example, when the color channel values obtained for the second and third regions of interest are the GR value and the GB value, the third characteristic may be obtained based on the GR value and the GB value, but is not limited thereto. Does not.

In addition, for example, the third feature may be obtained based on fourth time series data obtained with respect to the second region of interest and fifth time series data obtained with respect to the third region of interest, but is not limited thereto.

In addition, the third characteristic may be a characteristic for obtaining blood pressure. For example, the third characteristic includes a slope value, a maximum value, a minimum value, an average of a local maximum, an average of a minimum value, an average of the maximum value, and a difference value between the average of the minimum value, etc., obtained based on the fourth and fifth time series data. It may be, but is not limited thereto.

In addition, for example, the third characteristic may be a time difference between the fourth and fifth time series data, a time difference between a maximum value, a time difference between a minimum value, a time difference between inflection points, etc., but is not limited thereto. Does not.

In addition, the third feature may include a plurality of features. For example, the third feature may include at least two or more of the above-described features, but is not limited thereto.

Also, referring to FIG. 32, at least one biometric index 3370 may be obtained in order to obtain a plurality of biometric indexes according to an embodiment.

More specifically, each biometric index may be obtained based on a corresponding feature. For example, oxygen saturation may be obtained based on the first characteristic, heart rate may be obtained based on the second characteristic, and blood pressure may be obtained based on the third characteristic, but is not limited thereto. Does not.

In addition, when there are a plurality of the biometric indexes 3370, each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be obtained 6 seconds after measurement, and blood pressure may be obtained 8 seconds after measurement, but are not limited thereto.

In addition, when there are a plurality of biometric indexes 3370, each biometric index may be obtained based on each image frame group. For example, the oxygen saturation degree may be obtained based on the sixth image frame group, the heart rate may be obtained based on the seventh image frame group, and the blood pressure may be obtained based on the eighth image frame group.

In addition, each image frame group for obtaining the biometric index 3370 may be the same, but is not limited thereto, may be different from each other, and may at least partially overlap with each other. For example, the sixth, seventh, and eighth image frame groups may be identical to each other, may be different from each other, and may at least partially overlap each other.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3370. For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

In addition, the number of preliminary biomarkers for acquiring the biometric index 3370 may be the same for each biometric index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the oxygen saturation level and the preliminary oxygen saturation level and the preliminary blood pressure for acquiring the blood pressure may be at least two, but are not limited thereto.

7.3 Method for obtaining a plurality of biomarkers according to an embodiment

Referring to FIG. 33, an image frame 3410 may be obtained to acquire a plurality of biomarkers according to an embodiment.

In this case, the image frame 3410 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

In addition, the image frame 3410 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 33, according to an embodiment, at least one region of interest 3420 may be set to obtain a plurality of biomarkers. More specifically, a first region of interest and a second region of interest may be set.

In this case, the first region of interest may be an ROI for acquiring oxygen saturation, heart rate, and blood pressure, and the first and second ROI may be an ROI for acquiring blood pressure.

In addition, the first and second regions of interest may be the same or different from each other.

In addition, the first and second regions of interest may at least partially overlap each other.

In addition, the first and second regions of interest may be set in size and region based on a biometric index to be acquired. For example, in order to obtain heart rate and oxygen saturation, the first region of interest may be sized to include a cheek region of a subject in order to better detect changes in blood due to heart rate, and to obtain blood pressure The first and second regions of interest may be set along a direction of blood flow in order to better reflect the flow of blood, but the first and second regions of interest are not limited thereto, and the first and second regions of interest may be set to various sizes and various regions. I can.

Also, referring to FIG. 33, at least one pixel value 3430 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

Also, the pixel value 3430 may be obtained for at least one image frame among a plurality of acquired image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained for the first region of interest, and the second region of interest may be obtained in an RGB color space. At least some of the corresponding red channel pixel value, green channel pixel value, and blue channel pixel value may be obtained, but is not limited thereto.

Also, referring to FIG. 33, at least one color channel value 3440 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In this case, the color channel value 3440 may mean an average value of the color channel pixel values, or may mean a processed value, but a detailed description thereof will be omitted as described above.

In addition, the color channel value 3440 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least two or more color channel values may be obtained for the first ROI for obtaining oxygen saturation, heart rate, and blood pressure.

In addition, at least two color channel values may be obtained for the first and second ROI for obtaining blood pressure.

For example, for the first and second regions of interest for obtaining blood pressure, a red channel value, a green channel value, a blue channel value according to an RGB color space, a Hue channel value according to the HSV color space, a saturation channel value, and Value At least two color channel values among color channel values such as a channel value may be obtained, but are not limited thereto.

In addition, for example, for the first and second regions of interest for obtaining blood pressure, a GR value that is a difference between a red channel value and a green channel value, a GB value that is a difference between a green channel value and a blue channel value, a Hue channel value, and At least two color channel values among color channel values, such as an HV value, which is a difference value between the Value channel values, may be obtained, but are not limited thereto.

Also, referring to FIG. 33, at least one time heat data 3450 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In addition, the time series data 3450 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least two or more time series data may be obtained for the first region of interest. For example, first time series data and second time series data may be obtained to obtain oxygen saturation for the first region of interest, and third time series data may be obtained to obtain heart rate and blood pressure.

In addition, the first time sequence data may be obtained for at least some image frame groups among a plurality of acquired image frames.

In addition, in order to obtain blood pressure, at least one time series data may be obtained for the second region of interest, and third time series data obtained for the first region of interest and a fourth time series data obtained for the second region of interest Blood pressure may be obtained based on time series data.

Further, referring to FIG. 33, at least one feature 3460 may be obtained to obtain a plurality of biomarkers according to an embodiment.

In addition, at least one or more of the features 3460 may be obtained in consideration of a biometric index to be obtained.

More specifically, at least one feature may be obtained for the first region of interest. For example, a first characteristic of the first region of interest may be obtained to obtain an oxygen saturation degree, and a second characteristic of the first region of interest may be obtained to obtain a heart rate.

In addition, at least one feature may be obtained for the first and second ROIs for obtaining blood pressure. For example, in order to obtain a blood pressure, a third characteristic may be obtained for the first and second ROI.

In this case, the third characteristic may be obtained based on color channel values or time series data obtained for the first and second ROI. For example, when the color channel values obtained for the first and second ROI are the GR value and the GB value, the third characteristic may be obtained based on the GR value and the GB value, but is not limited thereto. Does not.

Also, for example, the third feature may be obtained based on third time series data obtained for the first region of interest and fourth time series data obtained for the second region of interest, but is not limited thereto.

In addition, the third characteristic may be a characteristic for obtaining blood pressure. For example, the third characteristic includes a slope value, a maximum value, a minimum value, an average of a local maximum, an average of a minimum value, an average of the maximum value, and a difference value between the average of the minimum value, etc. It may be, but is not limited thereto.

In addition, for example, the third characteristic may be a time difference between the third and fourth time series data, a time difference between a maximum value, a time difference between a minimum value, a time difference between inflection points, etc., but is not limited thereto. Does not.

Also, referring to FIG. 33, at least one biometric index 3470 may be obtained in order to obtain a plurality of biometric indexes according to an embodiment.

In addition, when there are a plurality of the biomarkers 3470, each biomarker may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be obtained 6 seconds after measurement, and blood pressure may be obtained 8 seconds after measurement, but are not limited thereto.

In addition, when there are multiple biometric indexes 3470, each biometric index may be obtained based on each image frame group. For example, the oxygen saturation degree may be obtained based on the fifth image frame group, the heart rate may be obtained based on the sixth image frame group, and the blood pressure may be obtained based on the seventh image frame group.

In addition, each image frame group for obtaining the biometric index 3470 may be the same, but is not limited thereto, may be different from each other, and may at least partially overlap with each other. For example, the fifth, sixth, and seventh image frame groups may be identical to each other, may be different from each other, and may at least partially overlap each other.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3470. For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

In addition, the number of preliminary biomarkers for obtaining the biometric index 3470 may be the same for each biometric index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the oxygen saturation level and the preliminary oxygen saturation level and the preliminary blood pressure for acquiring the blood pressure may be at least two, but are not limited thereto.

7.4 Method for obtaining a plurality of biomarkers according to an embodiment

Referring to FIG. 34, an image frame 3510 may be obtained to acquire a plurality of biomarkers according to an embodiment.

In this case, the image frame 3510 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

In addition, the image frame 3510 may include a plurality of image frames, but is not limited thereto.

Also, referring to FIG. 34, according to an embodiment, at least one region of interest 3520 may be set to obtain a plurality of biomarkers. More specifically, a first region of interest may be set.

In this case, the first region of interest may be a region of interest for obtaining oxygen saturation, heart rate, and blood pressure.

In addition, the size and area of the first region of interest may be set based on a biometric index to be acquired. For example, in order to obtain a heart rate and oxygen saturation, the first region of interest may be set in size to include a cheek region of a subject in order to better detect a change in blood due to a heart rate, but is not limited thereto. The first region of interest may be set to various sizes and various regions.

Also, referring to FIG. 34, at least one pixel value 3530 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

Also, the pixel value 3530 may be obtained for at least one image frame among a plurality of acquired image frames.

More specifically, at least some of a red channel pixel value, a green channel pixel value, and a blue channel pixel value according to an RGB color space may be obtained for the first ROI, but is not limited thereto.

Also, referring to FIG. 34, at least one color channel value 3540 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In this case, the color channel value 3540 may mean an average value of the color channel pixel values or a processed value, but a detailed description thereof will be omitted as described above.

In addition, the color channel value 3540 may be obtained in consideration of a biometric index to be obtained.

In addition, in order to obtain heart rate and blood pressure, the values of the at least two color channels may be selected to reduce noise due to movement and noise due to external light.

Also, referring to FIG. 34, at least one or more time series data 3550 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In addition, the time series data 3550 may be obtained in consideration of the biometric index to be obtained.

Also, referring to FIG. 34, at least one feature 3560 may be obtained in order to obtain a plurality of biomarkers according to an embodiment.

In addition, at least one or more of the features 3560 may be obtained in consideration of the biometric index to be obtained.

More specifically, at least one feature may be obtained for the first region of interest. For example, a first feature for the first region of interest may be obtained to obtain an oxygen saturation, a second feature may be obtained for the first region of interest to obtain a heart rate, and a blood pressure may be obtained. In order to do so, a third characteristic of the first region of interest may be obtained, but the present invention is not limited thereto.

Also, the third characteristic may be obtained based on a color channel value or time series data obtained for the first ROI. For example, when a color channel value obtained for the first ROI is a G-R value and a G-B value, the third characteristic may be obtained based on the G-R value and the G-B value, but is not limited thereto.

Also, for example, the third feature may be obtained based on third time series data obtained for the first ROI, but is not limited thereto.

In addition, the third characteristic may be a characteristic for obtaining blood pressure. For example, the third characteristic may be a gradient value, a maximum value, a minimum value, an average of a maximum value, an average of a minimum value, an average of the maximum value, and a difference value between the average of the minimum value, etc., obtained based on the third time series data. , Is not limited thereto.

In addition, the third feature may include a plurality of features. For example, the third characteristic includes at least two of the difference values between the slope value, the maximum value, the minimum value, the average of the local maximum, the average of the minimum value, the average of the local maximum, and the difference between the average of the minimum value obtained based on the third time series data. It may include, but is not limited thereto.

Also, referring to FIG. 34, at least one biometric index 3570 may be obtained in order to obtain a plurality of biometric indexes according to an embodiment.

In addition, when there are a plurality of biometric indexes 3570, each biometric index may be obtained at the same time, but is not limited thereto and may be obtained at different times. For example, oxygen saturation and heart rate may be obtained 6 seconds after measurement, and blood pressure may be obtained 8 seconds after measurement, but are not limited thereto.

In addition, when there are a plurality of biometric indexes 3570, each biometric index may be obtained based on each image frame group. For example, the oxygen saturation degree may be obtained based on the fourth image frame group, the heart rate may be obtained based on the fifth image frame group, and the blood pressure may be obtained based on the sixth image frame group.

In addition, each image frame group for obtaining the biometric index 3570 may be the same, but is not limited thereto, may be different from each other, and may at least partially overlap with each other. For example, the fourth, fifth, and sixth image frame groups may be identical to each other, may be different from each other, and may at least partially overlap each other.

In addition, at least one preliminary biometric index may be obtained to obtain the biometric index 3570. For example, at least four preliminary heart rates may be obtained to obtain a heart rate.

In addition, the number of preliminary biomarkers for obtaining the biometric index 3570 may be the same for each biometric index, but may be different. For example, the preliminary heart rate for acquiring the heart rate may be at least four, and the oxygen saturation level and the preliminary oxygen saturation level and the preliminary blood pressure for acquiring the blood pressure may be at least two, but are not limited thereto.

7.5 Method for obtaining a plurality of biomarkers according to an embodiment

Referring to FIG. 35, a plurality of image frames 3600 may be obtained to obtain a plurality of biomarkers according to an embodiment.

In this case, the image frame 3600 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, referring to FIG. 35, at least one or more biomarkers may be obtained based on at least one or more features.

For example, as shown in FIG. 35, a first biometric index 3660 may be obtained based on a first characteristic 3610, and a second biometric index 3670 may be obtained based on the second characteristic 3620 The third biometric index 3680 may be obtained based on the third feature 3630, and the fourth biometric index 3690 may be obtained based on the fourth feature 3640. It is not limited, and one biometric index may be obtained based on a plurality of features, or a plurality of biometric indexes may be obtained based on a single feature.

In this case, the first to fourth features are AC values, DC values, differences between the average of the maximum values and the average of the minimum values, average values, frequency component values, wavelength component values, and measurement time obtained based on at least one time series data The value for the number of times the cycle was repeated during, slope value, maximum value, minimum value, average of local maximum, average of minimum value, difference in acquisition time, time difference between maximum values, time difference between minimum values, time between inflection points It may include at least one of characteristics such as difference and skin temperature, but is not limited thereto.

In addition, the first to fourth biomarkers may include at least one of biomarkers such as heart rate, oxygen saturation, blood pressure, body temperature, and blood flow, but is not limited thereto.

However, detailed descriptions of the features and biomarkers have been described above, and thus, duplicate descriptions will be omitted.

In addition, the first to fourth features and/or biomarkers may be obtained based on at least some image frame groups among a plurality of image frames.

For example, the first feature and/or the first biometric index may be obtained based on a first image frame group, and the second feature and/or the second biometric index may be based on a second image frame group. And the third feature and/or the third biometric index may be obtained based on a third image frame group, and the fourth feature and/or the fourth biometric index may be a fourth image frame group It may be obtained based on, but is not limited thereto.

In this case, the first to fourth image frame groups may include at least two or more image frames.

For example, as shown in FIG. 35, the first image frame group may include N image frames, the second image frame group may include M image frames, and the third image The frame group may include K image frames, and the fourth image frame group may include L image frames, but is not limited thereto.

In addition, the number of image frames included in the first to fourth image frame groups may be the same.

For example, the first image frame group may include 180 image frames, the second image frame group may include 180 image frames, and the third image frame group may include 180 image frames. The fourth image frame group may include 180 image frames, but is not limited thereto.

In this case, the first to fourth biomarkers may be acquired or output at the same time point, but are not limited thereto and may be acquired or output at different time points.

In addition, in this case, the first to fourth biomarkers may obtain biomarkers from the same state of the subject, and each biomarker may have a correlation with each other.

In addition, the number of image frames included in the first to fourth image frame groups may be different from each other.

For example, the first image frame group may include 180 image frames, the second image frame group may include 90 image frames, and the third image frame group may include 240 image frames. May include, and the fourth image frame group may include 360 image frames, but is not limited thereto.

In this case, the first to fourth biomarkers may be acquired or output at different times, but are not limited thereto and may be acquired or output at the same time point.

In addition, the number of image frames included in the first to fourth image frame groups may be different according to a biometric index to be acquired.

For example, when the first biometric index is heart rate, the second biometric index is oxygen saturation, the third biometric index is blood pressure, and the fourth biometric index is body temperature, the first image frame group may include 180 image frames. The second image frame group may include 120 image frames, the third image frame group may include 60 image frames, and the fourth image frame may include 60 image frames. However, the present invention is not limited thereto, and the number of image frames included in the image frame group may be set in consideration of a biometric index to be acquired.

Referring to FIG. 36, a plurality of image frames 3700 may be obtained in order to obtain a plurality of correlated biomarkers according to an embodiment.

In this case, the image frame 3700 may be an image frame obtained from a visible light image or an infrared image, but is not limited thereto.

Also, referring to FIG. 36, at least one biometric index may be obtained based on at least one or more image frame groups.

More specifically, as shown in FIG. 36, the first biometric index 3760 may be obtained based on the first image frame group 3710, and the second biometric index 3770 is the second image frame group ( 3720), the third biometric index 3780 may be obtained based on the third image frame group 3730, and the fourth biometric index 3790 may be the fourth image frame group 3740 Can be obtained based on.

In this case, since the above-described contents may be applied to each image frame group and each biometric index, a redundant description will be omitted.

Referring to FIG. 36, in order to obtain the first bio-index 3710, the second bio-index 3720, the third bio-index 3730, and the fourth bio-index 3740 so as to be correlated with each other, the The first image frame group 3710, the second image frame group 3720, the third image frame group 3730, and the fourth image frame group 3740 may at least partially overlap each other.

For example, as shown in FIG. 36, the first image frame group 3710 includes a first image frame to a thirteenth image frame, and the second image frame group 3720 is a sixth image frame to A 19th image frame is included, the third image frame group 3730 includes a fourth image frame to a 17th image frame, and the fourth image frame group 3740 is a ninth image frame to a 23rd image frame It may include, but is not limited thereto.

Accordingly, the first to fourth

image frame groups

3710, 3720, 3730, and 3740 may include a ninth to a thirteenth image frame in common.

In addition, as described above, as the first to fourth image frame groups at least partially overlap each other, the first to fourth biometric indexes may be obtained to be correlated with each other.

For example, since the first to fourth image frame groups commonly include the ninth to thirteenth image frames, the first to fourth biometric indexes may be obtained including the same state of the subject. Accordingly, the first to fourth biomarkers may be obtained so as to be correlated with each other.

For a more specific example, when the state of the subject is referred to as the first state at the time when the ninth image frame is acquired, the first to fourth biometric indexes reflect the first state of the subject in common It may be obtained, and accordingly, the first to fourth biomarkers may be obtained so as to be correlated with each other.

In addition, when the first to fourth biometric indexes 3760, 3770, 3780, and 3790 are obtained to be correlated, a more accurate biometric index of the subject may be determined.

In addition, at least one or more biometric information may be obtained based on the first to fourth biometric indexes 3760, 3770, 3780, and 3790.

In addition, when the biometric information is obtained based on correlated biomarkers, the accuracy of biometric information can be improved. For example, when using heart rate and blood pressure that may be included in the first to fourth biometric indexes to obtain hypertension information that may be included in the biometric information, when the heart rate and blood pressure are related to each other, the hypertension information is It can be more accurate.

Referring to FIG. 37, at least one or more biomarkers may be obtained based on at least one or more preliminary biomarkers.

For example, as shown in FIG. 37, the first bio-index may be obtained based on N preliminary first bio-indexes, and the second bio-index may be obtained based on M preliminary second bio-indexes. In addition, the third bio-index may be obtained based on K preliminary third bio-indexes, and the fourth bio-index may be obtained based on the L Evi fourth bio-index.

However, since the above-described information may be applied to the method of obtaining the biometric index using the preliminary biometric index, the overlapping description will be omitted.

The number of preliminary first to fourth biomarkers for obtaining each of the first to fourth biomarkers may be the same.

For example, the preliminary first bio-index for obtaining the first bio-index may be four, the preliminary second bio-index for obtaining the second bio-index may be four, and the second 3 The preliminary third bio-index for obtaining the bio-index may be four, and the preliminary fourth bio-index for obtaining the fourth bio-index may be four, but the present invention is not limited thereto.

In addition, the number of preliminary first to fourth biomarkers for obtaining each of the first to fourth biomarkers may be different from each other.

For example, the preliminary first bio-index for obtaining the first bio-index may be four, the preliminary second bio-index for obtaining the second bio-index may be two, and the second 3 The preliminary third bio-index for obtaining the bio-index may be two, and the preliminary fourth bio-index for obtaining the fourth bio-index may be one, but is not limited thereto.

In addition, the number of preliminary first to fourth biomarkers for obtaining each of the first to fourth biomarkers may differ according to the biomarkers to be obtained.

For example, when the first biometric index is heart rate, the second biometric index is oxygen saturation, the third biometric index is blood pressure, and the fourth biometric index is body temperature, the preliminary first biometric index for obtaining the heart rate is 4 May be, and the preliminary second bio-index for obtaining the oxygen saturation may be two, the preliminary third bio-index for obtaining the blood pressure may be two, and the preliminary second bio-index for obtaining the blood pressure may be two, and the The preliminary fourth biometric index may be one, but is not limited thereto, and the number of preliminary biometric indexes to be obtained may be set in consideration of the biometric index to be acquired.

8. Drowsiness detection device and method

8.1 Drowsiness detection device

The drowsiness detection device provided in the present specification may mean a device that detects a state related to drowsiness of a subject. Specifically, the drowsiness detection device may detect whether the subject is drowsy and the intensity of drowsiness.

A person may feel drowsy during daily life. And in some cases, there may be situations in which drowsiness poses a safety hazard. For example, when a driver feels drowsy while driving, he cannot concentrate on driving and thus a traffic accident may be caused. In this case, the drowsiness detection device detects the driver's drowsiness state and notifies the driver or the manager of the drowsiness state, thereby preventing a situation that threatens the driver's safety, such as a traffic accident. As described above, it goes without saying that the drowsiness detection device can be used not only in situations where drowsiness poses a risk to safety, but also in places and fields where drowsiness serves as important information, such as reading rooms and infant monitoring.

The drowsiness detection device may detect a state related to drowsiness by using a contact-type device that involves physical contact with a subject, such as an attachable electrode sensor or a wearable device. In addition, the drowsiness detection device may detect a state related to drowsiness of the subject in a non-contact method using a camera. When the non-contact method is used, as the subject's drowsiness can be detected without being constrained by the contact type device, not only the user's convenience will increase, but also the drowsiness detection device may be used in various places and fields. .

The non-contact drowsiness detection device may detect a state related to drowsiness by detecting an abnormal state of the subject expressed externally, such as measuring the number of times the subject blinks or tracking the position of the pupil. However, when the subject consciously controls the blinking of the eye or the position of the pupil, it may be difficult to accurately detect the state related to drowsiness. In order to compensate for the accuracy, if the information displayed in the body of the test subject, which appears when the subject feels drowsy, is used, the state related to the sleepiness can be more accurately detected. In addition, if the subject does not feel drowsy but uses biometric information that appears in a situation where he will feel drowsy soon, a state related to drowsiness can be detected in advance in the initial stage.

Due to this advantage, the drowsiness detection device 6000 provided in the specification can detect whether the person to be measured is in a state of drowsiness and to what extent it corresponds to the level of drowsiness by using the biometric index of the subject obtained in a non-contact manner. I can.

Referring to FIG. 38 according to an embodiment, the drowsiness detecting device 6000 includes an acquired heart rate information acquisition unit 6100 for acquiring a heart rate from a subject, and a subject's drowsiness based on the acquired heart rate. It may include a drowsiness detection unit 6200 to detect.

Of course, the drowsiness detection device 6000 of the present invention may further include other components other than the heart rate information acquisition unit 6100 and the drowsiness detection unit 6200 mentioned above. For example, the drowsiness detection device 6000 may further include a notification unit 6300 in addition to the heart rate information acquisition unit 6100 and the drowsiness detection unit 6200. Here, the notification unit 6300 may provide a notification for waking the sleepiness to the subject or a notification to an object other than the subject based on a result detected by the sleepiness detection unit 6200. Here, the entity other than the subject may refer to a manager who manages the place to be measured or a third party around the subject.

In an embodiment, the drowsiness detection device 6200 may detect drowsiness of the subject at various points in time. That is, some or all components included in the drowsiness detection apparatus 6000 may operate at various times, and each step or all steps of various drowsiness detection methods to be described later may also be implemented at various times.

For example, the drowsiness detecting device 6000 may detect the drowsiness of the subject according to a predetermined period (eg, every minute, every hour). As another example, the drowsiness detecting device 6000 may detect the drowsiness of the subject without following a predetermined period of time. As an example, the drowsiness detection apparatus 6000 may detect the drowsiness of the subject every time information about the heartbeat of the subject is acquired, that is, in real time. In addition, the drowsiness detection apparatus may continuously detect the drowsiness of the subject, or when a signal for triggering the drowsiness detection is input or when a request to detect drowsiness is received, the drowsiness of the subject may be detected.

In another embodiment, the drowsiness detecting device 6000 may start detecting drowsiness when acquiring the initial heart rate. For example, the heart rate information acquisition unit 6100 may initially acquire a heart rate several seconds or minutes after starting measurement for obtaining a heart rate for a subject. In this case, the drowsiness detecting device 6000 may start detecting drowsiness for the subject from the time the initial heart rate is acquired.

Hereinafter, components included in the drowsiness detecting device 6000 will be described in detail with reference to FIG. 38 according to an embodiment.

8.1.1 Heart rate information acquisition unit

In an embodiment, the heart rate information acquisition unit 6100 may acquire information on a heart rate for a subject, for example, at least one of a heart rate of a subject or a ratio of sympathetic activity to parasympathetic activity.

Here, the heart rate may mean the number of times the heart beats during the reference time. The heart rate generally means the heart rate during the reference time of 1 minute, but the reference time may be arbitrarily set. For example, it can be set to 30 seconds or another length. In addition, the ratio of the sympathetic nerve activity to the parasympathetic nerve activity is due to the heartbeat signal, where the heartbeat signal may mean a signal that can fluctuate according to the heartbeat, and a signal that can be estimated to fluctuate according to the heartbeat. It can mean. In general, sympathetic activity and parasympathetic activity are highly related to the drowsy state of the test subject. Accordingly, the drowsiness detection device can detect the drowsiness of the subject by using the ratio of the sympathetic activity to the parasympathetic activity caused by the heartbeat signal.

In addition, the activity of the sympathetic nerve may be characterized in a low frequency band of the heart rate signal, and the activity of the parasympathetic nerve may be characterized in a high frequency band of the heart rate signal. Therefore, in the present specification, the ratio of the sympathetic activity to the parasympathetic activity of the heart rate signal may be expressed as LF/HF. LF may mean a characteristic value in a low frequency band (Low Frequency, LF) of the heartbeat signal, and HF may mean a characteristic value in a high frequency band (High Frequency, HF) of the heartbeat signal.

In an embodiment, a low frequency band and a high frequency band may be classified based on a reference frequency. For example, the low frequency band and the high frequency band are divided based on a reference frequency of 0.15 Hz, the low frequency band includes a signal having a frequency in the range of 0.04 to 0.15 Hz, and the high frequency band has a frequency in the range of 0.15 to 0.4 Hz. May contain signals. Of course, the reference frequency may be set to a frequency of a value other than 0.15 Hz.

In an embodiment, the heart rate information acquisition unit 6100 may convert a heart rate signal into a signal in the frequency domain through various methods such as Fourier transform, and through the conversion, a low frequency band (0.04-0.15 Hz) and a high frequency The LF/HF value can be obtained by extracting the signal in the band (0.15-0.4 Hz).

In an embodiment, the heart rate information acquisition unit 6100 may acquire information on the heart rate in an invasive or non-invasive manner.

In addition, according to another embodiment, the heart rate information acquisition unit 6100 may acquire information on the heart rate through a contact method or a non-contact method.

In more detail, the heart rate information acquisition unit 6100 may acquire information on the heart rate in a non-contact method using a device that does not physically contact the subject. For example, the heart rate information acquisition unit 6100 may acquire an image of a subject using a camera, and obtain information on a heart rate of the subject by analyzing the acquired image. A method and apparatus for obtaining information about heart rate in a non-contact manner using a camera have been described above, and detailed descriptions thereof will be omitted.

In addition, it goes without saying that the heart rate information acquisition unit 6100 may acquire information on the heart rate through a contact method using a device that physically contacts the subject. For example, the heart rate information acquisition unit 6100 may acquire information on the heart rate by using an attachable sensor for measuring heart rate or a wearable device including various attachable sensors and optical sensors. For convenience of explanation, the following describes the configuration of a sleepiness detection device and a sleepiness detection method, focusing on an example of acquiring heart rate information in a non-contact manner.

In an embodiment, the heart rate information acquisition unit 6100 may acquire information on the heart rate by directly measuring it. Specifically, the heart rate information acquisition unit may include a device for measuring heart rate, and may acquire information on the heart rate measured by the device for heart rate measurement. For example, the heart rate information acquisition unit 6100 may include a camera and may directly measure information on heart rate by analyzing an image acquired from the camera.

In addition, the heart rate information acquisition unit 6100 may obtain information on the heart rate by receiving information on the heart rate measured by an external device. For example, information on heart rate is measured by a device including an electrode sensor for measuring heart rate, and the heart rate information acquisition unit 6100 may receive information on the measured heart rate. In addition, the heart rate information acquisition unit 6100 may receive information on the measured heart rate through the heart rate information acquisition unit 6100 and another device (eg, a server) that mediates the external device.

In one embodiment, when the drowsiness detection device 6000 is included in the biometric index acquisition device 10, the heart rate information acquisition unit 6100 obtains the heart rate by reading the heart rate stored in the memory of the biometric index acquisition device 10 can do. In another embodiment, when the drowsiness detection device 6000 is not included in the biometric index acquisition device 10, the heart rate information acquisition unit 6100 calculates the heart rate measured by the biometric index acquisition device 10 as the sleepiness detection device ( 6000) can be received through a communication unit (not shown) included. Of course, the sleepiness detection device 6000 may acquire a heart rate from a device other than the biometric index acquisition device 10.

8.1.2 Drowsiness detection unit

In an embodiment, the drowsiness detecting unit 6200 may detect whether the subject is drowsy or the intensity of drowsiness felt by the subject based on the information on the heart rate. Specifically, the drowsiness detection unit 6200 may detect the drowsiness state and the normal state of the subject based on the information on the heart rate. Here, the drowsiness state can be divided into a plurality of stages.

In the present specification, the drowsy state may mean a state in which the subject is temporarily sleeping or a state that is likely to sleep within a predetermined time before sleeping, and the normal state is not a drowsy state, but the subject is within a predetermined time. This could mean a condition where you are less likely to fall asleep.

In an embodiment, the drowsiness state may be divided into a plurality of levels according to the intensity of drowsiness. The drowsiness state can be classified from the first stage to the Nth stage depending on the case. In the present specification, for convenience of explanation, the lowest level of drowsiness is indicated by level 1, and the level of the highest level of drowsiness is indicated by level N. For example, drowsiness can be divided into first to third stages. In this case, the first step may mean a step in which the intensity of drowsiness is the lowest, and the third step may mean a step in which the intensity of drowsiness is the highest.

In an embodiment, the first stage drowsiness state may mean a state in which the subject enters the drowsiness state physically, but the subject does not become aware of drowsiness. For example, in the first stage drowsy state, information on the heart rate of the test subject objectively indicates that the test subject is in a drowsy state, but the subject may not feel drowsy by itself. In the present specification, the drowsiness state may be expressed as an involuntary drowsiness state or an unconscious drowsiness state.

In an embodiment, the second stage drowsiness state and the third stage drowsiness state may mean a state in which the subject physically enters the drowsiness state and the subject is aware of drowsiness. In the present specification, the corresponding states may be expressed as a subjective drowsiness state or a conscious drowsiness state.

In general, the subject enters the insensitive drowsiness state before entering the subjective drowsiness state. When the subject is notified about drowsiness after entering the subjective drowsiness state, a notification is received when the subject is already drowsy, and there is a high possibility of a dangerous situation for the subject. In addition, even if a notification about the drowsiness state is received before a dangerous situation occurs, a lot of time may be required for the subject to escape from the drowsiness state.

In order to prevent such a dangerous situation, the drowsiness detection device 6000 may detect the insensitive drowsiness state before the subject enters the subjective drowsiness state, and may provide a notification based on the detection result to the outside. The drowsiness detection device 6000 may make the subject to be conscious of the drowsiness state from the involuntary drowsiness state in which he does not feel drowsiness through the notification.

Therefore, the detection and notification of involuntary drowsiness can increase the likelihood of getting the subject out of drowsiness before a dangerous situation occurs, and can more reliably protect the subject from dangerous situations caused by drowsiness.

In an embodiment, the drowsiness detection unit 6200 may measure the drowsiness state of the subject by using the information on the heart rate. In the present specification, the drowsiness state is a state in which the subject is temporarily sleeping or a state in which it is highly likely to sleep within a predetermined time immediately before sleeping, and is a state in which the parasympathetic nervous system is relatively activated compared to the normal state, and the parasympathetic nervous system When is activated, the heart rate decreases. That is, the change in heart rate may serve as a biomarker capable of estimating the drowsy state of the subject. For example, when the subject enters a drowsy state, the subject's heart rate decreases below a certain level, and the reduced heart rate may continue for a certain period of time. In some cases, the drowsiness detection unit 6200 may estimate the drowsiness state of the subject through the duration of the reduced heart rate.

In an embodiment, the drowsiness detection unit 6200 may measure a normal state of a subject by using information about a heart rate. For example, when the subject enters a normal state, the heart rate of the subject increases to a certain level or higher due to activation of the sympathetic nerve, and the increased heart rate may also continue for a certain period of time. In this case, the drowsiness detecting unit 6200 may estimate the normal state of the subject through the duration of the increased heart rate.

In addition, in another embodiment, the drowsiness detection unit 6200 may measure the drowsiness state and the normal state of the subject using LF/HF. As the subject enters the drowsy state, the parasympathetic nervous system of the subject is activated, and the LF value for the HF value of the heart rate signal may decrease. In addition, when the subject enters the normal state, the sympathetic nervous system of the subject is activated, and the LF value for the HF value may increase. Accordingly, the drowsiness detection unit may estimate the drowsiness state and the normal state of the subject through the LF/HF.

In another embodiment, the drowsiness detection unit 6200 may estimate the drowsiness state or the normal state of the subject by simultaneously considering the change in the heart rate and the LF/HF.

The drowsiness detection method will be described in detail below.

8.1.3 Notification

In an embodiment, the notification unit 6300 may provide information on a drowsy state sensed externally. Specifically, the notification unit 6300 may provide various information related to whether the subject is in a drowsy state, a drowsiness stage, and in addition to the sensed drowsiness state. For example, the notification unit 6300 may provide externally a time of maintaining the drowsy state, a time of entering each phase of the drowsy state, a time of recovery of the drowsy state, and the like.

In an embodiment, the notification unit 6300 may provide information on a drowsy state to a subject. For example, the notification unit 6300 may give a notification of an intensity corresponding to the stage of the drowsy state to the subject. The notification may induce recovery of the drowsy state by applying a certain stimulus to the drowsy state subject.

In another embodiment, information about a drowsy state may be provided to a subject other than the subject. For example, the notification unit 6300 may provide information on a drowsy state to an administrator or a server. Here, the manager may refer to a manager of a subject or an entity that manages a drowsy state of the subject. Provision of such information can help the manager effectively manage the safety of the subject based on the information on the drowsy state.

In an embodiment, the notification unit 6300 may provide information on the drowsiness state of all detected stages to the outside. For example, the notification unit 6300 may provide information on the drowsiness state to the subject in the first to third stage drowsiness states.

In addition, in another embodiment, the notification unit 6300 may provide only information on at least some of the sensed drowsiness states of all stages to the outside.

For example, the notification unit 6300 may provide information on the time and stage when the third stage drowsy state is detected to the administrator only when the third stage drowsy state is detected. When notifications for all stages of drowsiness interfere with the subject, the notification unit 6300 provides only information on the high-risk drowsiness stage (for example, information on the third stage drowsiness status), and By not providing information on this low drowsiness stage (for example, information on the first stage drowsiness state), it is possible to increase the efficiency of the work of the subject.

In addition, in an embodiment, when the subject is detected as a normal state, the notification unit 6300 may provide information on the detected normal state. Specifically, the notification unit 6300 may provide a variety of information related to the detected normal state, including whether the subject is in a normal state. For example, it is possible to provide the time of maintaining the normal state, the time of entering the normal state, etc.

In an embodiment, the notification unit 6300 may provide information on a normal state to a subject. For example, the notification unit 6300 may output a message for notifying that the drowsy test subject has recovered to the normal state.

In another embodiment, the notification unit 6300 may provide information on a normal state to a subject other than the subject. For example, the notification unit 6300 may provide information on a normal state to an administrator or a server. Providing such information can help the manager to recognize that the subject is in a safe situation.

In one embodiment, the notification unit 6300 generates a notification signal indicating information related to the drowsiness state, and a notification signal to the output unit (not shown) so that information related to the normal state and the drowsiness state is output to the outside of the drowsiness detection device. Can provide.

Here, the output unit is a device capable of notifying information to the outside. For example, the output unit is a display unit that visually displays information to the outside, an audio unit capable of audibly notifying information, and an audio unit capable of tactilely notifying information. It may include at least one of the tactile stimulation units. The output unit may be included in the drowsiness detection device 6000 or may be included in an external device of the drowsiness detection device 6000.

When the output unit is included in another device outside the drowsiness detection device 6000, the drowsiness detection device 6000 may further include a communication unit (not shown), and the notification unit 6300 transmits a notification signal to the output unit through the communication unit. Can provide. The output unit may receive a notification signal provided from the notification unit 6300 and output information on drowsiness to the outside according to the notification signal.

For example, the notification unit 6300 may give an audible notification through the audio unit. Such an audible notification is effective because it can give an immediate stimulus to the notification recipient and can change the intensity of the stimulus according to the intensity of the sound.

For another example, the notification unit 6300 may provide a notification by displaying a message on the display through the display unit. Such a visual notification can minimize a situation in which objects other than the notification recipient are disturbed by the notification.

For another example, the notification unit 6300 may provide notification through the tactile stimulation unit, for example, vibration, electrical stimulation, a thermal element or a cooling element. Such a tactile notification is effective because a situation in which an entity other than the notification recipient is not disturbed by the notification does not occur, can give an immediate stimulus, and can vary the intensity of the stimulus according to the intensity of the stimulus.

In addition, in an embodiment, the notification unit 6300 may provide notifications of different types and/or intensity to the subject according to the level of the sensed sleepiness. Accordingly, the notification unit 6300 may provide a notification with a type suitable for recognizing that the subject is in a drowsy state and/or an intensity necessary for the subject to get out of the drowsy state.

For example, when a first stage drowsiness state, which is a low intensity drowsiness state, is detected, the notification unit 6300 may recognize the drowsiness state by giving a notification using a visual message to the subject. In addition, when the third stage drowsiness, which is a high-intensity drowsiness state, is detected, the notification unit 6300 may provide a notification using electrical stimulation to the subject, thereby allowing the subject to escape from the drowsiness state.

For another example, when the first stage drowsiness state, which is a low intensity drowsiness state, is detected, the notification unit 6300 provides a low intensity alarm to the subject, and when the third stage drowsiness is detected, the notification unit 6300 ) Can provide a high-intensity alarm to the subject.

Of course, according to an embodiment, the notification unit 6300 may provide notifications of the same type and/or intensity to the subject regardless of the level of the sensed sleepiness.

In addition, the notification unit 6300 may notify at least one of a subject and an object other than the subject according to the stage of the sensed drowsy state.

In an embodiment, the notification unit 6300 may notify the subject when drowsiness in the first and second steps is detected. In this case, the notification unit 6300 may provide at least one of an audible notification, a visual notification, and a tactile notification to the subject.

In addition, the notification unit 6300 may notify not only a subject but also an object other than the subject when the third stage drowsy state is detected. Specifically, the notification unit 6300 may notify an object other than the subject through a device or a server that mediates the notification unit 6300 and an object other than the subject. For example, when an entity other than the subject is a manager managing the subject, such notification may be effective in a situation in which the subject turns off the power of the notification through the output unit or ignores the notification. Hereinafter, the drowsiness detection method performed by the drowsiness detection device 6000 will be described in detail.

8.2 Drowsiness detection method based on heart rate

Hereinafter, a method of detecting drowsiness based on a heart rate according to an exemplary embodiment will be described with reference to FIGS. 39 to 43.

In one embodiment, the method of detecting drowsiness based on a heart rate includes the step of obtaining a heart rate (S6110), comparing the acquired heart rate with a reference heart rate (S6120), and measuring the duration of the changed heart rate according to the comparison result. It may include a duration measurement step (S6130) and a step-by-step drowsiness detection step (S6140) of detecting drowsiness step by step based on the measured duration of the heart rate change.

In the heart rate acquisition step S6110, the drowsiness detection device 6000 may acquire the heart rate of the subject by using the heart rate information acquisition unit 6100.

In an embodiment, the drowsiness detecting device 6000 may acquire a heart rate of a subject according to a predetermined period of time. Here, the predetermined time period may include various time periods such as 1 second, 1 minute, and 5 minutes. In addition, of course, the predetermined time period may be fixed or may be variable.

In another embodiment, the drowsiness detecting device 6000 may acquire a heart rate of a subject without following a predetermined time period. For example, the drowsiness detecting device 6000 may acquire the heart rate of the subject whenever the heart rate of the subject is measured by the biometric index obtaining apparatus 10 or another device, that is, in real time.

Of course, if the heart rate of the subject is measured by the biometric index acquisition device 10 or another device according to a predetermined time period, the drowsiness detection device 6000 may perform a predetermined time period measured by the biometric index acquisition device 10 or another device. As a basis, the heart rate of the subject can be obtained.

As another example, the drowsiness detection device 6000 may request to provide a heart rate to the biometric index acquisition device 10 or another device according to an external input or request, and each time the request is made, the heart rate of the subject Can be obtained

In an embodiment, the drowsiness detecting device 6000 may obtain an average heart rate of a subject for a predetermined time period. The predetermined time period may be arbitrarily determined. Here, the drowsiness detecting device 6000 may calculate an average heart rate for a predetermined time period using the heart rate of the subject received from an external device. In addition, the drowsiness detecting device 6000 may receive an average heart rate for a predetermined time period of the subject from an external device.

Hereinafter, with reference to FIG. 40, the effect of obtaining the average heart rate as the heart rate of the subject will be described.

Referring to FIG. 40 according to an embodiment, the drowsiness detecting apparatus 6000 may acquire an average heart rate 6003 and thus a heart rate from which noise is removed.

Here, the noise indicates an error caused by various causes such as movement of the subject or external light, and when noise is included in the heart rate, the corresponding heart rate may be difficult to accurately reflect the state of the subject.

For example, when the heart rate information acquisition unit 6100 acquires a heart rate every second from 1 second to 10 seconds, when noise occurs when the number of seconds is 5 seconds, the drowsiness detection device 6000 may not accurately detect drowsiness. However, when the heart rate information acquisition unit 6100 acquires the average heart rate 6003, the deviation between noise and other values may be corrected in the process of calculating the average heart rate 6003, so the drowsiness detection device 6000 is more You can accurately detect drowsiness.

The method of obtaining the heart rate of the subject by the heart rate information acquisition unit 6100 is described in Table of Contents 8.1.1, so a detailed description thereof will be omitted.

In the following, referring to FIG. 41, the step of comparing the heart rate of the subject with the reference heart rate (S6120) and the step of measuring the duration of the acquired heart rate of the subject in a reduced/increased state (S6130). Let's explain about.

Referring to FIG. 41 according to an embodiment, in the step (S6120) of comparing the heart rate of the subject and the reference heart rate, the drowsiness detecting device 6000 includes the heart rate of the subject acquired by the heart rate information acquisition unit 6100 ( 6001) and the reference heart rate 6002 can be compared.

Here, the reference heart rate 6002 may mean a heart rate set as a criterion for distinguishing between a drowsy state and a normal state. As described above, according to the activation of the parasympathetic nervous system, the heart rate in the drowsy state decreases compared to that in the normal state. Accordingly, in order to distinguish between the drowsy state and the normal state, a heart rate equal to or similar to the reduced heart rate may be set as the reference heart rate 6002. In this case, when the subject's heart rate is less than or equal to the reference heart rate (6002), the subject may be determined to be in a drowsy state, and when the subject’s heart rate (6001) is greater than or equal to the reference heart rate (6002), the subject is determined to be in a normal state. It will be possible.

Hereinafter, various methods for setting the reference heart rate 6002 will be described.

In an embodiment, the drowsiness detection device 6000 may set the reference heart rate 6002 by using an average of heart rates acquired for a predetermined time from the time when the heart rate is first acquired. The predetermined time may be set to various values such as 1 minute and 5 minutes. In general, there is a high possibility that the subject is not in a drowsy state but in a normal state for a certain period of time from the time when the heart rate of the subject is first acquired. Accordingly, the average heart rate of the subject acquired during the predetermined time is obtained, and the average heart rate is multiplied by a predetermined value a (eg, a represents a real number of 1 or less. As a more specific example, a is 0.9). The value may be obtained as the reference heart rate 6002. The predetermined value a is not limited to the above example.

In another embodiment, the sleepiness detection device 6000 may set the reference heart rate 6002 using the resting heart rate and/or the active heart rate. Specifically, the resting heart rate may mean a heart rate in a state in which the subject does not move (or when the amount of movement is small). In this specification, the heart rate when the subject is awake but not moving (or when the amount of movement is small) is defined as the first resting heart rate, and the heart rate when the subject is sleeping is defined as the second resting heart rate. Do it.

In addition, the active heart rate may mean a heart rate when the subject is actively moving.

In an embodiment, the drowsiness detection device 6000 may acquire the measured heart rate as the first resting heart rate in a situation where it is recognized that the subject is awake but does not move (or when the amount of movement is small). For example, the drowsiness detection device 6000 is measured for a certain period of time from the first time the heart rate is acquired when the subject is in an environment in which the subject cannot actively move (for example, when the subject is sitting in the driver's seat). Assuming that the self is awake but not moving, the heart rate measured during the predetermined time may be set as the first resting heart rate.

In addition, the drowsiness detection device 6000 acquires the heart rate measured in the situation where the subject is recognized as sleeping, as the second resting heart rate, and the measured heart rate in the situation in which the subject is recognized as being in an active movement state. It can be obtained with the active heart rate.

In addition, the sleepiness detection device 6000 may acquire a first resting heart rate, a second resting heart rate, and an active heart rate using a camera or a wearable device.

For example, the drowsiness detecting device 6000 may obtain information on whether the subject moves and whether the subject's pupil is tracked by using the camera. For example, the drowsiness detection device 6000 may acquire a heart rate measured when the subject moves as an active heart rate. In addition, the drowsiness detection device 6000 acquires the measured heart rate as the first resting heart rate when the subject's pupil is continuously tracked even if the subject does not move using the camera, and the subject does not move and When the pupil is not continuously tracked, the measured heart rate may be obtained as the second resting heart rate.

As another example, the drowsiness detecting device 6000 may obtain information on whether a subject moves by using a wearable device and whether a body part (eg, wrist) on which the wearable device is worn moves. For example, the drowsiness detection device 6000 may acquire a heart rate measured when a movement of a subject is detected using a wearable device as an active heart rate. In addition, the drowsiness detection device 6000 does not detect the movement of the subject, but when the movement of the body part on which the wearable device is worn is detected, the measured heart rate is acquired as the first resting heart rate, and the movement of the subject is not detected. , When the movement of the body part is not detected for a predetermined period of time, the measured heart rate may be obtained as the second resting heart rate.

In addition, the drowsiness detection device 6000 may pre-set the first resting heart rate, the second resting heart rate, and the active heart rate before starting to detect drowsiness, and the first resting heart rate from the outside before or after the start of sleep detection, The second resting heart rate and the active heart rate may be received.

According to an embodiment, the reference heart rate 6002 may be set based on the first resting heart rate. Specifically, the reference heart rate 6002 may be set at a constant rate of the first resting heart rate. For example, the reference heart rate 6002 may be calculated by multiplying the first resting heart rate by a predetermined value a (eg, a represents a real number of 1 or less. As a more specific example, a is 0.9). Of course, the predetermined value a is not limited to the above example.

According to another embodiment, the reference heart rate 6002 may be set based on the second resting heart rate. Specifically, the reference heart rate 6002 is set equal to the second resting heart rate, or a predetermined value b for the second resting heart rate (eg, b represents a real number greater than 1. As a more specific example, b is It can be calculated by multiplying by 1.1). Of course, the predetermined value b is not limited to the above example.

In addition, according to another embodiment, the reference heart rate 6002 may be set based on the active heart rate. Specifically, the reference heart rate 6002 may be calculated by multiplying the active heart rate by a predetermined value c (eg, c represents a real number of 1 or less. As a more specific example, c is 0.8). In addition, in consideration of the fact that the active heart rate is generally higher than the first resting heart rate, the predetermined value c may be set lower than the predetermined value a. Of course, the predetermined value c is not limited to the above example.

The method of setting the reference heart rate 6002 is not limited to a method of calculating a constant ratio of the first and second resting heart rates and the active heart rate, and various calculations such as sum and difference may be applied.

Specifically, in an embodiment, the sleepiness detection device 6000 may set the reference heart rate 6002 by adding or subtracting a predetermined value to the first or second resting heart rate or the active heart rate. For example, when the first resting heart rate is 75, the drowsiness detecting device may set the reference heart rate 6002 to (first resting heart rate-10) = 65 (times/minute) based on this. For example, when the second resting heart rate is 65, the drowsiness detecting device 6000 may set the reference heart rate 6002 to (second resting heart rate + 10) = 65 (times/minute) based on this.

In addition, in another embodiment, the drowsiness detecting device 6000 may change from the previously set reference heart rate 6002 to a new reference heart rate 6002. That is, the drowsiness detecting device 6000 may set a new reference heart rate 6002 by updating the previously set reference heart rate 6002.

For example, as described above, the sleepiness detection apparatus 6000 may set the reference heart rate 6002 based on the heart rate acquired for a predetermined time from the time when the heart rate of the subject is first acquired. The sleepiness detection device 6000 may detect a normal state based on the heart rate of the subject acquired after the predetermined time, and change the average heart rate in the time interval detected as the normal state to a new reference heart rate 6002. have. In addition, the drowsiness detection device 6000 may change to a new reference heart rate 6002 based on the first and second resting period and active period heart rate acquired for the same subject after the predetermined time. Accordingly, the drowsiness detection device 6000 may set the reference heart rate 6002 reflecting the most recent state of the subject, and thus the drowsiness detection apparatus 6000 is based on the more accurate reference heart rate 6002. You can detect drowsiness.

For another example, the drowsiness detecting device 6000 may preset the reference heart rate 6002 on the basis of the first and second resting period and active period heart rate as described above. At this time, after the reference heart rate 6002 is set based on the first, second resting or active heart rate, the drowsiness detecting device 6000 may newly acquire the first, second resting or active heart rate for a predetermined time. . The sleepiness detection device 6000 may set a new reference heart rate 6002 according to the method of setting the reference heart rate 6002 described above using the newly acquired first, second resting or active heart rate.

Referring to FIG. 41 according to an embodiment, in the step (S6120) of comparing the heart rate 6001 of the subject and the reference heart rate 6002, the sleepiness detecting device 6000 is the heart rate 6001 of the subject and the reference Heart rate 6002 can be compared.

In one embodiment, the drowsiness detection device 6000 compares the heart rate 6001 of the subject and the reference heart rate 6002, and the time point at which the heart rate 6001 of the subject becomes less than or equal to the reference heart rate 6002 (t1) ) Can be detected. The sensed point in time t1 may be a starting point for measuring the drowsiness state and the normal state in each step.

That is, in the above embodiment, the detection of the time point t1 at which the heart rate 6001 of the subject becomes less than/greater than the reference heart rate 6002 may serve as a trigger for step-by-step sleepiness and steady state measurement. I can.

In the duration measurement step (S6130), the drowsiness detection device 6000 has the heart rate 6001 of the subject being equal to or less than the reference heart rate 6002 based on the time point when the heart rate 6001 of the subject becomes less than the reference heart rate 6002. You can measure how long the state lasts. In addition, the drowsiness detection device 6000 measures the duration of the state in which the heart rate 6001 of the subject is more than the reference heart rate 6002 based on the time when the heart rate 6001 of the subject becomes more than the reference heart rate 6002 can do.

In addition to drowsiness, heart rate can fluctuate due to a number of physical factors. Among various physical factors, since the decrease/increase in heart rate due to drowsiness continues to decrease/increase, the continuous decrease or increase in heart rate for several seconds to several minutes can be a good biomarker indicating drowsiness.

For example, when a person's tension is relieved, the heart rate temporarily decreases, but the reduced heart rate can be easily recovered. If the drowsiness detection device 6000 detects the drowsiness state based on the duration of the reduced heart rate state, the drowsiness detection device 6000 avoids detecting a situation in which the heart rate temporarily decreases due to relaxation of tension as a drowsiness state. I can. Thereby, the drowsiness detection device 6000 may more accurately detect the drowsiness state.

Also, as the subject enters a higher intensity drowsiness state, the duration of the reduced heart rate due to drowsiness will be longer. Therefore, if the drowsiness detecting device 6000 measures the length of time the reduced heart rate lasts, it may be possible to grasp which stage of the drowsy state the subject has entered. That is, by dividing the stages of the drowsiness state according to the length of the duration, the drowsiness detection device 6000 may more specifically detect the drowsiness state of the subject.

Referring to FIG. 41 according to an embodiment, the drowsiness detecting device 6000 has a heart rate 6001 of a subject based on a time point t1 where the heart rate 6001 of the subject and the reference heart rate 6002 are less than or equal to the reference heart rate 6002. The duration of the state below the reference heart rate 6002 can be measured.

For example, the reference heart rate 6002 may be set to 72 (times/minute) corresponding to 90% of the first resting heart rate. Here, when the heart rate of the subject is 74 (times/minute) at the first time point and the heart rate of the subject is 72 (times/minute) at the second time point immediately after the first time point, the drowsiness detecting device 6000 is the second time point. From, it is possible to measure the duration of the heart rate of the subject being 72 (times/minute) or less.

Of course, according to another embodiment, the drowsiness detecting device 6000 may measure the duration of a state in which the heart rate of the subject is equal to or greater than the reference heart rate based on the heart rate of the subject and the time point when the reference heart rate becomes abnormal.

For example, the reference heart rate 6002 may be set to 72 (times/minute) corresponding to 90% of the first resting heart rate. Here, when the heart rate 6001 of the subject at the third time point is 70 (times/minute) and the heart rate 6001 of the subject at the fourth time point immediately after the third time point is 72 (times/minute), the drowsiness detection device ( 6000) may measure a time duration in which the heart rate 6001 of the subject is 72 (times/minute) or more from the fourth time point.

Referring to FIG. 42 according to an embodiment, when measuring the duration of the state in which the heart rate 6001 of the subject is equal to or less than/above the reference heart rate 6002, the heart rate including noise The duration can be measured by correcting the measured time interval (Δtn). For example, the sleepiness detection device 6000 may correct a heart rate detected as noise based on a previously measured heart rate. As another example, the time interval Δtn in which the heart rate including noise is measured may be excluded when measuring the duration.

Here, the noise indicates an error caused by various causes such as movement of the subject or external light, and the heart rate including the noise may be difficult to reflect the state of the subject. In general, a heart rate including noise is temporarily acquired, and has a large deviation from a previously acquired heart rate of a subject and a later acquired heart rate.

Based on this, in one embodiment, the drowsiness detecting device 6000 may measure a heart rate 6001 obtained previously and/or a heart rate 6001 obtained later and a heart rate having a large deviation by a predetermined value or more. When obtained, the heart rate 6001 of the subject may be detected as a heart rate including noise. For example, when the drowsiness detection device acquires a heart rate that has a deviation of 20 or more from a previously detected heart rate and/or a heart rate acquired later during a time period of 3 seconds or less, the heart rate of the subject acquired during the 3 seconds (6001) can be detected as a heart rate containing noise. Here, the time at which the heart rate including the noise is acquired and the deviation between the heart rate including the noise and other heart rates are not limited to a specific value.

According to an embodiment, even though the heart rate including noise is a heart rate greater than or equal to the reference heart rate 6002, the drowsiness detection device 6000 measures a duration in which the heart rate 6001 of the subject is less than or equal to the reference heart rate 6002. , You can treat the noise as a heart rate below the reference heart rate and measure the duration.

According to an embodiment, although the heart rate including noise is a heart rate less than or equal to the reference heart rate, the drowsiness detection device 6000 refers to the noise as a heart rate greater than or equal to the reference heart rate when measuring a duration in which the heart rate of the subject is equal to or greater than the reference heart rate. Can be treated and the duration measured.

According to an embodiment, although the heart rate including noise is a heart rate less than or equal to the reference heart rate, the drowsiness detection device 6000 measures a heart rate including noise when measuring a duration in which the heart rate of the subject is less than or equal to the reference heart rate. It can be measured excluding the time interval (Δtn).

According to an embodiment, although the heart rate including noise is a heart rate less than or equal to the reference heart rate, the drowsiness detection device 6000 measures a heart rate including noise when measuring a duration in which the heart rate of the subject is greater than or equal to the reference heart rate. It can be measured excluding the time interval (Δtn).

Referring to FIG. 41 according to an embodiment, in the step-by-step drowsiness detection step (S6140), the drowsiness detection device 6000 may detect a drowsiness state or a normal state by comparing the measured duration and the reference duration. have.

The drowsiness detection device 6000 measures the duration of a state in which the heart rate 6001 of the subject is less than or equal to the reference heart rate, and detects the drowsiness when the measured duration reaches the reference duration (Δta, Δtb, Δtc). I can. In this case, the reference duration (Δta, Δtb, Δtc) may be plural. For example, the reference duration (Δta, Δtb, Δtc) may be three, the first reference duration (Δta) is 30 seconds, the second reference duration (Δtb) is 60 seconds, and the third reference duration ( Δtc) can be set to 90 seconds.

At this time, when the duration of the heart rate 6001 of the subject being equal to or less than the reference heart rate 6002 reaches 30 seconds, the drowsiness detecting device may detect that the subject is in the first stage drowsiness state.

At this time, when the duration of the heart rate 6001 of the subject being equal to or less than the reference heart rate 6002 reaches 60 seconds, the drowsiness detecting device 6000 may detect that the subject is in the second stage drowsiness state. .

At this time, when the duration of the heart rate 6001 of the subject being equal to or less than the reference heart rate 6002 reaches 90 seconds, the drowsiness detecting device 6000 may detect that the subject is in the third stage drowsiness state. .

In addition, the drowsiness detection device 6000 may maintain the sensed drowsiness state until it detects another stage of drowsiness or a normal state for the subject in which the drowsiness state is detected.

In another embodiment, if the duration of the heart rate 6001 of the subject in a state that is less than or equal to the reference heart rate 6002 does not reach the first reference duration Δta, the drowsiness detecting device 6000 is the subject's normal state. It can be detected as being.

For example, if the first reference duration (Δta) is 30 seconds and the subject's heart rate (6001) is less than or equal to the reference heart rate (6002) for 10 seconds and then recovers to a heart rate equal to or higher than the reference heart rate, the drowsiness detection device 6000 ) Can be detected that the subject is in a normal state.

Hereinafter, a method of detecting the recovery of the drowsy state based on the heart rate will be described with reference to FIG. 43.

In addition, in the state detection step (S6140) related to drowsiness, the drowsiness detection device 6000 detects that the measured duration has recovered from the drowsy state when the measured duration reaches the recovery reference duration (Δtd, Δte, Δtf). can do.

The recovery of the drowsiness state may mean a state that the subject has escaped from the sensed drowsiness state after entering the drowsiness state. Specifically, the recovery of the drowsiness state may refer to a situation in which a drowsiness state at a lower stage than the drowsiness state at a sensed stage is detected after the drowsiness state at a predetermined stage is detected. In addition, recovery of the drowsy state may mean a situation in which a normal state is detected after the drowsiness state is detected.

According to an embodiment, when the duration of the state in which the heart rate 6001 of the subject whose drowsiness is detected is equal to or greater than the recovery reference heart rate 6004 reaches the recovery reference duration (Δtd, Δte, Δtf), drowsiness is detected. The device 6000 may detect the recovery of the drowsy state with respect to the subject.

Specifically, referring to (a) of FIG. 46 according to an embodiment, the drowsiness detection device 6000 is the duration of a state in which the heart rate 6002 of the subject is equal to or greater than the recovery reference heart rate 6004 from the time point t1 Is measured, and at the time points (t1, t2, t3) when the duration reaches the recovery reference duration, it is possible to detect that the subject has recovered from the drowsiness state.

Here, the recovery criterion duration time recovery criterion duration may be set in various ways. For example, recovery duration time recovery reference duration may be set in various units such as several seconds, tens of seconds, minutes, tens of minutes.

For example, the recovery reference duration may be set to 30 seconds. In this case, the drowsiness detection device 6000 detects that the test subject has recovered to a normal state when the duration of the state in which the heart rate of the subject detected as the third stage drowsy state is more than the recovery reference heart rate 6004 reaches 30. can do.

According to an embodiment, a plurality of recovery reference durations may be set. For example, the recovery standard duration may be 3, the first recovery standard duration may be set to 20 seconds, the second recovery standard duration time may be set to 40 seconds, and the third recovery standard duration time may be set to 60 seconds. Here, the first to third recovery reference durations are not limited to the values suggested in the above example.

Referring to Figure 43 (a) according to an embodiment, the drowsiness detection device 6000 is the heart rate 6001 of the subject detected as the third stage drowsiness state, the heart rate 6002 of the subject is the recovery reference heart rate ( 6004) From the time point (t1) above, the duration of the state of being equal to or higher than the recovery reference heart rate (6004) is measured, and at the time point (t2) when the duration reaches the first recovery reference duration (t2), the subject is drowsy in the second stage. It can be detected as being restored to the state.

In addition, after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the drowsiness detection device 6000 operates from a time point t1 when the subject's heart rate 6002 is greater than or equal to the recovery reference heart rate 6004. The duration of the heart rate (6001) of which is greater than or equal to the recovery reference heart rate (6004) is measured, and at a time point (t3) when the duration reaches the second recovery reference duration (t3), the test subject drowsiness detection device 6000 May detect that the subject has recovered to the first stage drowsiness state.

However, referring to FIG. 43(b) according to an embodiment, after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the subject's heart rate 6002 is greater than or equal to the recovery reference heart rate 6004. From the time point t4, before the duration of the duration of the heart rate 6001 of the subject being equal to or greater than the recovery reference heart rate 6004 reaches the second recovery duration, the drowsiness detecting device 6000 is the heart rate ( When 6001) detects a time point t6 that is less than or equal to the recovery standard heart rate 6004, the drowsiness detection device 6000 can maintain the subject as a second-stage drowsy state, and resets to a third-stage drowsiness state again. You may.

It goes without saying that the drowsiness detection device 6000 can detect the recovery of the drowsy state based on the duration of the state in which the average heart rate 6003 of the subject is equal to or greater than the recovery reference heart rate 6004.

The recovery reference heart rate 6004 may be the same as or different from the reference heart rate 6002 for detecting a drowsy state.

The recovery reference duration may be the same as or different from the duration for detecting a drowsy state.

8.3 Drowsiness detection method based on LF/HF

Hereinafter, a method of detecting drowsiness based on LF/HF according to an embodiment will be described with reference to FIGS. 44 to 47.

In one embodiment, the drowsiness detection method based on LF/HF is the step of acquiring the LF/HF 6005 of the subject (S6210), comparing the size of the LF/HF 6005 of the subject with the reference LF/HF It may include a step (S6220) and detecting drowsiness step by step based on the comparison result (S6230).

The drowsiness detection method may further include giving a notification according to the drowsiness step to the outside according to an embodiment.

In the step of acquiring the LF/HF (S6210), the heart rate information acquisition unit 6100 may acquire the LF/HF 6005 of the subject.

In an embodiment, the drowsiness detecting device 6000 may acquire the LF/HF 6005 of the subject according to a predetermined time period. In another embodiment, the drowsiness detecting device 6000 may acquire the LF/HF 6005 of the subject without following a predetermined time period.

Of course, if the LF/HF 6005 of the subject is measured in the biometric index acquisition device 10 or other device according to a predetermined time period, the drowsiness detection device 6000 is measured by the biometric index acquisition device 10 or another device. The LF/HF 6005 of the subject may be acquired based on the predetermined time period.

As another example, the drowsiness detection device 6000 may request to provide the LF/HF to the biometric index acquisition device 10 or another device according to an external input or request, and each time the request is made, the subject Can acquire LF/HF (6005) of

In an embodiment, the drowsiness detecting device 6000 may obtain an average LF/HF of a subject for a predetermined time period.

As the drowsiness detection device 6000 obtains the average LF/HF, noise is corrected to have the effect of accurately detecting drowsiness, and a detailed description thereof will be omitted since it is the same as the effect obtained by obtaining the average heart rate. .

Referring to FIG. 45 according to an embodiment, in the step (S6220) of comparing the acquired LF/HF (6005) and the reference LF/HF (6006, 6007, 6008), the drowsiness detecting device 6000 is obtained. It is possible to compare the LF/HF (6005) of the measured subject and the reference LF/HF.

Here, the reference LF/HF (6006, 6007, 6008) may mean a value set as a reference for distinguishing between a drowsy state and a normal state. As described above, according to the activation of the parasympathetic nervous system, the LF/HF in the drowsy state may be smaller than the LF/HF in the normal state.

In one embodiment, the drowsiness detection device 6000 uses the LF/HF average of the heart rate signal acquired for a certain time from the time when the LF/HF of the heart rate signal is first acquired, and uses the reference LF/HF (6006, 6007, 6008). Can be set. The predetermined time may be set to various values such as 1 minute and 5 minutes. In general, there is a high possibility that the subject is in a normal state for a certain period of time from the time when the LF/HF 6005 of the subject is first acquired. Accordingly, the average LF/HF of the subject acquired during the predetermined time is obtained, and a predetermined value a for the average LF/HF (eg, a represents a real number of 1 or less. As a more specific example, a is 0.9) can be obtained as the reference LF/HF. In addition, the drowsiness detection device may set a plurality of reference LF/HFs by using other predetermined real numbers. The predetermined value a is not limited to the above example.

In another embodiment, the drowsiness detection device 6000 may set the reference LF/HF (6006, 6007, 6008) using the resting LF/HF and/or the active LF/HF. In this case, there may be a plurality of reference LF/

HFs

6006, 6007, and 6008.

Specifically, the drowsiness detection device 6000 uses a camera or a wearable device based on the first resting LF/HF based on the first resting heart rate, the second resting LF/HF based on the second resting heart rate, and the active heart rate. The active period LF/HF can be obtained.

According to an embodiment, the reference LF/HF (6006, 6007, 6008) may be set based on the first resting period LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) may be set at a constant ratio of the first resting period LF/HF. For example, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006) multiplied by a predetermined value 0.9 by the first resting LF/HF, and the second reference LF/HF (6007) multiplied by 0.8 And a third reference LF/HF 6008 multiplied by 0.7. Of course, the predetermined value is not limited to the above example.

According to another embodiment, the reference LF/HF (6006, 6007, 6008) may be set based on the second resting period LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) may be set at a constant ratio of the first resting period LF/HF. For example, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006) multiplied by a predetermined value 1.5 by the first resting LF/HF, and the second reference LF/HF (6007) multiplied by 1.3. And a third reference LF/HF 6008 multiplied by 1.1. Of course, the predetermined value is not limited to the above example.

In addition, according to another embodiment, the reference LF/HF (6006, 6007, 6008) may be set based on the active phase LF/HF. Specifically, the reference LF/HF (6006, 6007, 6008) is the first reference LF/HF (6006) multiplied by the active phase LF/HF by a predetermined value of 0.8, the second reference LF/HF (6007) and 0.6 It may be set as the third reference LF/HF 6008 multiplied by. Of course, the predetermined value is not limited to the above example.

The method of setting the reference LF/HF (6006, 6007, 6008) is not limited to a method of calculating a constant ratio of the first and second resting LF/HF and active LF/HF, and various operations such as sum and difference are applied. Of course you can. In addition, in another embodiment, the drowsiness detection device 6000 may change from the previously set reference LF/HF (6006, 6007, 6008) to the new reference LF/HF (6006, 6007, 6008).

For example, as described above, the drowsiness detection device 6000 is based on the heart rate acquired for a certain period of time from the time when the LF/HF 6005 of the subject is first acquired, and is based on the reference LF/HF (6006, 6007, 6008). ) Can be set. The sleepiness detection device 6000 can detect the normal state based on the LF/HF 6005 of the subject acquired after the predetermined time, and the average LF/HF in the time interval detected as the normal state is a new standard. It can be changed to LF/HF (6006, 6007, 6008). In addition, the drowsiness detection device 6000 is changed to a new standard LF/HF (6006, 6007, 6008) based on the first and second LF/HF and active phase LF/HF obtained for the same subject after the predetermined time. I can.

For another example, the drowsiness detection device 6000 may preset the reference LF/HF (6006, 6007, 6008) based on the first and second resting periods and active periods LF/HF as described above. At this time, after the reference LF/HF (6006, 6007, 6008) is set based on the first and second resting LF/HF and active LF/HF, the drowsiness detecting device 6000 is A new criterion LF/HF (6006, 6007, 6008) may be set based on the newly set first and second resting period LF/HF and active period LF/HF.

In addition, the drowsiness detection device 6000 may set a new reference LF/HF (6006, 6007, 6008) based on the LF/HF acquired for a certain time from the time when the LF/HF is first acquired.

In addition, the drowsiness detection device 6000 may set a new reference LF/HF (6006, 6007, 6008) based on the average LF/HF during a time period in which the subject is detected as a normal state.

Referring to Figure 45 according to an embodiment, the drowsiness detection device 6000 is the LF / HF (6005) of the subject is more than or less than any one of a plurality of reference LF / HF (6006, 6007, 6008). The time points t1, t2, t3, and t4 can be detected. The sensed point in time may be a point in time for measuring the drowsiness state and the normal state in each step.

In one embodiment, the drowsiness detection device 6000 may detect the drowsiness state at a time point (t1, t2, t3) when the LF/HF (6005) of the subject is less than or equal to the reference LF/HF (6006, 6007, 6008). have.

In one embodiment, the drowsiness detection device 6000 is the point at which the LF/HF 6005 of the subject becomes less than or equal to one of the plurality of reference LF/

HFs

6006, 6007, and 6008 (t1, t2, At t3), the drowsiness state of the step corresponding to the one reference LF/HF may be detected. Here, the drowsiness detection device 6000 may detect the drowsiness based on the result of additionally comparing the LF/HF 6005 of the subject and the one reference LF/HF as well as the other reference LF/HF. .

Specifically, the plurality of reference LF/HF (6006, 6007, 6008) may be set as a first reference LF/HF (6006), a second reference LF/HF (6007), and a third reference LF/HF (6008). . For convenience of description below, the first reference LF/HF 6006 is defined as the reference LF/HF for detecting the lowest intensity of drowsiness.

For example, if the LF/HF (6005) of the subject at the first time point is less than or equal to the first reference LF/HF (6006) and is greater than or equal to the second reference LF/HF (6007), the drowsiness detecting device 6000 is the subject The drowsiness state of may be determined as the first stage drowsiness state.

In addition, when the LF/HF (6005) of the subject at the first time point is less than or equal to the second reference LF/HF (6007) and is equal to or greater than the third reference LF/HF (6008), the drowsiness detection device 6000 is The state may be determined as the second stage drowsy state.

In addition, when the LF/HF 6005 of the subject at the first time point is less than or equal to the third standard LF/HF 6008, the drowsiness detection device 6000 may determine the drowsiness state of the subject as the third stage drowsiness state. have.

Hereinafter, a method of detecting the recovery of a drowsy state based on LF/HF will be described with reference to FIG. 46.

Referring to Figure 46 (a) according to an embodiment, in the step of detecting a state related to drowsiness (S6230), the drowsiness detecting device 6000 is the measured LF / HF (6005) of the measured subject is the recovery criterion LF /HF (6006, 6007, 6008, 6010) or higher at the time point (t1, t2, t3, t4), the subject can detect that he has recovered from drowsiness. Here, the drowsiness detection device 6000 detects the recovery of the drowsy state based on the result of additionally comparing the LF/HF 6005 of the subject and the one recovery criterion LF/HF and another recovery criterion LF/HF. I can.

In one embodiment, the plurality of criteria LF/HF (6006, 6007, 6008, 6010) are the first recovery criteria LF/HF (6006), the second recovery criteria LF/HF (6007), and the third recovery criteria LF/HF. It can be set to (6008). For convenience of description below, the first recovery criterion LF/HF 6006 is set to be the recovery criterion LF/HF having the largest value among the recovery criterion LF/HF of 1 or less. For example, at the first time point, the subject's LF/HF (6005) is less than or equal to the third recovery criterion LF/HF (6008), and at the second time point thereafter, the subject's LF/HF (6005) recovers to the third time. If the standard LF/HF (6008) or more and the second recovery standard LF/HF (6007) or less, the drowsiness detection device 6000 is the second stage drowsiness in the third stage drowsiness state at the second time point. It can be judged that it has recovered to its state. In addition, at the third time point after the subject recovers from the third stage drowsiness state to the second stage drowsiness state, the drowsiness detection device 6000 determines that the LF/HF 6005 of the subject is the second recovery criterion LF/HF ( 6007) or more and less than the first recovery criterion LF/HF (6006), the drowsiness detection device 6000 determines that the drowsy state of the subject has recovered from the second stage drowsiness state to the first stage drowsiness state at the third time point. can do.

In one embodiment, the LF/HF (6005) of the subject at the first time point is less than or equal to the third recovery criterion LF/HF (6008), and at the second time point thereafter, the LF/HF (6005) of the subject is at the second time point. If the recovery criterion LF/HF (6007) or more and the first recovery criterion LF/HF (6006) or less, the drowsiness detection device 6000 is the first step in the third step drowsy state at the second point in time. It can be judged that he has recovered to a drowsy state.

However, referring to (b) of FIG. 46 according to an embodiment, after the time point t6 the subject recovers from the third stage drowsiness state to the first stage drowsiness state, the drowsiness detecting device 6000 is When the LF/HF (6005) of the second recovery criterion LF/HF (6007) or less is detected within a certain period of time (t7), the drowsiness detection device 6000 is Can be detected. In addition, it may be detected that the subject has reset to the third stage drowsiness state.

The first recovery criterion LF/HF 6006 is the first criterion LF/HF 6006 in FIG. 45, and the second recovery criterion LF/HF 6010 is the second criterion LF/HF ( 6007) and the third recovery criterion LF/HF 6011 may be the same as or different from the third criterion LF/HF 6006 in FIG. 45.

In addition, a fourth recovery criterion LF/HF 6010 having a value greater than the first to third recovery criterion LF/HF may be set in order to detect that the drowsiness detection device 6000 recovers from the drowsy state to the normal state. have. For example, when the drowsiness detection device 6000 detects a point in time when the LF/HF 6005 of the subject detected as drowsiness becomes more than the fourth recovery criterion LF/HF 6010, the subject is You can also judge that you have recovered from a drowsy state to a normal state. In general, when the subject is in a normal state, LF/HF is measured within the range of 0.9 to 1. Therefore, even when the subject is detected as a very small intensity of drowsiness, it may be difficult for the subject's LF/HF to have a value of 1 or more. That is, by setting the fourth recovery criterion LF/HF (6010) having a value of 1 or more as the recovery criterion LF/HF, the drowsiness detection device 6000 is determined to be in a normal state only when the subject completely recovers from a drowsy state. You may.

8.4 Drowsiness detection method based on heart rate and LF/HF

Hereinafter, a method of detecting drowsiness based on heart rate and LF/HF will be described with reference to FIG. 47.

In an embodiment, the step of detecting drowsiness based on the heart rate and LF/HF (S6100, S6200) and determining the final drowsiness state (S6300) may be included.

The method of detecting drowsiness may further include, according to an embodiment, giving a notification to an object other than the person to be measured/not to be measured according to the stage of the drowsiness state.

This table of contents describes the drowsiness detection method in which both the heart rate-based drowsiness detection method described in Table of Contents 8.3 and the LF/HF-based drowsiness detection method described in Table of Contents 8.4 are applied.

The heart rate and LF/HF are due to information about the heart rate obtained from the same subject, but in some cases, the level of drowsiness determined based on the heart rate and the level of drowsiness determined based on the LF/HF may be different. have. For example, the drowsiness detection device may detect a three-stage drowsiness state based on a heart rate at the same time point from the same subject and a second-stage drowsiness state based on LF/HF. In this way, the fact that the drowsiness detection device judges the drowsiness state for the same subject at the same time, but the different drowsiness stages are determined according to the drowsiness detection method is a possibility that the result of determining the drowsiness state by any one of the drowsiness detection methods is wrong. This can mean that there is.

Therefore, in order to reduce such errors and more accurately detect drowsiness, a description will be given of a drowsiness detection method based on both heart rate and LF/HF.

In an embodiment, the state of sleepiness detected in the method for detecting sleepiness based on heart rate and LF/HF may be expressed as a final sleepiness state and a final normal state. In one embodiment, the final drowsiness state may be divided into a plurality of stages according to the intensity of drowsiness. For example, it may be divided into a first final drowsy state, a second final drowsy state, and a third final drowsy state. Here, the first final drowsiness state may mean the final drowsiness state with the lowest intensity, and the third final drowsiness state may mean the final drowsiness state with the highest intensity.

In one embodiment, the final drowsiness state and the final normal state mean a state corresponding to the drowsiness state and the normal state detected based on each of the drowsiness detection method based on heart rate and the drowsiness detection method based on LF/HF. I can. In other words, the final drowsiness state may mean a state in which the subject is temporarily sleeping or a state that is likely to sleep within a predetermined time before sleeping, and the final normal state is not the final drowsiness state, but a state determined by the subject. This can mean a condition where you are less likely to fall asleep within time.

As described above, the drowsiness detection method based on the heart rate change and LF/HF value is divided into four stages: a normal state, a first stage drowsy state, a second stage drowsy state, and a third stage drowsy state, respectively, depending on the intensity of drowsiness. Can be divided. Here, the first-stage drowsiness state may mean an involuntary/unconscious drowsiness state, and the second and third-stage drowsiness states may mean a self-conscious/conscious drowsiness state.

In one embodiment, the final drowsiness state divided into a plurality of stages according to the intensity of drowsiness is the first stage drowsiness detected based on each of the drowsiness detection method based on heart rate and the drowsiness detection method based on LF/HF It may mean a state, a state corresponding to the second step drowsy state, and the third step drowsy state. Therefore, the first final drowsiness state may mean an involuntary/unconscious drowsiness state. In addition, the second final drowsiness state and the third final drowsiness state may mean a subjective/conscious drowsiness state. In one embodiment, the drowsiness detection device may acquire the final drowsiness state of the subject according to a predetermined period of time. Of course, the predetermined time period may be fixed or may be variable.

In another embodiment, the device for detecting drowsiness may acquire a final drowsiness state of the subject without following a predetermined period of time. For example, the drowsiness detection device may acquire the subject's sol whenever the subject's heart rate and/or LF/HF are measured by the biometric index acquisition device 10 or another device, that is, in real time.

Of course, if the heart rate or LF/HF of the subject is measured in the biometric index acquisition device 10 or another device according to a predetermined time period, the drowsiness detection device is a predetermined time period measured by the biometric index acquisition device 10 or another device. The final drowsiness state of the subject can be obtained based on.

As another example, the drowsiness detection device may request the biometric index acquisition device 10 or other device to provide LF/HF and heart rate according to an external input or request, and each time the request is made, the subject's Final drowsiness can be achieved

In an embodiment, the drowsiness detection device may obtain an average heart rate and an average LF/HF of a subject for a predetermined time period.

The drowsiness detection device can have the effect of accurately detecting drowsiness by correcting the noise as it acquires the average heart rate and average LF/HF, and is the same as the effect that can be obtained according to the average heart rate and average LF/HF acquisition. Description will be omitted.

In an embodiment, the drowsiness detection apparatus may determine the final drowsiness stage according to the level of the drowsiness state sensed based on the heart rate and the level of the drowsiness state sensed based on the LF/HF. In some cases, the level of the sleepy state detected based on the heart rate and the level of the sleepy state detected based on the LF/HF may or may not coincide.

Hereinafter, for convenience of description, the normal state detected based on the heart rate is defined as a drowsiness state of a lower stage than the first stage drowsiness state. In addition, the normal state detected based on the LF/HF is also defined as a lower level of drowsiness than the first stage of drowsiness.

When the level of the sleepy state sensed based on the heart rate and the level of the sleepy state sensed based on the LF/HF match, the drowsiness detection device may determine the matching step as the final sleepy state. For example, when the drowsiness detection device detects the first stage drowsiness state based on the heart rate of the subject and the first stage drowsiness state is detected based on the LF/HF, the drowsiness detection apparatus returns the first stage to the final drowsiness state. The final drowsiness state can be determined. In addition, when the drowsiness detection device detects the second stage drowsiness state based on the heart rate of the subject and detects the second stage drowsiness state based on the LF/HF, the drowsiness detection apparatus returns the final drowsiness state to the second stage final. Can determine drowsiness. In addition, if the drowsiness detection device detects the third stage drowsiness state based on the heart rate of the subject and detects the third stage drowsiness state based on the LF/HF, the drowsiness detection apparatus returns the final drowsiness state to the third stage. Can determine drowsiness.

When the level of the sleepy state sensed based on the heart rate and the level of the sleepy state sensed based on the LF/HF do not match, the drowsiness detection device may determine the final sleepiness level in consideration of various situations.

In one embodiment, if the level of the sleepy state detected based on the heart rate and the level of the sleepy state detected based on the LF/HF do not match, the drowsiness detection device selects the higher of the two stages to the final sleepy state. Can be determined by For example, if the drowsiness detection device detects the first stage drowsiness condition based on the heart rate of the subject and the second stage drowsiness condition is detected based on the LF/HF, the drowsiness detection apparatus returns the second stage to the final drowsiness condition. The final drowsiness state can be determined.

In another embodiment, if the level of the sleepy state detected based on the heart rate and the level of the sleepy state detected based on the LF/HF do not match, the drowsiness detection device selects the lower of the two stages to the final sleepy state. Can be determined by For example, when the drowsiness detection device detects the first stage drowsiness condition based on the heart rate of the subject and the second stage drowsiness condition is detected based on the LF/HF, the drowsiness detection apparatus returns the first stage to the final drowsiness condition. The final drowsiness state can be determined.

In another embodiment, when the level of the sleepy state detected based on the heart rate and the level of the sleepy state detected based on the LF/HF do not match, the drowsiness detection device determines the average of the two stages as the final sleepy state. I can. Here, when the average is not an integer, rounding or rounding may be applied to the number of the first decimal place. For example, if the drowsiness detection device detects the first stage drowsiness condition based on the heart rate of the subject and the third stage drowsiness condition is detected based on the LF/HF, the drowsiness detection apparatus is The final drowsiness state of the second stage, which is the average of the stages, can be determined. For another example, if the first stage drowsiness is detected based on the heart rate and the second stage drowsiness is detected based on the LF/HF, the drowsiness detection device is the final drowsiness state with respect to the average of the two stages, 1.5. It is possible to determine the final drowsiness state of the first stage by applying a drop in the first decimal place. In addition, the drowsiness detection device may determine the final drowsiness state of the second step by applying rounding up from the first decimal place to 1.5, which is the average of the two phases, as the final drowsiness state.

Of course, in some cases, the drowsiness detection device newly defines the drowsiness phase between the drowsiness phase detected based on the heart rate and the drowsiness phase detected based on the LF/HF, and finalizes the newly defined drowsiness phase. It can be determined by the level of drowsiness. For example, when the first stage drowsiness state is detected based on the heart rate and the second stage drowsiness state is detected based on the LF/HF, the drowsiness detection device is An intermediate level of drowsiness indicating intensity may be defined, and the intermediate level of drowsiness may be determined as a final level of drowsiness.

In another embodiment, if the level of the drowsiness state detected based on the heart rate and the level of the drowsiness state detected based on the LF/HF do not match, the drowsiness detection device determines the final drowsiness state or the final normal state previously determined for drowsiness. It can be maintained as a state related to. For example, in a situation where the drowsiness detection device detects the final drowsiness state of the first step for the subject, it detects the drowsiness state of the first step based on the heart rate afterwards, and detects the drowsiness state of the third step based on the LF/HF. In one case, the drowsiness detecting device may maintain the final drowsiness state of the first stage, which is the final drowsiness state determined previously, as a state related to drowsiness. For example, if the drowsiness detection device detects the first stage drowsiness condition based on the heart rate in a situation where the drowsiness detection device detects the final normal condition for the subject and detects the third stage drowsiness condition based on the LF/HF, the drowsiness detection The device may maintain a previously determined final steady state as a state related to drowsiness.

Hereinafter, a detailed description will be given focusing on the case where the level of the drowsy state sensed based on the heart rate and the level of the drowsy state sensed based on the LF/HF do not match.

8.4.1 Specific Examples

In an embodiment, when the level of the drowsiness state sensed based on the heart rate and the level of the drowsiness state sensed based on the LF/HF coincide with the normal state, the drowsiness detection device may determine that the subject is in the final normal state. .

In an embodiment, when the level of the drowsiness state detected based on the heart rate and the level of the drowsiness state detected based on the LF/HF match as the first stage drowsiness, the drowsiness detection device is the subject of the first stage final drowsiness. It can be determined to be in a state.

In one embodiment, when the level of the drowsiness state detected based on the heart rate and the level of the drowsiness state detected based on the LF/HF coincide with the second stage drowsiness state, the drowsiness detection device is the subject to the second stage final drowsiness. It can be determined to be in a state.

In one embodiment, when the level of the drowsy state detected based on the heart rate and the level of the drowsy state sensed based on the LF/HF coincide with the third stage drowsy state, the drowsiness detection device is the subject of the third stage final drowsiness. It can be determined to be in a state.

In one embodiment, the level of the sleepy state detected based on the heart rate and the level of the sleepy state detected based on the LF/HF do not match, and the level of the sleepy state detected based on the heart rate and the LF/HF are detected. The method of determining the lower stage of the drowsiness state as the final drowsy state (hereinafter referred to as the first method) can be applied as a method for detecting a stage in which the need to be sensitively sensed is low. .

For example, when the first, second, and third stages of drowsiness are detected, a notification may be given. In some cases, the first and second stages of drowsiness may have less impact on the safety of the subject. In this case, if the notification is frequently given, the subject may feel uncomfortable or may not operate the drowsiness detection device. In addition, as the level of the drowsy state detected based on the heart rate and the level of the drowsy state detected based on the LF/HF is determined as the final drowsy state, the drowsiness detection device more accurately determines the state related to drowsiness. You can also detect it. That is, determining the final drowsiness state using the drowsiness detecting device according to the first method may be advantageous for improving user convenience and accuracy.

In one embodiment, the level of the drowsiness state detected based on the heart rate and the level of the drowsiness state detected based on the LF/HF do not match, and one of the sleepy states indicates a normal state, and the other drowsiness state When the step of is detected as a sleepiness state that is one or two steps higher than the normal state, the sleepiness detection device may determine that the subject is the final normal state based on the normal state, which is the lower level.

For example, when the drowsiness detection device detects the normal state based on the heart rate and detects the first stage drowsiness state based on the LF/HF, the drowsiness detection device may determine that the subject is the final normal state.

In one embodiment, the level of the drowsiness state detected based on the heart rate and the level of the drowsiness state detected based on the LF/HF do not match, and one of the stages of the drowsiness state indicates the first stage drowsiness state, and the other When the stage of the drowsy state of is detected as a drowsy state that is one level higher than the normal state, the drowsiness detecting device may determine that the subject is the first stage drowsy state based on the first stage drowsiness state, which is a lower level.

For example, when the drowsiness detection device detects the first stage drowsiness condition based on the heart rate and also detects the second stage drowsiness condition based on the LF/HF, the drowsiness detection apparatus is Can be determined.

In one embodiment, the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, and one of the stages of drowsiness indicates the second stage of drowsiness, and the other When the level of the drowsy state of is detected as a drowsy state higher than the normal state, the drowsiness detection device may determine that the subject is in the second phase drowsiness state based on the second phase drowsiness state, which is the lower level. On the other hand, heart rate The level of the sleepy state detected based on the LF/HF and the level of the sleepy state detected based on the LF/HF do not match, the level of the sleepy state detected based on the heart rate and the level of the sleepy state detected based on the LF/HF The method of determining the level of the highest level as the level of the final drowsy state (hereinafter referred to as the second method) may be applied as a method for detecting a level that needs to be sensitively detected.

For example, when the first, second, and third stages of drowsiness are detected, a notification may be given.In some cases, the third stage of drowsiness is the highest level of drowsiness and may directly affect the safety of the subject. I can. At this time, if the drowsiness detecting device does not give a strong notification to the subject as soon as the third stage drowsiness is detected, the subject may be in a serious danger situation. In addition, as the stage of drowsiness detected based on heart rate and the stage of drowsiness detected based on LF/HF is determined as the stage of final drowsiness, the drowsiness detection device seeks to detect more various situations. The step to do can be determined as the step of the final drowsiness state. That is, it may be advantageous to prepare for user risk to determine the final drowsiness state by using the drowsiness detecting device according to the second method.

In one embodiment, the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not match, and one of the stages of drowsiness indicates the third stage of drowsiness, and the other When the drowsy state of the third stage is detected as one, two, or three lower drowsiness conditions than the third stage drowsiness, the drowsiness detection device is based on the third stage drowsiness state, which is the higher level, and the subject is in the third stage. It can be determined that you are in the final drowsy state.

For example, if the drowsiness detection device detects the third stage drowsiness state based on the heart rate and detects the first stage drowsiness state based on the LF/HF, the drowsiness detection device indicates that the subject is in the third stage drowsiness state. You can decide.

In another embodiment, the drowsiness detection device may follow the second method to detect the second final drowsiness state. This is because, in some cases, the drowsiness detection device may need to detect not only the third stage drowsiness condition but also the second stage drowsiness condition in more various situations. Specifically, when the subject is a driver of a freight vehicle or a driver of a public transportation vehicle, when the driver is in danger of a traffic accident, a situation in which the driver and the passengers of surrounding vehicles other than the driver are in serious danger may occur. Therefore, the level of drowsiness detected based on heart rate and the level of drowsiness detected based on LF/HF do not coincide, and one of the stages of drowsiness indicates the second stage of drowsiness, and the other When the step of is detected as a sleepy state one or two steps lower than the sleepiness of the second step, the drowsiness detecting device determines that the final sleepiness of the second step is based on the second step, which is the high level. May be.

For example, when the drowsiness detection device detects the second stage drowsiness state based on the heart rate and detects the first stage drowsiness state based on the LF/HF, the drowsiness detection device indicates that the subject is in the second stage drowsiness state. You can decide.

9. Various applications using biometric index acquisition device

9.1 Various embodiments of the display device

The above-described biometric index acquisition method or drowsiness detection method may be used in various applications. For example, the method of obtaining the biometric index or the method of detecting drowsiness may be used in a display device.

In this case, the display device includes a smart mirror including a mirror display, but is not limited thereto, and may mean a device including a display such as a smart phone or a tablet.

*In addition, the biometric index or biometric information obtained according to the biometric index acquisition method may be output through the display, and the sleepiness information obtained according to the drowsy detection method may also be output through the display.

In addition, the display device may perform an operation corresponding to the biometric index or biometric information obtained according to the biometric index acquisition method.

In addition, the display device may perform an operation corresponding to the drowsiness information obtained according to the drowsiness detection method.

* In addition, the user of the display device may set the biometric index of interest, and when the biometric index of interest is set, only the biometric index of interest may be obtained or output. For example, when the first user sets the heart rate and the second user sets the blood pressure, only the heart rate may be acquired for the first user and only the blood pressure may be acquired for the second user, but is not limited thereto.

In addition, the display device may be installed in a public place such as a hotel lobby, a hotel front desk, and a public restroom and used to measure the biometric index of people who come and go, but the present invention is not limited thereto.

In addition, hereinafter, the operation of the biometric index measuring device will be described, but the description may be performed by other processors such as an ECU mounted on a vehicle, and may also be performed by a processor included in a server.

In addition, the device described as the biometric index measuring device may mean an image acquisition device, and in this case, an operation according to the described content may be performed by another processor such as an ECU or a processor included in the server.

Referring to FIG. 48, a smart mirror device 7000 according to an embodiment may include at least one of an image sensor 7010, a mirror display 7020, and a control unit 7030. For example, the smart mirror device 7000 may be configured with the mirror display 7020 and the controller 7030, but is not limited thereto.

In this case, the image sensor 7010 may be provided as a visible camera for acquiring a visible light image, an infrared camera for acquiring an infrared image, but is not limited thereto. A hybrid type camera for acquiring an infrared image may be provided.

In addition, the image sensor 7010 may be provided as one package with the mirror display 7020, but is not limited thereto and may be provided as a separate unit separated from the mirror display 7020.

In addition, the image sensor 7010 may acquire a plurality of image frames and transmit the acquired image to the controller 7030.

In addition, the mirror display 7020 may mean a medium capable of transmitting information while functioning as a mirror, but is not limited thereto.

In addition, the mirror display 7020 may include a mirror capable of functioning as a mirror and a display capable of transmitting information, and may be provided in a form in which a mirror film is added to the display, but is not limited thereto.

In addition, the mirror display 7020 may include a half mirror that transmits light in one direction but reflects it in another direction, but is not limited thereto.

In addition, the mirror display 7020 may include a polarizing plate, but is not limited thereto.

In addition, the mirror display 7020 may include a mirror display that can be commonly understood in addition to the above-described exemplary principles and exemplary elements. More specifically, the mirror display 7020 is a device that functions as a mirror and transmits information. Can be understood.

Also, the controller 7030 may obtain a biometric index and biometric information based on a plurality of image frames obtained from the image sensor 7010.

In this case, the method of obtaining the biometric index and biometric information by the control unit 7030 based on the plurality of image frames obtained from the image sensor 7010 may be applied to the above description, so a redundant description will be omitted. .

In addition, the controller 7030 may obtain a biometric index and biometric information from an external sensor (not shown). For example, the controller 7030 may acquire heart rate information through an ECG sensor attached to a subject, but is not limited thereto.

In addition, the controller 7030 may acquire personal and statistical data through an input device (not shown). For example, the control unit 7030 may obtain personal and statistical data such as height, age, and weight of a subject through a keyboard, but is not limited thereto.

In this case, the input device may be a keyboard, a mouse, a key pad, a dome switch, a touch pad (positive pressure/power failure), a jog wheel, a jog switch, and the like. It is not limited to this.

In addition, the controller 7030 may acquire personal and statistical data through an external device (not shown). For example, the controller 7030 may obtain personal and statistical data such as height, age, and weight of the subject through the subject’s smartphone, but is not limited thereto.

In this case, the external device may include a mobile terminal such as a mobile phone, a smart phone, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), and a navigation system In addition, it may include a fixed terminal such as a digital TV or a desktop computer, but is not limited thereto.

In addition, the controller 7030 may control an operation of the mirror display 7020. For example, the controller 7030 may control the operation of the mirror display 7020 to output the biometric index or biometric information obtained based on the image frame, but is not limited thereto.

In addition, the controller 7030 may control an operation of the mirror display 7020 to output various information. For example, the controller 7030 may control the operation of the mirror display 7020 to output various information such as weather information, date information, calendar information, internal humidity information, and internal temperature information, but is not limited thereto. .

In addition, the controller 7030 may control the operation of the mirror display 7020 to output various personal information. For example, the controller 7030 may control the operation of the mirror display 7020 to output various personal information such as schedule information and medication information of a subject, but is not limited thereto.

Referring to FIG. 49, a smart mirror device 7100 according to an embodiment may include an image sensor 7110 and a mirror display 7120.

In this case, since the above-described operations can be applied to the image sensor 7110 and the mirror display 7120, a redundant description will be omitted.

According to an embodiment, basic information may be output on the mirror display 7120. For example, as shown in FIG. 49, at least one of basic information such as date information, time information, external temperature information, weather information, calendar information, news information, etc. may be output on the mirror display 7120. It is not limited to this.

In addition, personal information may be output on the mirror display 7120. For example, as illustrated in FIG. 49, at least one of personal information such as age, height, and weight of a subject may be output on the mirror display 7120, but is not limited thereto.

In addition, for example, as illustrated in FIG. 49, at least one of personal information such as schedule information and medication information of a subject may be output on the mirror display 7120, but is not limited thereto.

In addition, the biometric index of the subject may be output on the mirror display 7120. For example, as illustrated in FIG. 49, at least one biometric index, such as heart rate, oxygen saturation, and blood pressure, may be output on the mirror display 7120, but is not limited thereto.

In this case, the biometric index may be obtained based on the image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained by an external sensor or the like.

In addition, biometric information of the subject may be output on the mirror display 7120. For example, as illustrated in FIG. 49, at least one of biometric information, such as condition information, may be output on the mirror display 7120, but is not limited thereto.

In this case, the biometric information may be acquired based on the image frame acquired from the image sensor 7110, but is not limited thereto, and may be acquired by an external sensor.

In addition, the biometric information may be obtained based on at least one biometric index among the biometric indexes, but is not limited thereto.

In addition, a biosignal of a subject may be output to the mirror display 7120. For example, as illustrated in FIG. 49, at least one of bio signals, such as a heart rate signal, may be output to the mirror display 7120, but is not limited thereto.

In this case, the biosignal may be obtained based on an image frame obtained from the image sensor 7110, but is not limited thereto, and may be obtained based on an external sensor or the like.

In addition, the mirror display 7120 may include an input device. For example, as illustrated in FIG. 49, the mirror display 7120 may include at least one of input devices such as a touch panel, but is not limited thereto.

In addition, for example, although not shown in FIG. 49, the smart mirror device 7100 may track a user's pupil, or the like, or may receive an input from a user by recognizing a user's gesture, but is not limited thereto.

In addition, the mirror display 7120 may obtain information on the subject through the input device, and the obtained information on the subject may be output to the mirror display 7120.

Further, the drawings and description of the above-described mirror display 7120 are only an example, and it is obvious that various types of information may be output in various ways without being limited to FIG. 49 and the related description.

9.1.1 Various embodiments of the display device on which the guide area is output

Referring to FIG. 50, a smart mirror device 7150 according to an embodiment may include an image sensor 7160 and a mirror display 7170, and the image sensor 7160 and the mirror display 7170 are described above. Since one content may be applied, duplicate descriptions will be omitted.

In addition, a guide area 7180 may be displayed on the mirror display 7170 according to an exemplary embodiment. In this case, the guide area 7180 may function to approximate a measurement position of a measurement object.

When obtaining a biometric index using an image sensor, the measurement accuracy may vary depending on the measurement position of the subject. Therefore, when using the guide area 7180 that can guide the measurement position of the subject, It is possible to improve the accuracy of the biometric index measurement.

In addition, the guide area 7180 may be displayed in a rectangular shape as shown in FIG. 50, but is not limited thereto, and may be displayed in various shapes such as a human face contour, a circle, and an oval shape.

In addition, the displayed position of the guide area 7180 may be changed according to the subject. For example, the guide area 7180 for a tall subject may be located relatively above the mirror display 7170, and the guide area 7180 for a short subject may be relatively It may be located under the mirror display 7170, but is not limited thereto.

In addition, the position of the guide area 7180 may change in real time. For example, when the subject moves during measurement, the guide area 7180 may change in real time in response to the subject’s movement, but is not limited thereto.

In addition, the guide area 7180 may be displayed before measuring the biometric index of the subject. For example, the guide area 7180 may be displayed before the subject enters the measurement area and may function to inform the approximate measurement position of the subject, but is not limited thereto.

In addition, the guide area 7180 may function to inform a subject to be measured when there are a plurality of persons who may be subject to measurement. For example, when person A and person B enter the measurement area, but only the person B becomes the subject, the guide area 7180 may be displayed corresponding to the person B, but is not limited thereto.

9.1.2. Various embodiments of a display device in which predetermined information is displayed during biometric index measurement

In the case of obtaining a biometric index using an image sensor, since measurement accuracy may vary according to the movement of the subject, a function for minimizing the movement of the subject may be required between measurement of the biometric index.

Accordingly, when a display device displaying predetermined information is used during biometric index measurement, it is possible to induce movement of the subject to be minimized by using the displayed information.

Referring to FIG. 51, different information may be displayed on the smart mirror device 7200 according to an embodiment according to a biometric index measurement time.

More specifically, the measurement of the biometric index for the subject may be started at the first time point 7210.

In this case, the first point of view 7210 may be a point in time at which the measurement target area of the subject is positioned within the measurement area, but is not limited thereto, and the first point of view 7210 is a point in time at which a person capable of being measured is positioned within the angle of view of the image sensor. I can.

In addition, the first time point 7210 may be a time point at which the subject inputs an intention for measurement. For example, the first time point 7210 may be a time point when the subject touches the measurement button, but is not limited thereto.

Also, the first information may be displayed at the second viewpoint 7220.

In this case, the second time point 7220 may be a time point at which face recognition of the subject is completed, but is not limited thereto.

Further, the second time point 7220 may mean a time point after a predetermined time elapses after the measurement of the biometric index for the subject is started, but is not limited thereto.

In addition, the first information may be recognition information for the subject. For example, as illustrated in FIG. 51, the first information displayed at the second view point 7220 may be greeting information for the subject whose face has been recognized, but is not limited thereto.

In addition, the first information may be information on a measurement progress time. For example, when the second point in time 7220 is a point in time two seconds after the start of measurement, information corresponding to two seconds may be displayed.

In addition, the first information may be information on the remaining time for measurement. For example, it takes 6 seconds to measure the biometric index, and when the second time point 7220 is a time point 2 seconds after starting the measurement, information corresponding to the remaining 4 seconds may be displayed.

In addition, second information may be displayed at a third view point 7230.

In this case, the third viewpoint 7230 may mean a viewpoint after a predetermined time elapses after the face recognition of the subject is completed, but is not limited thereto.

Further, the third time point 7230 may mean a time point after a predetermined time elapses after the measurement of the biometric index for the subject is started, but is not limited thereto.

In addition, the second information may be health information on the subject. For example, as shown in FIG. 51, the second information displayed at the third time point 7230 may be the average biometric index per month of the subject, but is not limited thereto.

In addition, the second information may be information on a measurement progress time. For example, when the third time point 7230 is a time point 4 seconds after the start of measurement, information corresponding to 4 seconds may be displayed.

In addition, the second information may be information on the remaining time for measurement. For example, it takes 6 seconds to measure the biometric index, and when the third time point 7230 is a time point 4 seconds after starting the measurement, information corresponding to the remaining two seconds may be displayed.

In addition, third information may be displayed at a fourth viewpoint 7240.

In this case, the fourth time point 7240 may mean a time point at which the biometric index measurement for the subject is completed, but is not limited thereto.

In addition, the third information may be information related to a biometric index for the subject. For example, as illustrated in FIG. 51, the third information displayed at the fourth time point 7240 may be the heart rate, oxygen saturation, and blood pressure of the subject, but is not limited thereto.

In addition, the third information may be information related to biometric information on the subject. For example, although not shown in FIG. 51, the fourth information displayed at the fourth viewpoint 7240 may be biometric information such as a condition index of the subject, but is not limited thereto.

9.1.3 Various embodiments of a smart mirror device displaying an image acquired through an image sensor

Referring to FIG. 52, a smart mirror device 7250 according to an embodiment may include an image sensor 7260 and a mirror display 7270.

In this case, since the above-described operations can be applied to the image sensor 7260 and the mirror display 7270, a redundant description will be omitted.

According to an embodiment, an image may be displayed on the mirror display 7270. For example, as shown in FIG. 52, an image acquired through the image sensor 7260 may be displayed on the mirror display 7270, but is not limited thereto.

In addition, information on a subject to be measured may be displayed on the displayed image. For example, an area displaying a measurement object included in the image may be displayed, but is not limited thereto.

In this way, even though the subject can see his or her own image through the mirror display 7270, the additional display of the image can be a measurement object by clearly displaying the subject to be measured for the biometric index when a plurality of measurement objects exist. It is possible to prevent confusion among multiple personnel.

In addition, the displayed image of the subject and the appearance of the subject reflected through the mirror display 7270 may be in different positions, but are not limited thereto and may overlap with each other.

9.1.4. Various embodiments of a display device measuring a biometric index in real time

Referring to FIG. 53, the display device 7300 according to an embodiment may measure a biometric index in real time, and the biometric index may be displayed on the display device 7300 in real time.

More specifically, the biometric index for the subject acquired in the first time interval at the first time point 7310 may be displayed.

In this case, the first point in time 7310 may be a point in time after the biometric index for the subject is acquired in the first time interval.

In addition, a biometric index for the subject acquired in the second time interval may be displayed at the second time point 7320.

In this case, the second point in time 7320 may be a point in time after the biometric index for the subject is obtained in the second time interval.

*In addition, the second time period may be different from the first time period and may overlap at least partially.

In addition, the second time point 7320 may be a time point later than the first time point 7310.

In addition, a biometric index for the subject acquired in the third time interval may be displayed at the third time point 7330.

In this case, the third time point 7330 may be a time point after obtaining a biometric index for the subject in the third time interval.

In addition, the third time interval may be different from the first and second time intervals, and may overlap at least partially.

In addition, the third time point 7330 may be a time point later than the first and

second time points

7310 and 7320.

In addition, at the fourth time point 7340, a biometric index for the subject acquired in the fourth time interval may be displayed.

In this case, the fourth time point 7340 may be a time point after obtaining a biometric index for the subject in the fourth time interval.

*In addition, the fourth time period may be different from the first, second, and third time periods, and may at least partially overlap.

In addition, the fourth time point 7340 may be a time point later than the first, second, and

third time points

7310, 7320, and 7330.

In addition, as described above, when obtaining a biometric index in real time, the biometric index obtained at a specific time point may be used to store the final biometric index. For example, the biometric index obtained at the time when the biometric index is obtained for the first time may be stored, but the present invention is not limited thereto, and the biometric index obtained at the last time point at which the biometric index is obtained may be stored. It can also store biomarkers.

In addition, as described above, when obtaining a biometric index in real time, a plurality of biometric indexes may be used to store the final biometric index. For example, the average between a plurality of biomarkers acquired during a certain time period may be stored as a final biomarker, but is not limited thereto.

In addition, when the biometric index is obtained in real time as described above, a plurality of biometric indexes can be obtained even if noise is generated according to the motion of the subject during measurement of the biometric index, so that a more accurate biometric index can be obtained.

In addition, as described above, when obtaining a biometric index in real time, a period in which the biometric index is updated to be displayed may be set. For example, the user may set a period in which the biometric index is updated to be displayed, and the biometric index may be updated according to the set period, but is not limited thereto.

9.1.5. Various embodiments of a display device in which predetermined information is displayed during biometric index measurement

Referring to FIG. 54, different information may be displayed on the smart mirror device 7350 according to an embodiment according to a biometric index measurement time.

More specifically, the first time point 7360 may be a time point before or before the measurement of the biometric index for the subject is started.

In this case, the first point of view 7360 may be a point in time at which the measurement target area of the subject is positioned within the measurement area, but is not limited thereto. I can.

Also, first information may be displayed at the first viewpoint 7360.

In this case, the first information may include basic information. For example, as shown in FIG. 54, basic information such as date, time, and weather information may be included, but the present invention is not limited thereto.

In addition, the first information may include information on a guide area. For example, as illustrated in FIG. 54, the guide area may be displayed at the first viewpoint 7360, but is not limited thereto.

Also, second information may be displayed at a second view point 7370.

In this case, the second time point 7370 may be a time point at which the biometric index for the subject is measured, but is not limited thereto.

In addition, the second time point 7370 may mean a time point after a predetermined time elapses after the measurement of the biometric index for the subject is started, but is not limited thereto.

In addition, the second information may include personal information on the subject. For example, as illustrated in FIG. 54, the second information displayed at the second point in time 7370 may include personal information such as the subject's main schedule and medication information, but is not limited thereto.

In addition, the second information may include information related to biometric index measurement. For example, as illustrated in FIG. 54, the second information displayed at the second viewpoint 7370 may include information indicating that the biometric index of the subject is being measured, but is not limited thereto.

Also, third information may be displayed at a third view point 7380.

In this case, the third time point 7380 may be a time point at which the measurement of the biometric index for the subject is completed, but is not limited thereto, and may be a time point at which at least one measurement is completed in the case of real-time measurement.

In addition, the third information may include the measured biometric index. For example, as illustrated in FIG. 54, the third information displayed at the third time point 7380 may include the measured heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

In addition, as shown in FIG. 54, the first information and the second information may be simultaneously displayed at the second time point 7370, and the first, second, and third information may be displayed at the third time point 7380. Information can be displayed simultaneously.

9.1.6. Various embodiments of a smart mirror device including an opening and closing device

Referring to FIG. 55, a smart mirror device 7400 according to an embodiment may include an image sensor 7410, a mirror display 7420, and an opening/closing device 7430.

In this case, since the above-described contents may be applied to the image sensor 7410 and the mirror display 7420, redundant descriptions will be omitted.

In addition, the open state of the opening and closing device 7430 described below may mean a state in which a window for acquiring an image by the image sensor 7410 is secured, and the closed state of the opening and closing device 7430 is the This may mean a state in which a window for the image sensor 7410 to acquire an image is not secured.

In addition, the image sensor 7410 may be used to detect the open state and the closed state. For example, when the illuminance detected by the image sensor 7410 is less than a reference value, the closed state may be detected. In addition, when the illuminance is greater than or equal to a reference value, the open state may be detected, but the present disclosure is not limited thereto, and the open state and the closed state may be detected using the image sensor 7410 in various ways.

In addition, an external sensor may be used to detect the open state and the closed state. For example, when the external sensor detects the degree of opening of the opening/closing device 7430, it is detected as an open state. And, if it is opened below a reference value, it may be detected as a closed state, but the present disclosure is not limited thereto, and the open state and the closed state may be detected using the external sensor in various ways.

According to an embodiment, the opening/closing device 7430 may open/close the image sensor 7410. For example, the opening/closing device 7430 opens or closes the front of the image sensor 7410 stored in the internal storage space to adjust the image acquired by the image sensor 7410 or externally the image sensor 7410 Can be prevented from being observed.

More specifically, at the first time point 7440 when the opening and closing device 7430 is located in a closed state, the image sensor 7410 may not be able to be observed from the outside due to the opening and closing device 7430.

In addition, at the first viewpoint 7440, the image sensor 7410 may not acquire an image. For example, the power to the image sensor 7410 may be cut off through the closing operation of the opening/closing device 7430, but is not limited thereto.

In addition, the image sensor 7410 may be observable from the outside at a second time point 7450 when the opening/closing device 7430 is located in an open state.

Also, at the second viewpoint 7450, the image sensor 7410 may acquire an image. For example, the power to the image sensor 7410 may be supplied through the opening operation of the opening/closing device 7430, but is not limited thereto.

In addition, the surface of the upper opening and closing device 7430 may be formed of a mirror, and may be formed of the same material as the outer surface of the mirror display 7420, but is not limited thereto.

In addition, the opening/closing device 7430 may perform a function of protecting the image sensor 7410 from external dust or the like. For example, when the opening/closing device 7430 is in a closed state, external dust or the like may be prevented from contacting the image sensor 7410, but is not limited thereto.

9.1.7. Various embodiments of a smart mirror device disposed in a shoe rack

A smart mirror device disposed at a shoe rack of a house or an entrance of a building can measure a biometric index for a person entering and exiting. Usually, a shoebox of a house or an entrance of a building is a place where people wear clothes and there is little privacy problem caused by an image sensor, so it can be easy to install an image sensor to measure a biometric index.

Hereinafter, a smart mirror device disposed at an entrance of a building, which may be represented by a shoe rack, will be described, and for convenience of explanation, a smart mirror device disposed at the shoe rack will be described.

Referring to FIG. 56, a smart mirror device 7460 according to an embodiment may include an image sensor 7470 and a mirror display 7480.

In this case, the image sensor 7470 and the mirror display 7480 may be applied to the above-described information, and thus, overlapping descriptions will be omitted.

The image sensor 7470 according to an embodiment may acquire an image of the subject 7490. For example, the image sensor 7470 may acquire an image of the subject 7490 who enters and exits a shoebox where the smart mirror device 7460 is located.

*In addition, the smart mirror device 7460 may acquire a biometric index based on the image frame obtained from the image sensor 7470.

In addition, the mirror display 7480 may reflect and provide the shape of the subject 7490. For example, the subject 7490 may observe a self-image through the mirror display.

In addition, the mirror display 7480 may display the biometric index of the subject 7490. For example, the mirror display 7480 may display a biometric index obtained based on the image frame obtained from the image sensor 7470, but is not limited thereto.

In addition, the mirror display 7480 may display biometric information of the subject 7490. For example, the mirror display 7480 may display biometric information obtained based on the obtained biometric index, but is not limited thereto.

In addition, the mirror display 7480 may be formed on at least a part of the smart mirror device 7460. For example, as illustrated in FIG. 56, the mirror display 7480 may be disposed to occupy a predetermined area in the center of the smart mirror device 7460.

Further, although not shown in FIG. 56, it is obvious that the mirror display 7480 can display basic information, personal information, and the like.

9.1.7.1. Various embodiments of a smart mirror device using a trigger signal

When a smart mirror device for measuring a biometric index is continuously operated, power consumption for acquiring an image and analyzing it may continue. Therefore, a trigger signal may be used to save energy and provide a smarter mirror device.

Referring to FIG. 57, a method 7500 of operating a smart mirror device according to an embodiment includes obtaining an on-trigger (S7510), obtaining a biometric index of a subject (S7520), and off. At least one of acquiring an off-trigger (S7530) and stopping at least one operation of the smart mirror device (S7540) may be included, but is not limited thereto.

More specifically, the method 7500 of operating a smart mirror device according to an embodiment may include obtaining an on-trigger (S7510).

In this case, the on-trigger may be a trigger signal for starting at least one operation of the smart mirror device. For example, the on-trigger may be a trigger signal for performing an operation for an image sensor to acquire an image, but is not limited thereto. It may be a trigger signal for starting at least one operation.

In addition, the on trigger may be provided in various ways.

For example, when the smart mirror device includes a motion detection sensor, the on trigger may be obtained from the motion detection sensor. More specifically, when a predetermined motion is detected by the motion detection sensor, the smart mirror device may obtain a trigger signal corresponding thereto, and such a trigger signal may be the on trigger, but is not limited thereto.

In this case, the motion detection sensor may be disposed inside the smart mirror device, but is not limited thereto and may be disposed outside the smart mirror device.

In addition, for example, when the smart mirror device includes an input device such as a touch panel, the on trigger may be obtained from the input device. More specifically, a predetermined input may be obtained from the input device, and the smart mirror device may obtain a trigger signal corresponding thereto, and the trigger signal may be the ON trigger, but is not limited thereto.

In addition, for example, when the smart mirror device includes an image sensor, the on trigger may be obtained from the image sensor. More specifically, a lighting device located in a shoe rack may detect a predetermined motion to emit light, and the image sensor may detect light emitted from the lighting device, and a smart mirror device may have a trigger signal corresponding thereto. May be obtained, and such a trigger signal may be the ON trigger, but is not limited thereto.

In addition, for example, when an entrance door is included in the angle of view of the image sensor, the on trigger may be obtained based on a change in the entrance door. More specifically, the image sensor may determine the opening of the door, and the smart mirror device may obtain a trigger signal corresponding thereto, and such a trigger signal may be the ON trigger, but is not limited thereto. .

In addition, the method 7500 of operating a smart mirror device according to an exemplary embodiment may include obtaining a biometric index of a subject (S7520).

In this case, the biosignal for the subject may be acquired based on an image frame acquired through an image sensor, but detailed descriptions thereof have been described above, and thus redundant description will be omitted.

In addition, the method 7500 of operating a smart mirror device according to an embodiment may include obtaining an off-trigger (S7530).

In this case, the off-trigger may be a trigger signal for stopping at least one operation of the smart mirror device. For example, the off trigger may be a trigger signal for stopping an operation of an image sensor acquiring an image, but is not limited thereto, and at least one operation of the smart mirror device, such as turning off the power of the smart mirror device, is performed. It may be a trigger signal to stop.

In addition, the off trigger may be provided in various ways.

For example, when the smart mirror device includes a motion detection sensor, the off trigger may be obtained from the motion detection sensor. More specifically, when a predetermined motion is not detected from the motion detection sensor for a predetermined period of time, the smart mirror device may obtain a trigger signal corresponding thereto, and such a trigger signal may be the off trigger, but is limited thereto. It doesn't work.

Also, for example, when the smart mirror device includes an input device such as a touch panel, the off trigger may be obtained from the input device. More specifically, a predetermined input may be obtained from the input device, and the smart mirror device may obtain a trigger signal corresponding thereto, and such a trigger signal may be the off trigger, but is not limited thereto.

Also, for example, when the smart mirror device includes an image sensor, the off trigger may be obtained from the image sensor. More specifically, the lighting device located in the shoe rack may be turned off by sensing that there is no predetermined operation for a certain period of time, the image sensor may detect the intensity of the turned off light, and the smart mirror device may generate a trigger signal corresponding thereto. It can be obtained, and such a trigger signal may be the off trigger, but is not limited thereto.

In addition, for example, the off trigger may be obtained based on obtaining a biometric index of the smart mirror device. More specifically, if there is no area to be measured within the angle of view of the image sensor, biometric index may not be obtained. If the biometric index is not obtained for a certain period of time or longer, the smart mirror device may obtain a trigger signal corresponding thereto. , Such a trigger signal may be the off trigger, but is not limited thereto.

In addition, for example, when included in the door within the angle of view of the image sensor, the off trigger may be obtained based on a change in the door. More specifically, the image sensor may determine the opening of the door, and the smart mirror device may obtain a trigger signal corresponding thereto, and such a trigger signal may be the off trigger, but is not limited thereto. .

58 and 59, the smart mirror device 7550 according to an embodiment may include an image sensor 7560 and a mirror display 7570.

In this case, since the above-described contents may be applied to the image sensor 7560 and the mirror display 7570, a redundant description will be omitted.

Referring to FIG. 58 according to an embodiment, when the subject 7580 is located in a predetermined on-trigger generation area, at least one operation of the smart mirror device may be started.

In this case, the predetermined on-trigger generation area may be a detection range of a motion detection sensor when the smart mirror device includes a motion detection sensor, and may be a detection range of a motion detection sensor of a lighting device installed in a shoe rack. Not limited.

In addition, referring to FIG. 59 according to an embodiment, when the smart mirror device 7550 receives a predetermined input from the subject 7580, at least one operation of the smart mirror device may be started.

In this case, a predetermined input may be obtained through an input device such as a touch panel, but is not limited thereto.

9.1.7.2. Various embodiments of a smart mirror device measuring a biometric index according to priority

When a plurality of people exist in the biometric index measurement area of the smart mirror device, the biometric index may be measured according to priority.

Referring to FIG. 60, in the operation method 7600 of the smart mirror device according to an embodiment, the steps of acquiring an image including a plurality of people (S7610), determining a priority for measuring a biometric index (S7620) , At least one of obtaining a biometric index according to the priority (S7630) and outputting the obtained biometric index (S7640), but is not limited thereto.

More specifically, the operation method 7600 of the smart mirror device according to an embodiment may include the step of acquiring an image including a plurality of people (S7610), and the content of acquiring the above-described image may be applied. As it may be, duplicate descriptions will be omitted.

In addition, the operation method 7600 of the smart mirror device according to an embodiment may include determining a priority for measuring a biometric index (S7620) and obtaining a biometric index according to the priority (S7630). have.

At this time, the ease of measuring the biometric index may be considered in order to determine the priority. For example, a biometric index for a person occupying a larger area in the acquired image among the plurality of people may be measured, but is not limited thereto.

In addition, for example, a biometric index for a person recognized as a person from an image obtained among the plurality of people may be measured, but is not limited thereto.

In addition, for example, a biometric index for a person occupying an area closer to the center of the acquired image among the plurality of people may be measured, but is not limited thereto.

Also, for example, a biometric index for a person located closer to the image sensor in the acquired image among the plurality of people may be measured, but the present invention is not limited thereto.

In addition, information stored in advance may be used to determine the priority. For example, a biometric index for a person previously stored as a subject among the plurality of persons may be measured, but is not limited thereto.

In addition, for example, a biometric index may be obtained according to a preset priority. More specifically, if the priority for a child among family members is set high, the child and the father are simultaneously located within the biometric index measurement area. The biometric index may be measured, but is not limited thereto.

In addition, biometric index measurement order information may be used to determine the priority. For example, when an image of a first subject is acquired first and an image of the second subject is acquired later, a biometric index of the first subject may be measured, but is not limited thereto.

In addition, the operation method 7600 of the smart mirror device according to an embodiment may include outputting the obtained biometric index (S7640).

In this case, the biometric index may be a biometric index for a person determined according to the priority.

In addition, while outputting the biometric index, an indicator for a measured person may be output together. For example, a guide region for a person determined according to the priority and subjected to biometric index measurement may be output, but is not limited thereto.

Referring to FIG. 61, a smart mirror device 7650 according to an embodiment may include an image sensor 7760 and a mirror display 7670.

In this case, since the above-described contents may be applied to the image sensor 7760 and the mirror display 7670, a redundant description will be omitted.

Referring to FIG. 61 according to an embodiment, a first person 7680 and a second person 7690 may be located within a measurement range of the image sensor 7560.

In addition, the first person 7680 may be a target for measuring the biometric index of the smart mirror device 7650.

For example, as shown in FIG. 61, when the first person 7680 is located closer to the image sensor 7760 than the second person 7690, the first person 7680 is the target of the biometric index measurement. It may be, but is not limited thereto.

In addition, for example, if the area occupied by the image for the first person 7680 in the image acquired by the image sensor 7760 is larger than the area occupied by the image for the second person 7690, the first person (7680) may be the target of the biometric index measurement, but is not limited thereto.

In addition, for example, when only the first person 7680 is recognized as a person in the smart mirror device 7650, the first person 7680 may be a target for measuring the biometric index, but is not limited thereto.

In addition, for example, if the area occupied by the image for the first person 7680 in the image acquired by the image sensor 7760 is closer to the center than the area occupied by the image for the second person 7690 1 The person 7780 may be a target for measuring the biometric index, but is not limited thereto.

In addition, for example, if the priority for the first person 7680 is stored higher than the priority for the second person 7690, the first person 7680 may be the target of the biometric index measurement. , Is not limited thereto.

In addition, for example, when the first person 7680 is stored as a subject, the first person 7680 may be the target of the biometric index measurement, but is not limited thereto.

In addition, when the biometric index for the first person 7680 is measured, the biometric index for the first person 7680 may be output. For example, as shown in FIG. 61, the heart rate, oxygen saturation, and blood pressure for the first person 7680 may be output, but the present invention is not limited thereto.

In addition, when the biometric index for the first person 7680 is measured, an indicator for the first person 7680 may be output. For example, as shown in FIG. 61, a guide area for the first person 7680 may be output, but is not limited thereto.

9.1.7.3. Various embodiments of a smart mirror device measuring biomarkers of a plurality of subjects

Referring to FIG. 62, in the operation method 7700 of the smart mirror device according to an embodiment, acquiring an image including a plurality of subjects (S7710), acquiring biomarkers for a plurality of subjects ( S7720) and outputting biomarkers for a plurality of subjects (S7730) may include at least one step, but the present invention is not limited thereto.

More specifically, the operation method 7700 of the smart mirror device according to an embodiment may include a step (S7610) of acquiring an image including a plurality of subjects, and the content of acquiring the above-described image is Since it may be applied, the overlapping description will be omitted.

In addition, the operation method 7700 of the smart mirror device according to an embodiment may include acquiring biometric indexes for a plurality of subjects (S7720), and the biometric index acquisition method described above may be applied to this. Duplicate description will be omitted.

According to an embodiment, an area for each of the plurality of subjects may be set in order to obtain biomarkers for the plurality of subjects. For example, when the first subject and the second subject are located in the biometric index measurement area, the area for the first subject and the area for the second subject may be set.

In addition, a plurality of biomarkers may be obtained for a plurality of subjects to be measured. For example, heart rate, oxygen saturation, and blood pressure for a plurality of subjects may be obtained, but are not limited thereto.

In addition, the types of the plurality of biomarkers obtained for each of the plurality of subjects may be different from each other. For example, a heart rate of a first subject may be obtained, and an oxygen saturation degree and a blood pressure of a second subject may be obtained, but the present invention is not limited thereto.

In addition, detailed regions for each of the plurality of subjects may be set in order to obtain a plurality of biomarkers for the plurality of subjects. For example, an area for a first subject is set, and a first sub-area for measuring heart rate for the first subject, a second sub-area for measuring oxygen saturation, and a third for measuring blood pressure. A detailed area may be set, an area for a second subject is set, and a fourth detailed area for measuring heart rate, a fifth detailed area for measuring oxygen saturation, and blood pressure are measured for the second subject. The sixth detailed area for performing may be set, but is not limited thereto.

In addition, the first to sixth detailed regions may be different from each other, and may at least partially overlap.

In addition, the operating method 7700 of the smart mirror device according to an embodiment may include outputting biometric indexes for a plurality of subjects (S7730).

In this case, the biometric index may be at least one biometric index for each of the plurality of subjects.

In addition, while outputting the biometric index, indicators for a plurality of subjects may be simultaneously output. For example, when the biomarkers of the first and second subjects are obtained, guide areas for the first and second subjects may be output, but the present invention is not limited thereto.

In addition, while outputting the biometric index, an indicator indicating which subject's output biometric index belongs to may be output together. For example, when a biometric index for a first subject is output, an indicator that can identify a first subject may be output together, and when a biometric index for a second subject is output, a second subject is An identifiable indicator may be output together.

Referring to FIG. 63, a smart mirror device 7750 according to an embodiment may include an image sensor 7760 and a mirror display 7700.

In this case, since the above-described contents may be applied to the image sensor 7760 and the mirror display 7700, a redundant description will be omitted.

Referring to FIG. 63 according to an embodiment, a first subject 7780 and a second subject 7790 may be located within a measurement range of the image sensor 7760.

In addition, an area for each of the plurality of subjects may be set in the acquired image frame in order to obtain biomarkers for the first and

second subjects

7780 and 7790.

In addition, biomarkers for the first subject 7780 and the second subject 7790 may be obtained. For example, heart rates for the first and

second subjects

7780 and 7790 may be obtained, but the present disclosure is not limited thereto.

In addition, a plurality of biomarkers may be obtained for each of the first subject 7780 and the second subject 7790. For example, as shown in FIG. 63, the heart rate, oxygen saturation and blood pressure for the first subject 7780 may be obtained, and the heart rate, oxygen saturation and blood pressure for the second subject 7790 may be It may be obtained, but is not limited thereto.

In addition, for example, although not shown in FIG. 63, a heart rate for the first subject 7780 may be obtained, and an oxygen saturation degree and a blood pressure for the second subject 7790 may be obtained. It is not limited to this.

9.1.7.4. Various embodiments of a smart mirror device that outputs different information according to the situation

When the smart mirror device is placed at the entrance of a building, such as a shoe rack, information required for a user may be different between entering and leaving the building.

Accordingly, the smart mirror device may output different information to provide different information to the user according to the situation.

In addition, the smart mirror device may output different biometric indexes to provide different biometric indexes to the user according to the situation.

Referring to FIG. 64, a method 7800 of operating a smart mirror device according to an embodiment includes determining an information provision situation (S7810), outputting first information according to the first situation (S7820), and At least one step of outputting second information according to the second situation (S7830) may be included, but is not limited thereto.

More specifically, the method 7800 of operating a smart mirror device according to an embodiment may include determining an information provision situation (S7810).

In this case, the information provision situation may mean a situation of the user of the smart mirror device. For example, the information provision situation may include, but is not limited to, a situation in which the user of the smart mirror device enters a building or exits a building.

In addition, the information provision situation may simply mean a situation determined to provide information. For example, the information provision situation may include a first situation or a second situation, but is not limited thereto.

In addition, the information provision situation may mean a situation related to whether or not to measure the biometric index. For example, the information provision situation may include a situation in which it is not necessary to measure a biometric index or a situation where a biometric index measurement is required, but is not limited thereto.

In addition, the information provision situation may mean a time-related situation. For example, the information provision situation may include a morning situation or an afternoon situation, but is not limited thereto.

In addition, the information provision situation may mean a situation according to an external environment. For example, the information provision situation may include a situation according to an external environment such as an external infectious disease situation, but is not limited thereto.

In addition, the smart mirror device may include an additional sensor to determine the information provision situation. For example, a motion detection sensor may be additionally included to determine the user's entry/exit status, and when the user's motion is first detected by the first motion detection sensor, the first situation is determined and the second motion detection sensor If the user's motion is first detected, it may be determined as the second situation, but is not limited thereto.

In addition, an image sensor included in the smart mirror device may be used to determine the information provision situation. For example, an image frame obtained through the image sensor may be used to determine the access situation of the user, and more specifically, an area corresponding to the user in a plurality of image frames acquired in a time series If the first aspect approaches the central part, the first situation may be determined, and when the second aspect approaches the central part, the second situation may be determined, but is not limited thereto.

In addition, an image sensor included in the smart mirror device may be used to determine the information provision situation. For example, an image frame obtained through the image sensor may be used to determine whether it is necessary to measure a biometric index and provide information. If the moving speed of the applicable area exceeds the reference value, the first situation may be determined, and if the moving speed is less than the reference value, the second situation may be determined, but is not limited thereto.

In addition, time information may be used to determine the information provision situation. For example, if it is before the reference time, it may be determined as a first situation, and if it is after the reference time, it may be determined as a second situation, but the present invention is not limited thereto.

In addition, recognition information may be used to determine the information provision situation. For example, face recognition information about a person may be used to determine whether a situation where biometric index measurement and information provision is required, and more specifically, if a user registered as a result of face recognition, it may be determined as the first situation, and registration If the user is not the same, it may be determined as the second situation, but is not limited thereto.

In addition, information on the direction of movement of the subject may be used to determine the information provision situation. For example, the movement direction information of the subject detected by sensing or using an external sensor may be used to determine the information provision situation. More specifically, when the subject moves in the first direction, the first The first situation may be determined, and when the subject moves in the second direction, the second situation may be determined, but the present disclosure is not limited thereto.

In addition, a method 7800 of operating a smart mirror device according to an embodiment includes outputting first information according to a first situation (S7820) and outputting second information according to a second situation (S7830). It can be, but is not limited thereto.

In this case, when the first situation includes a situation where the first situation goes out of the building and the second situation includes a situation where the second situation includes a situation where the inside of the building is entered, the first information and the second information may be different from each other.

For example, the first information may include external weather information, today's schedule information, product information according to external weather, and the like, but is not limited thereto, and may be information according to the first situation.

In addition, the second information may include internal temperature information, internal humidity information, security information, internal air quality information, and the like, but is not limited thereto, and may be information according to the second situation.

In addition, biometric index information may be included in at least one of the first information and the second information, but is not limited thereto.

In addition, a biometric index included in the first information and a biometric index included in the second information may be different from each other. For example, a heart rate may be output in a situation in which a biometric index of a first subject is provided, and a blood pressure may be output in a situation in which a biometric index of a second subject is provided, but the present disclosure is not limited thereto.

Referring to FIGS. 65 and 66, a smart mirror device 7850 according to an embodiment may include an image sensor 7860 and a mirror display 7870.

In this case, since the above-described contents may be applied to the image sensor 7860 and the mirror display 7870, a redundant description will be omitted.

According to an embodiment, FIG. 65 is a diagram illustrating an operation of the smart mirror device 7850 in a first situation, and FIG. 66 may be a diagram illustrating an operation of the smart mirror device 7850 in a second situation.

In addition, information output in the first situation and the second situation may be different from each other, and since the above-described information may be applied, a redundant description will be omitted.

In addition, the external sensor 7861 may be used to determine the first situation and the second situation, but is not limited thereto, and the above-described examples for determining the first situation and the second situation may be applied.

Further, although not shown in FIGS. 65 and 66, information output in the first situation and the second situation may overlap at least partially with each other. For example, the biometric index may be included in the information output in the first situation and the second situation, but is not limited thereto.

9.2. Various embodiments of a biometric index measuring device disposed on a vehicle

The above-described biometric index acquisition method and drowsiness detection method can be used in various applications. For example, the biometric index acquisition method or drowsiness detection method may be used in a biometric index measurement device disposed in a vehicle.

In addition, the biometric index measurement apparatus may perform an operation corresponding to the biometric index or biometric information obtained according to the biometric index acquisition method.

In addition, the biometric index measuring apparatus may perform an operation corresponding to the drowsiness information obtained according to the drowsiness detection method.

Referring to FIG. 67, the biometric index measuring apparatus 8010 according to an embodiment may be disposed in the vehicle 8000 and measure the biometric index of the occupant 8020 in the vehicle 8000.

In this case, the biometric index measuring device 8010 may be disposed at various locations of the vehicle 8000.

For example, the biometric index measuring device 8010 may be disposed in an overhead console, a sun visor, a rearview mirror, a dashboard, an instrument panel, a steering wheel, a Santa fascia, a console box, etc. of the vehicle 8000, but this Not limited.

In addition, the occupant 8020 may be a driver, but is not limited thereto, and may be a person who is boarding a vehicle such as a passenger.

In addition, when the occupant 8020 is a plurality of persons, the biometric index measuring device 8010 may measure the biometric index for the plurality of persons.

In addition, the biometric index measuring apparatus 8010 may acquire a biometric index based on an image frame obtained from the image sensor including an image sensor, but the present invention is not limited thereto, and a contact sensor or the like may be used.

In addition, when the biometric index measuring device 8010 includes an image sensor, the image sensor may be provided as a visible camera for obtaining a visible light image, an IR camera for obtaining an infrared image, etc. However, the present invention is not limited thereto, and a hybrid type camera for acquiring a visible light image and an infrared image may be provided.

In addition, the biometric index measuring device 8010 may operate by changing a mode according to the amount of external light. For example, the first mode may be operated in the daytime when there is a lot of external light, and the second mode may be operated in the night when there is little external light, but the present invention is not limited thereto.

In addition, for example, the biometric index measurement device 8010 may operate in a first mode for obtaining a biometric index based on an RGB image when external light is greater than or equal to a reference value, and an IR image when external light is less than or equal to the reference value. The second mode may be used to obtain a biometric index based on, but is not limited thereto.

Referring to FIG. 68, the apparatus for measuring a biometric index according to an embodiment may be disposed at various locations of a vehicle.

For example, as shown in FIG. 68, the first biometric index measuring device 8011 may be disposed on the dashboard on the driver's seat side, and the second biometric index measuring device 8012 may be disposed on the dashboard on the assistant seat side. The third biometric index measuring device 8013 may be disposed in the room mirror, but is not limited thereto.

In addition, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may be disposed inside the vehicle.

In addition, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may obtain the driver's biometric index.

For example, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may measure a biometric index such as heart rate, blood pressure, oxygen saturation, and body temperature of the driver. It is not limited to this.

In addition, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may perform an operation according to the acquired biometric index of the driver.

For example, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 operates to guide a nearby hospital when the obtained driver's biometric index is abnormal. May be performed, but is not limited thereto.

In addition, for example, at least one biometric index measurement device among the first to third biometric

index measurement devices

8011, 8012, and 8013 may use another mobile device when the obtained driver's biometric index is abnormal. Information can be transmitted, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 is hardware-based, such as sound and vibration, when the obtained driver's biometric index is abnormal. The alarm can be operated, but is not limited thereto.

In addition, for example, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may switch the vehicle to the autonomous driving mode when the obtained driver's biometric index is abnormal. An operation for changing may be performed, but is not limited thereto.

In addition, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may acquire biometric information of the driver.

For example, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may acquire the driver's biometric information, such as the driver's sleepiness information and condition information, It is not limited to this.

In addition, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may perform an operation according to the acquired biometric information of the driver.

For example, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 operates a hardware alarm such as sound vibration when the acquired driver's sleepiness information is abnormal. However, it is not limited thereto.

In addition, for example, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may guide a nearby drowsy shelter when the acquired driver's sleepiness information is abnormal. An operation such as may be performed, but is not limited thereto.

In addition, for example, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may use another mobile device when the acquired driver's sleepiness information is abnormal. Information can be transmitted, but is not limited thereto.

In addition, for example, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may set the vehicle to the autonomous driving mode when the acquired driver's sleepiness information is abnormal. An operation for changing may be performed, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 performs an operation such as selecting appropriate music according to the acquired condition information of the driver. It can be performed, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 performs an operation such as guiding appropriate contents according to the acquired condition information of the driver. It can be performed, but is not limited thereto.

In addition, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may obtain a passenger's biometric index.

For example, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may measure biometric indexes such as heart rate, blood pressure, oxygen saturation, and body temperature of the passenger, It is not limited to this.

In addition, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may perform an operation according to the obtained biometric index of the passenger.

For example, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may provide an alarm to the driver when there is an abnormality in the acquired biometric index of the passenger. May be performed, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 guides a nearby hospital when the obtained biometric index of the passenger is abnormal. The operation of can be performed, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 may use another mobile device when the obtained biometric index of the passenger is abnormal. Information can be transmitted, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 switches the vehicle to the autonomous driving mode when the obtained biometric index of the passenger is abnormal. An operation for changing may be performed, but is not limited thereto.

In addition, at least one of the first to third biometric

index measuring devices

8011, 8012, and 8013 may obtain the passenger's biometric information.

For example, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may obtain the passenger's biometric information, such as the passenger's sleepiness information and condition information, It is not limited to this.

In addition, at least one of the first to third biometric

index measurement devices

8011, 8012, and 8013 may perform an operation according to the acquired biometric information of the passenger.

For example, at least one biometric index measurement device among the first to third biometric

index measurement devices

8011, 8012, and 8013 informs the driver of the obtained biometric information to the driver when there is an abnormality. An operation such as may be performed, but is not limited thereto.

index measurement devices

8011, 8012, and 8013 may use another mobile device when the obtained biometric information of the passenger is abnormal. Information can be transmitted, but is not limited thereto.

Referring to FIG. 69, the apparatus for measuring a biometric index according to an embodiment may be disposed at various locations of a vehicle.

For example, as shown in FIG. 69, the first biometric index measurement device 8014 may be disposed on one side of the table, and the second biometric index measurement device 8015 may be disposed on the other side of the table, and 3 The biometric index measuring device 8016 may be disposed on the ceiling of an autonomous vehicle, but is not limited thereto.

In addition, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 may be disposed inside the vehicle.

In addition, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 may obtain a passenger's biometric index.

For example, at least one of the first to third biometric

index measuring devices

8014, 8015, and 8016 may measure biometric indexes such as heart rate, blood pressure, oxygen saturation, and body temperature of the passenger, It is not limited to this.

In addition, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 may perform an operation according to the obtained biometric index of the passenger.

For example, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 operates to guide a nearby hospital when the obtained biometric index of the passenger is abnormal. May be performed, but is not limited thereto.

index measurement devices

8014, 8015, and 8016 drives to a nearby hospital when the obtained biometric index of the passenger is abnormal. The operation of can be performed, but is not limited thereto.

index measurement devices

8014, 8015, and 8016 may use another mobile device when the obtained biometric index of the passenger is abnormal. Information can be transmitted, but is not limited thereto.

index measurement devices

8014, 8015, and 8016 is hardware-based, such as sound and vibration, when the obtained biometric index of the passenger is abnormal. The alarm can be operated, but is not limited thereto.

In addition, for example, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 may display information about the obtained biometric index of the passenger when there is an abnormality. However, it is not limited thereto.

In addition, at least one of the first to third biometric

index measuring devices

8014, 8015, and 8016 may perform an operation according to a result of recognizing the passenger.

For example, at least one of the first to third biometric

index measuring devices

8014, 8015, and 8016 may recognize the face of the passenger. Information may be displayed, but is not limited thereto.

index measurement devices

8014, 8015, and 8016 is applied to the recognized passenger only when the passenger's face is recognized and the biometric index is obtained. Information about may be displayed, but is not limited thereto.

In addition, for example, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 may obtain biometric information of the passenger.

For example, at least one of the first to third biometric

index measuring devices

8014, 8015, and 8016 may obtain the passenger's biometric information, such as the passenger's sleepiness information and condition information. Not limited.

In addition, at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 may perform an operation according to the acquired biometric information of the passenger.

For example, when at least one of the first to third biometric

index measurement devices

8014, 8015, and 8016 has an abnormality in the acquired biometric information of the passenger, a hardware alarm such as sound and vibration is operated. However, it is not limited thereto.

index measurement devices

8014, 8015, and 8016 may use another mobile device when the obtained biometric information of the passenger is abnormal. Information can be transmitted, but is not limited thereto.

In addition, although not shown in FIGS. 68 and 69, the biometric index measurement device may be disposed in an ambulance or the like, and may be used to measure a patient's biometric index and obtain biometric information.

9.2.1. Various embodiments of a biometric index measuring device that performs an operation according to sleepiness information

Referring to FIG. 70, a method 8100 of operating a biometric index measuring apparatus according to an embodiment may include obtaining a biometric index (S8110) and determining a drowsiness stage (S8120). You can output alarms according to steps. For example, as illustrated in FIG. 70, a first alarm S8130, a second alarm S8140, and a third alarm S8150 may be output, but the present invention is not limited thereto.

In this case, since the above-described method of obtaining a biometric index may be applied to the step of obtaining the biometric index (S8110), a redundant description will be omitted.

In addition, since the above-described method for detecting drowsiness may be applied to the step of determining the drowsiness step (S8120), a redundant description will be omitted.

In addition, the first alarm S8130 may be an alarm output in the weakest drowsiness stage.

For example, the first alarm S8130 may include a visual alarm, but is not limited thereto.

In addition, the second alarm S8140 may be an alarm output in an intermediate step.

For example, the second alarm S8140 may include an audible alarm having a low intensity or a long repetition period, but is not limited thereto.

In addition, the third alarm S8150 may be an alarm output at the highest level of drowsiness.

For example, the third alarm S8150 may include an audible alarm having a strong intensity or a short repetition period, but is not limited thereto.

However, the above-described first to third alarms S8130, S8140, and S8150 are not limited to the above-described examples, and it is obvious that all examples of outputting different alarms according to the drowsiness stage can be applied.

In addition, the method 8100 of operating the apparatus for measuring a biometric index according to an exemplary embodiment may include transmitting drowsiness information (S8160).

Also, the step of transmitting the drowsiness information (S8160) may be performed only in at least some of the drowsiness steps. For example, in the case of the first drowsiness step, the step of transmitting the drowsiness information (S8160) may not be performed, and in the case of the second drowsiness step and the third drowsiness step, the step of transmitting the drowsiness information (S8160) is performed. It may be, but is not limited thereto.

In addition, transmitting the drowsiness information (S8160) may include transmitting the drowsiness information to the administrator. For example, information on the drowsiness stage of a driver of a vehicle may be transmitted to a manager who dispatches a vehicle to reflect the vehicle dispatch.

In addition, transmitting the drowsiness information (S8160) may include transmitting the drowsiness information to the driver's mobile terminal. For example, sleep information and information on a corresponding driving section may be transmitted to the driver's mobile terminal so that the driver pays attention to the next driving.

In addition, transmitting the drowsiness information (S8160) may include transmitting the drowsiness information to the road manager. For example, information on a driver's drowsiness stage and information on a driving section may be transmitted to be used for making a drowsiness map and the like.

9.2.2. Various embodiments of driving scheduling method using biometric index and biometric information

Referring to FIG. 71, the method 8200 of operating a biometric index measuring apparatus according to an embodiment includes obtaining a biometric index (S8210), obtaining biometric information (S8220), and calculating a driving scheduling index ( S8230) may include at least a portion, but is not limited thereto.

In this case, since the above-described method of obtaining a biometric index may be applied to the step of obtaining the biometric index (S8210), a redundant description will be omitted.

In addition, since the above-described method of obtaining biometric information may be applied to the step of acquiring biometric information (S8220), redundant descriptions will be omitted.

In addition, the driving scheduling index may be an index for assisting driving scheduling.

In this case, the driving scheduling index may be obtained based on the biometric index and biometric information. For example, the driving scheduling index may be obtained based on the driver's heart rate information and drowsiness information, but is not limited thereto.

In addition, the driving scheduling index may be obtained based on driving information of a vehicle. For example, the driving scheduling index may be obtained based on a driving distance, driving time, and number of driving of the vehicle, but is not limited thereto.

In addition, the driving scheduling index may be obtained based on a driver's biometric index, biometric information, and vehicle driving information. For example, the driving scheduling index may be obtained based on the driver's sleepiness information, the driving distance of the vehicle, and the number of driving, but is not limited thereto.

Referring to FIG. 72, the driving scheduling assistance device may obtain information from a first biometric index measuring device disposed in a first vehicle and a second biometric index measuring device disposed in a second vehicle.

More specifically, the first biometric index measurement device may obtain a biometric index and biometric information for a first driver who is in the first vehicle. For example, the first biometric index measurement device may acquire heart rate and drowsiness information for the first driver, but is not limited thereto.

In addition, the second biometric index measurement device may obtain a biometric index and biometric information for a second driver who is in the second vehicle. For example, the second biometric index measurement device may acquire heart rate and drowsiness information for the second driver, but is not limited thereto.

In addition, as shown in FIG. 72, the driving scheduling assistance device may obtain a biometric index and biometric information for the first driver obtained from the first biometric index measurement device, and the second biometric index measurement device The biometric index and biometric information for the second driver obtained from can be obtained, but the present invention is not limited thereto.

In addition, as illustrated in FIG. 72, the driving scheduling assistance device may obtain first driving information for a first driving of the first vehicle and second driving information for a second driving of the second vehicle.

In this case, the first and second driving information may include a driving distance, a driving number, a driving time, and the like, but are not limited thereto.

In addition, the driving scheduling assistance device includes a biometric index and biometric information for the first driver, a biometric index and biometric information for the second driver, first driving information for the first vehicle, and a second vehicle for the second vehicle. 2 A driving scheduling index can be calculated based on the driving information.

For example, a first driving scheduling index may be calculated based on first drowsiness information for the first driver and first driving information for the first vehicle, and second drowsiness information for the second driver and The second driving scheduling index may be calculated based on the second driving information for the second vehicle, but is not limited thereto.

In addition, for example, when the first driving distance included in the first driving information and the second driving distance included in the second driving information are the same, among the first driver and the second driver, sleepiness according to the sleepiness information A driving scheduling index may be calculated so that a driver with a lower index may perform the third driving.

In addition, for example, the first driving scheduling index may be calculated by adding different weights to the first drowsiness information and the first driving information for calculating the first driving scheduling index.

More specifically, the first driving scheduling index may be calculated by adding more weight to the first driving information, but is not limited thereto.

Also, for example, different weights may be added to the second drowsiness information and the second driving information for calculating the second driving scheduling index to calculate the second driving scheduling index.

More specifically, the second driving scheduling index may be calculated by adding more weight to the second driving information, but is not limited thereto.

In addition, the driving scheduling auxiliary device may perform driving scheduling for a third driving by using the calculated driving scheduling index, but is not limited thereto.

In addition, the driving scheduling auxiliary device may display the calculated driving scheduling index, but is not limited thereto.

9.2.3. Various embodiments of a biometric index measuring device used for unlocking a vehicle

Referring to FIG. 73, a method 8300 of operating a biometric index measuring apparatus according to an embodiment includes recognizing a face of a subject (S8310), acquiring a biometric index of the subject (S8320), and operation of a vehicle. At least a part of the step of releasing at least some of the locks (S8330) may be included, but is not limited thereto.

In this case, since the step of recognizing the subject's face (S8310) can be applied not only to the above-described content but also to a conventional face recognition method, a duplicate description will be omitted.

In addition, in the step of acquiring the biometric index of the subject (S8320), the above-described biometric index acquisition method may be applied, and thus redundant description will be omitted.

In addition, the step of unlocking at least some of the vehicle operations (S8330) may be performed when the subject's face is recognized and a biometric index is obtained.

For example, when the face of the subject is recognized but the biometric index is not obtained, the step of releasing at least some of the vehicle operations (S8330) may not be performed, but is not limited thereto.

In addition, in this case, it is possible to prevent unlocking by stealing photos, etc., and to further strengthen the security of the vehicle.

In addition, the step of unlocking at least a portion of the vehicle operation (S8330) may be performed when the recognized face of the subject matches an ID stored in the server and a biometric index is obtained.

For example, when the face of the subject is recognized and the biometric index is obtained, but the recognized face of the subject does not match the ID stored in the server, unlocking at least some of the operation of the vehicle (S8330) May not be performed, but is not limited thereto.

In addition, the step of unlocking at least a portion of the vehicle operation (S8330) may be performed when the face of the subject is recognized and the biometric index corresponds to the ID stored in the server.

For example, if the subject's face is recognized and the biometric index is obtained, but the obtained biometric index of the subject does not match the ID stored in the server, unlocking at least some of the operation of the vehicle (S8330) ) May not be performed, but is not limited thereto.

In addition, at least some of the operations of the vehicle may be locked for starting the vehicle, but is not limited thereto.

9.2.4. Various embodiments of a method of operating a driving index calculating device using a biometric index measuring device

Referring to FIG. 74, the operation method 8350 of the driving index calculation apparatus according to an embodiment includes obtaining driving information (S8360), obtaining drowsiness information for a subject (S8370), and At least a part of the step of calculating the driving index (S8380) may be included, but is not limited thereto.

In this case, the driving index may be a comprehensive evaluation index for the driver's operation, and may be a numerical value calculated by comprehensively considering various information, but is not limited thereto.

In this case, the driving information may include, but is not limited to, driving distance information, driving number information, driving time information, driving section information, road information, and the like for a vehicle in operation.

In addition, since the above-described method for detecting drowsiness may be applied in the step of acquiring the drowsiness information for the subject (S8370), a redundant description will be omitted.

In addition, calculating a driving index for the subject (S8380) may include calculating a driving index based on the driving information and the drowsiness information.

For example, the driving index may be calculated by comprehensively considering the number of times drowsiness occurs during the driving time, the degree of drowsiness, and a response to this, but is not limited thereto.

In addition, a compensation for the driver may be awarded based on the calculated driving index. For example, a compensation for a driver may be awarded based on driving time information of a driver who has been driving for a long time and a driving index calculated based on information about sleepiness in which sleepiness does not occur, but is not limited thereto.

In addition, in addition to the above-described examples, a driving index for the subject may be calculated according to various methods based on the driving time and drowsiness information.

9.3. Various embodiments of a biometric index measuring device disposed on an infant monitoring device

Monitoring infants and toddlers using image sensors such as cameras is a trend that is becoming essential for the health and safety of infants and toddlers.

In addition, monitoring the health and safety of infants and toddlers by measuring biomarkers of infants and toddlers is also becoming an essential trend.

However, attaching a separate sensor or disposing a separate contact sensor in order to monitor the biometric index of infants and toddlers may cause inconvenience to infants and young children, thereby limiting the role of a comfortable and comfortable shelter.

Therefore, monitoring infants and toddlers using image sensors such as cameras and simultaneously monitoring the biometric indexes of infants and toddlers using image sensors such as cameras will protect infants' health and safety and provide a comfortable and comfortable shelter for infants and toddlers. I can.

Referring to FIG. 75, the infant monitoring apparatus 8400 according to an embodiment may obtain the biometric index of the infant 8410 in a non-contact manner.

More specifically, the infant monitoring apparatus 8400 may include an image sensor, and may obtain a biometric index based on an image frame obtained from the image sensor, but is not limited thereto.

In addition, since the above-described contents may be applied to the infant/child monitoring device 8400 obtaining a biometric index based on an image frame, a redundant description will be omitted.

In addition, the infant monitoring device 8400 may store or transmit an image acquired through the image sensor.

For example, the infant/child monitoring device 8400 may acquire an image including at least a part of the infant 8410 to be measured, and may store the acquired image or transmit it to the user's mobile device.

In addition, the infant monitoring device 8400 may detect the motion of the infant 8410. For example, the infant monitoring device 8400 may detect the overturning of the infant 8410, but is not limited thereto.

In addition, the infant monitoring device 8400 may detect a sound generated from the infant 8410. For example, the infant monitoring device 8400 may detect the crying sound of the infant 8410, but is not limited thereto.

In addition, the infant monitoring device 8400 may generate sound. For example, when there is an input for sound from an input device, the infant monitoring device 8400 may output a sound corresponding to the input, and more specifically, when the voice of the parents of the infant 8410 is input, this It can be output, but is not limited thereto.

In addition, the infant monitoring device 8400 may output lighting. For example, when there is an input for lighting from an input device, the infant monitoring device 8400 may output lighting corresponding to the input, but is not limited thereto.

In addition, the infant and toddler monitoring device 8400 may control the operation of the cradle in which the infant 8410 is present. For example, when there is an input for a cradle from an input device, the infant monitoring device 8400 may perform an operation of the cradle corresponding to the input, but is not limited thereto.

In addition, the infant monitoring device 8400 may output an alarm. For example, when an event occurs in the infant 8410, an audible alarm may be output, but the present invention is not limited thereto, and a visual and tactile alarm may be output, and information may be transmitted to the user's mobile terminal.

In addition, the infant and toddler monitoring device 8400 may perform various operations for monitoring the infant and toddler 8410 in addition to the examples described above.

9.3.1. Various embodiments of an infant and toddler monitoring device that performs an action according to the occurrence of an event

76 is a diagram for describing occurrence of an event for infants and toddlers.

Referring to FIG. 76, it can be seen that various events may occur for infants and toddlers.

Referring to (a) of FIG. 76, it can be seen that an event in which an infant is turned over may occur.

In addition, if the infant is turned over, the passage for breathing may be blocked, which may pose a great risk to the infant.

Also, referring to (b) of FIG. 76, it can be seen that an event in which infants and toddlers are moved out of the monitoring area may occur.

In this case, the infant may have been moved by a nurse at a postpartum care center, but the infant may have been moved by an unintended visitor.

Also, although not shown in FIG. 76, various events may occur that may threaten the health and safety of infants and toddlers.

Therefore, there may be a need for an infant monitoring device capable of monitoring and tracking such events.

Referring to FIG. 77, a method 8500 of operating an infant monitoring device according to an embodiment includes obtaining a biometric index (S8510), determining an event occurrence (S8520), and performing an operation corresponding to the event. It may include at least a portion of (S8530), but is not limited thereto.

In this case, in the step of obtaining the biometric index (S8510), the above-described biometric index obtaining method may be applied, and thus redundant descriptions will be omitted.

According to an embodiment, determining the occurrence of the event (S8520) may be determined based on the obtained biometric index.

For example, when the biometric index is not obtained through the obtaining of the biometric index (S8510), it may be determined that an event has occurred. For a more specific example, when an infant or toddler subject to obtaining a biometric index is turned over, the biometric index may not be obtained, and in this case, it may be determined that an event has occurred, but is not limited thereto.

In addition, for example, when there is an abnormality in the biometric index obtained through the step of obtaining the biometric index (S8510), it may be determined that an event has occurred. For a more specific example, when there is an abnormality in the heart rate of an infant to be acquired, the biometric index may be determined as an event, but is not limited thereto.

In addition, according to an embodiment, the step of determining the occurrence of the event (S8520) may be determined based on a plurality of biomarkers.

For example, if at least one biometric index is not obtained through the step of acquiring the biometric index (S8510), it may be determined as the occurrence of the first event, and if all biometric indexes are not obtained, it is determined that the second event occurs. It can be, but is not limited thereto.

In addition, for example, when at least one biometric index obtained through the step of acquiring the biometric index (S8510) has an abnormality, it may be determined as a first event, and if all biometric indexes are abnormal, the second It may be determined that the event has occurred, but is not limited thereto.

In addition, according to an embodiment, the step of determining the occurrence of the event (S8520) may be determined based on a change in the biometric index.

For example, when a sudden change in the biometric index obtained through the step of acquiring the biometric index (S8510) is detected, it may be determined as an event. When changing from a sleeping state to a wake-up state, the heart rate of the infant may increase, and this case may be determined as an event occurrence, but is not limited thereto.

In addition, according to an embodiment, performing the operation corresponding to the event (S8530) may include performing an operation corresponding to the determined event information.

In this case, an operation corresponding to the event may include a hardware alarm such as a visual, audible, and tactile alarm. For example, the infant monitoring device may output an audible sound alarm when it is determined that an event occurs, but is not limited thereto.

In addition, an operation corresponding to the event may include an operation for recording an image. For example, the infant monitoring apparatus may operate to record an image when it is determined that an event occurs, but is not limited thereto.

In addition, the operation corresponding to the event may include an operation of transmitting information on the event. For example, the infant monitoring device may transmit information on the event to the user's mobile terminal when it is determined that the event occurs, but is not limited thereto.

In addition, the operation corresponding to the event may include an operation of storing information on when the event occurs. For example, when it is determined that an event occurs, the infant monitoring device may store information on the event occurrence time, but is not limited thereto.

In addition, the operation corresponding to the event may include various operations corresponding to the event that occurred in addition to the above-described examples.

Referring to FIG. 78, a method of operating an infant monitoring apparatus 8550 according to an embodiment includes obtaining a biometric index (S8560), determining an event occurrence (S8570), and a first operation corresponding to a first event. At least a part of performing (S8580) and performing a second operation corresponding to the second event (S8590) may be included, but is not limited thereto.

In this case, in the step of obtaining the biometric index (S8560), the above-described method of obtaining the biometric index may be applied, and thus redundant description will be omitted.

According to an embodiment, determining the occurrence of the event (S8570) may be determined based on the obtained biometric index.

For example, when the biometric index is not obtained through the obtaining of the biometric index (S8560), it may be determined that an event has occurred. For a more specific example, when an infant or toddler subject to obtaining a biometric index is turned over, the biometric index may not be obtained, and in this case, it may be determined that an event has occurred, but is not limited thereto.

In addition, for example, when there is an abnormality in the biometric index obtained through the step of obtaining the biometric index (S8560), it may be determined that an event has occurred. For a more specific example, when there is an abnormality in the heart rate of an infant to be acquired, the biometric index may be determined as an event, but is not limited thereto.

In addition, according to an embodiment, the step of determining the occurrence of the event (S8570) may be determined based on a plurality of biomarkers.

For example, if at least one biometric index is not obtained through the step of acquiring the biometric index (S8560), it may be determined as the occurrence of the first event, and if all biometric indexes are not obtained, it is determined that the second event occurs. It can be, but is not limited thereto.

In addition, for example, when at least one biometric index obtained through the step of obtaining the biometric index (S8560) has an abnormality, it may be determined as a first event, and if all biometric indexes are abnormal, the second It may be determined that the event has occurred, but is not limited thereto.

In addition, according to an embodiment, the step of determining the occurrence of the event (S8570) may be determined based on a change in the biometric index.

For example, when a sudden change in the biometric index obtained through the step of obtaining the biometric index (S8560) is detected, it may be determined as an event. When changing from a sleeping state to a wake-up state, the heart rate of the infant may increase, and this case may be determined as an event occurrence, but is not limited thereto.

In addition, according to an embodiment, performing a first operation corresponding to a first event (S8580) and performing a second operation corresponding to a second event (S8590) include operations corresponding to the determined event information. It may include the step of performing.

In this case, the first operation and the second operation may include hardware alarms such as visual, auditory, and tactile alarms. For example, the infant monitoring device may output an audible sound alarm when it is determined that an event occurs, but is not limited thereto.

In addition, the first operation and the second operation may include an operation for recording an image. For example, the infant monitoring apparatus may operate to record an image when it is determined that an event occurs, but is not limited thereto.

In addition, the first operation and the second operation may include an operation of transmitting information on an event. For example, the infant monitoring device may transmit information on the event to the user's mobile terminal when it is determined that the event occurs, but is not limited thereto.

In addition, the first operation and the second operation may include an operation of storing information on an event occurrence time point. For example, when it is determined that an event occurs, the infant monitoring device may store information on the event occurrence time, but is not limited thereto.

In addition, the first operation and the second operation may be different from each other, but are not limited thereto, and may be identical to each other, and at least some of the same operations may be performed with each other.

In addition, the first operation and the second operation may include various operations corresponding to the first event and the second event that have occurred in addition to the above-described examples.

9.3.2. Various embodiments of the infant monitoring system

Referring to FIG. 79, a mobile application 8600 according to an embodiment includes an image display area 8610, a biometric index display area 8620, an input button display area 8630, an image time display area 8640, and an event time. It may include at least a part of the recorded image display area 8650, but is not limited thereto.

In this case, the image displayed on the image display area 8610 may be an image or an image acquired from an infant monitoring device, but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 includes at least one biometric index. For example, as shown in FIG. 79, the biometric index may include heart rate, oxygen saturation, blood pressure, etc., but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 may be a biometric index obtained from an infant monitoring device, but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 may be a biometric index obtained from at least one sensor, but is not limited thereto.

In addition, the biometric index displayed on the biometric index display area 8620 may be a biometric index obtained from different sensors. For example, the heart rate displayed on the biometric index display area 8620 is a biometric index obtained from an infant monitoring device, and the blood pressure may be a biometric index obtained from an external blood pressure sensor, but is not limited thereto.

Also, the input button displayed on the input button display area 8630 may include at least one input button.

In addition, the input button may include at least some of various input buttons such as a video recording button, a lighting button, a speaking button, a cradle shake button, and an alarm button, but is not limited thereto.

In addition, the image time displayed on the image time display area 8640 may be time information on the recorded image.

In addition, an event occurrence time point may be displayed at the video time displayed on the video time display area 8640. For example, as illustrated in FIG. 79, a first event occurrence time and a second event occurrence time may be displayed at the video time, but the present invention is not limited thereto.

Also, a thumbnail of the recorded image may be displayed on the image display area 8650 recorded at the event time.

Also, information on the recorded image may be displayed on the image display area 8650 recorded at the event time. For example, as illustrated in FIG. 79, time information of a recorded image may be displayed, but the present disclosure is not limited thereto.

However, the above-described examples and drawings are for illustrative purposes only, and are not limited thereto, and an application for implementing the infant monitoring system may be provided as various types of applications.

9.4. Various embodiments of a biometric index measuring device disposed in a reading room

Measuring biometric indexes and biometric information in spaces for personal work and study, such as a reading room, can enable efficient work or study by monitoring the individual's concentration while continuously monitoring the individual's health.

In addition, such concentration may be obtained based on the heart rate. For example, concentration information may be obtained based on a change in heart rate, a size, and a change pattern, but is not limited thereto.

Referring to FIG. 80, the apparatus 8700 for measuring a biometric index according to an embodiment may obtain a biometric index and biometric information of a subject 8710.

In this case, the biometric index may include a biometric index such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, the biometric information may include biometric information such as sleepiness information, condition information, and concentration information, but is not limited thereto.

In addition, since the above-described contents may be applied to the method for obtaining the biometric index and biometric information, a redundant description will be omitted.

Referring to FIG. 81, a method of operating a biometric index measuring apparatus 8720 according to an embodiment includes obtaining a biometric index (S8730), obtaining biometric information (S8740), and performing an operation corresponding to the biometric information. At least a part of the step S8750 may be included.

In this case, since the above-described contents may be applied to the step of acquiring the biometric index (S8730) and the step of acquiring the biometric information (S8740), redundant descriptions will be omitted.

In addition, performing an operation corresponding to the biometric information (S8750) may include performing various operations corresponding to various biometric information.

For example, when the biometric information is drowsiness information, the biometric index measurement apparatus may perform an operation of outputting an alarm according to the drowsiness step, but is not limited thereto.

In addition, for example, when the biometric information is drowsiness information, the biometric index measurement apparatus may perform an operation of transmitting information to a manager according to the drowsiness stage, but is not limited thereto.

Further, for example, when the biometric information is drowsiness information, the biometric index measurement apparatus may perform an operation of transmitting information to the user's terminal according to the drowsiness step, but is not limited thereto.

Also, for example, when the biometric information is drowsiness information, the biometric index measurement device may perform an operation of transmitting information to the scheduling device according to the drowsiness step, but is not limited thereto.

In addition, for example, when the biometric information is drowsiness information, the biometric index measurement device may perform an operation of outputting a rest recommendation phrase according to the drowsiness stage, but is not limited thereto.

Further, for example, when the biometric information is condition information, the biometric index measurement apparatus may perform an operation of outputting an alarm according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric index measurement device may perform an operation of transmitting information to an administrator according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric index measurement apparatus may perform an operation of transmitting information to a user's terminal according to the condition, but is not limited thereto.

Further, for example, when the biometric information is condition information, the biometric index measurement device may perform an operation of transmitting information to a scheduling device according to the condition, but is not limited thereto.

Also, for example, when the biometric information is condition information, the biometric index measurement device may perform an operation of outputting a rest recommendation phrase according to the condition, but is not limited thereto.

Further, for example, when the biometric information is concentration information, the biometric index measurement device may perform an operation of outputting an alarm according to concentration, but is not limited thereto.

In addition, for example, when the biometric information is concentration level information, the biometric index measurement device may perform an operation of transmitting information to a manager according to the concentration level, but is not limited thereto.

Further, for example, when the biometric information is concentration information, the biometric index measurement apparatus may perform an operation of transmitting information to the user's terminal according to the concentration, but is not limited thereto.

Further, for example, when the biometric information is concentration information, the biometric index measurement device may perform an operation of transmitting information to a scheduling device according to concentration, but is not limited thereto.

In addition, the biometric index measurement device may perform an operation corresponding to the biometric information and the biometric index.

In addition, the apparatus for measuring the biometric index may perform various operations corresponding to biometric information and biometric index in addition to the above-described examples.

Referring to FIG. 82, a method of operating a biometric index measuring apparatus 8760 according to an embodiment includes obtaining a biometric index (S8770), obtaining biometric information (S8780), and calculating a study or work scheduling index. It may include at least some of the steps S8790.

In this case, since the above-described contents may be applied to the step of acquiring the biometric index (S8770) and the step of acquiring the biometric information (S8780), a redundant description will be omitted.

In addition, the step of calculating the study or work scheduling index (S8790) may be performed based on the obtained biometric index and/or biometric information.

For example, a study or work scheduling index may be calculated based on the acquired heart rate and drowsiness information, but is not limited thereto.

In addition, the step of calculating the study or work scheduling index (S8790) may be performed based on the obtained biometric index, biometric information, and/or information on study.

For example, a study or work scheduling index may be calculated based on the acquired sleepiness information and study time information, but is not limited thereto.

In addition, the study or work may be scheduled so that study or work is performed in a high-efficiency time period by using the study or work scheduling index.

In addition, by using the study or work scheduling index, it can be scheduled so that a break can be taken during a time period in which the efficiency is low.

Therefore, when the study or work scheduling index is used, it may be possible to schedule so that study or work can be performed with maximum efficiency within a limited time.

9.5. Various embodiments of a biometric index measuring device used for cognitive rehabilitation treatment

In the case of patients receiving cognitive rehabilitation treatment, such as dementia patients, treatment effects can be maximized only when high-intensive treatment is performed during cognitive rehabilitation treatment.

However, the supply and demand of rehabilitation therapists to monitor the individual patient's concentration may not meet the demand, so it may be difficult to perform high-quality treatment by increasing the individual patient's concentration.

Therefore, in order to compensate for this, when a biometric index and biometric information are obtained using a biometric index measuring device, it is possible to monitor the individual patient's concentration, and quality treatment can be performed using the biometric index and biometric information.

Referring to FIG. 83, a biometric index measurement device according to an embodiment includes a first biometric index measurement device 8800, a second biometric index measurement device 8801, a third biometric index measurement device 8802, and a fourth biometric index. At least one of the measuring device 8803 and the fifth biological index measuring device 8804 may be included.

In addition, at least one of the first to fifth biometric index measurement devices (8800, 8801, 8802, 8803, 8804) may be applied to at least one of the first to fourth patients (8811, 8812, 8813, 8814). The biomarker can be measured.

In this case, since the above-described information may be applied to the method of measuring the biometric index, a duplicate description will be omitted.

In addition, at least one of the first to fifth biometric index measurement devices (8800, 8801, 8802, 8803, 8804) may be applied to at least one of the first to fourth patients (8811, 8812, 8813, 8814). Biometric information can be obtained. For example, at least one of the first to fifth biometric

index measurement devices

8800, 8801, 8802, 8803, and 8804 may be used among the first to fourth patients (8811, 8812, 8813, 8814). At least one concentration information may be obtained, but the present invention is not limited thereto.

In addition, at least one of the first to fifth biometric index measurement devices (8800, 8801, 8802, 8803, 8804) may be applied to at least one of the first to fourth patients (8811, 8812, 8813, 8814). When there is an abnormality in the biometric information, an alarm may be output so that the rehabilitation therapist 8810 can recognize it. For example, at least one of the first to fifth biometric

index measurement devices

8800, 8801, 8802, 8803, and 8804 may be used among the first to fourth patients (8811, 8812, 8813, 8814). When there is an abnormality in the biometric information of at least one person, an audible alarm may be output, and information related to the biometric information may be transmitted to the mobile terminal of the rehabilitation therapist 8810, but is not limited thereto.

In addition, at least one of the first to fifth biometric index measurement devices (8800, 8801, 8802, 8803, 8804) may be applied to at least one of the first to fourth patients (8811, 8812, 8813, 8814). When there is an abnormality in the biometric information, an alarm may be output so that a patient with an abnormality in the biometric information can recognize it. For example, at least one of the first to fifth biometric

index measurement devices

8800, 8801, 8802, 8803, and 8804 may be used among the first to fourth patients (8811, 8812, 8813, 8814). When there is an abnormality in the biometric information of at least one person, an audible alarm can be output, and information related to the biometric information can be displayed through a corresponding patient's display.

In addition, as described above, monitoring a patient's biometric index and biometric information in real time to maintain concentration during treatment may be a method of maximizing the effect of cognitive rehabilitation treatment.

Referring to FIG. 84, a method 8850 of operating a biometric index measuring apparatus according to an embodiment includes obtaining a patient's biometric index (S8860), obtaining a patient's biometric information (S8870), and patient biometric information. It may include at least some of the step of performing an operation corresponding to (S8880).

At this time, since the above-described contents may be applied to the step of acquiring the patient's biometric index (S8860) and the step of acquiring the patient's biometric information (S8870), a duplicate description will be omitted.

*In addition, performing an operation corresponding to the biometric information of the patient (S8880) may include performing various operations corresponding to various biometric information.

For example, when the biometric information is concentration level information, the biometric index measurement device may perform an operation of outputting an alarm according to the concentration level, but is not limited thereto.

In addition, for example, when the biometric information is concentration information, the biometric index measurement device may perform an operation of transmitting information to be displayed on the patient's display according to the concentration, but is not limited thereto.

Further, for example, when the biometric information is concentration information, the biometric index measurement device may perform an operation of transmitting information to a mobile terminal of a rehabilitation therapist according to concentration, but is not limited thereto.

Further, for example, when the biometric information is concentration information, the biometric index measurement device may perform an operation of outputting a display for increasing the concentration according to the concentration, but is not limited thereto.

9.6. Various embodiments of a biometric index measuring device used for immigration

Quarantine at airports where people of various countries and races come and go can serve as an important gatekeeper to the national quarantine system.

However, it may be difficult to achieve close management of individuals due to a lack of management personnel compared to those who enter the airport.

Therefore, when an individual simply measures a biometric index at a place for immigration, close management of the individual may be more easily possible.

Referring to FIG. 85, the biometric

index measuring apparatus

8901,8902 according to an embodiment may be disposed in the immigration kiosk 8900 and measure the biometric index of the immigration 8911,8912.

In this case, the biometric

index measuring apparatus

8901,8902 may perform facial recognition used for immigration, but is not limited thereto.

In addition, the biometric index may include at least one of biomarkers such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, since the above-described contents may be applied to the method of measuring the biometric index by the biometric

index measuring apparatus

8901,8902, a redundant description will be omitted.

In addition, the biometric

index measuring apparatus

8901,8902 may perform an operation corresponding to the measured biometric index. For example, as shown in FIG. 85, when the measured body temperature of the first immigration person 8911 is within the normal range, the first biometric index measurement device 8901 displays first information 8921 indicating access permission. When the measured body temperature of the second immigration person 8912 is out of the normal range, the second biometric index measuring device 8902 operates to display the second information 8922 that requires more accurate body temperature measurement. It can be, but is not limited thereto.

In addition, strengthening quarantine by monitoring the biometric index of the immigration during immigration as described above may be a method of maximizing the effect of national quarantine.

Referring to FIG. 86, a method 8950 of operating a biometric index measuring apparatus according to an embodiment includes at least a portion of acquiring a biometric index of an immigration person (S8960) and performing an operation corresponding to the biometric index (S8970). It may include.

In this case, since the above-described contents may be applied to the step of acquiring the biometric index of the immigration person (S8960), a duplicate description will be omitted.

In addition, performing an operation corresponding to the biometric index (S8970) may include performing various operations corresponding to various biometric indexes.

For example, when the biometric index is a heart rate and the heart rate is within a normal range, the biometric index measurement device may perform an operation corresponding thereto, but is not limited thereto.

In addition, for example, when the biometric index is a heart rate and the heart rate is out of a normal range, the biometric index measurement device may perform an operation corresponding thereto, but is not limited thereto.

Further, for example, when the biometric index is an oxygen saturation degree and the oxygen saturation degree is within a normal range, the biometric index measurement device may perform an operation corresponding thereto, but is not limited thereto.

In addition, for example, the apparatus for measuring the biometric index may perform an operation corresponding to that when the biometric index is an oxygen saturation degree and the oxygen saturation degree is out of a normal range, but is not limited thereto.

In addition, for example, when the biometric index is a blood pressure and the blood pressure is within a normal range, the biometric index measurement device may perform an operation corresponding thereto, but is not limited thereto.

Further, for example, when the biometric index is a blood pressure and the blood pressure is out of a normal range, the biometric index measuring apparatus may perform an operation corresponding thereto, but is not limited thereto.

In addition, for example, when the biometric index is a body temperature and the body temperature is within a normal range, the biometric index measurement device may perform an operation corresponding thereto, but is not limited thereto.

In addition, for example, the apparatus for measuring the biometric index may perform an operation corresponding to the body temperature when the biometric index is a body temperature and the body temperature is out of a normal range, but is not limited thereto.

In addition, for example, when the biometric index is within a normal range, an operation corresponding to the biometric index may include displaying information allowing access, but is not limited thereto.

In addition, for example, when the biometric index is out of a normal range, the corresponding operation may include an operation of disallowing access or displaying information indicating a subject for detailed examination, but is not limited thereto.

In addition, the apparatus for measuring the biometric index may perform various operations corresponding to the biometric index in addition to the examples described above.

9.7. Various embodiments of a biometric index measuring device used in a security device

Security devices through biometric recognition such as face recognition, iris recognition, and fingerprint recognition are being activated.

However, there is also a situation in which the security device is disabled through photocopying and fingerprint copying.

Therefore, when a security device using a biometric index in addition to biometric recognition is provided, it is possible to prevent the security device from being disabled and become a stronger security device.

Referring to FIG. 87, the biometric index measuring apparatus 9010 according to an embodiment may be disposed on the security device 9000 and may measure the biometric index of the subject 9020.

More specifically, the security device 9000 may perform biometric recognition such as face recognition, fingerprint recognition, and iris recognition of the subject 9020, but is not limited thereto.

In addition, the biometric index measurement device 9010 may obtain the biometric index of the measurement target 9020, but is not limited thereto.

In this case, the biometric index may include at least one of biometric indexes such as heart rate, oxygen saturation, blood pressure, and body temperature, but is not limited thereto.

In addition, since the above-described contents may be applied to the method of measuring the biometric index by the biometric index measuring apparatus 9010, a redundant description will be omitted.

In addition, the security device 9000 may release security based on biometric recognition and a measured biometric index. For example, the security may be canceled by determining whether the user is primarily designated through iris recognition, and then secondly determining whether the user is a person through obtaining a biometric index, but is not limited thereto.

In addition, the security device 9000 may release security based on a plurality of biometrics and a plurality of biometrics. For example, the security may be canceled by determining whether a user is primarily designated through iris recognition and fingerprint recognition, and then secondly determining whether a person is a person through whether or not a heart rate and oxygen saturation are obtained, but is not limited thereto.

In addition, the security device 9000 may release security based on the biometric index. For example, when a signal related to the acquired heartbeat matches a signal related to a designated user's heartbeat, the security may be canceled by determining that the user is a designated user, but the present invention is not limited thereto.

In addition, as described above, enhancing security by additionally utilizing a biometric index measuring device to a security device may be a method of maximizing the effect of security enhancement.

Referring to FIG. 88, a method 9050 of operating a security device according to an embodiment includes determining whether or not biometrics are matched (S9060), determining whether to obtain a biometric index (S9070), and performing an operation according to the result. It may include a step (S9080).

In this case, the biometric recognition may include at least one of biometric recognition such as fingerprint recognition, face recognition, and iris recognition.

In addition, the biometric index may include at least one biometric index such as heart rate, oxygen saturation, blood pressure, and body temperature.

In addition, determining whether the biometrics match (S9060) may be performed using a conventional biometrics method, but is not limited thereto.

In addition, since the above-described contents may be applied to the method of obtaining the biometric index to determine whether to obtain the biometric index, a duplicate description will be omitted.

In addition, determining whether to obtain the biometric index (S9070) may include determining whether to obtain the biometric index. For example, determining whether the biometric index is acquired (S9070) may include determining whether a heart rate is acquired, but is not limited thereto.

In addition, determining whether the biometric index is acquired (S9070) may include determining whether or not the biometric signals for obtaining the biometric index are matched. For example, determining whether the biometric index is acquired (S9070) may include determining whether a heart rate signal for acquiring a heart rate matches a designated user's heart rate signal, but is not limited thereto.

In addition, the determining whether the biometric index is acquired (S9070) may include determining whether the biometric index matches. For example, determining whether the biometric index is acquired (S9070) may include determining whether a previously measured heart rate for the subject and a currently measured heart rate within a reference range based on the measured heart rate. , Is not limited thereto.

In addition, performing the operation according to the result (S9080) may include releasing security when all security cancellation conditions are satisfied, and maintaining security when the security cancellation conditions are not satisfied, but is limited thereto. It doesn't work.

In addition, the security device may perform various operations for enhancing security by using a biometric index in addition to the above-described examples.

9.8. Various embodiments of a biometric index measuring device used in a kiosk

When a biometric index measuring device that can easily measure a biometric index is placed on a kiosk for displaying information, it is possible to monitor the individual's health more continuously by simply monitoring the biometric index in daily life.

Referring to FIG. 89, a biometric index measuring apparatus 9110 according to an embodiment may be disposed in a kiosk 9100 and may measure a biometric index of a subject 9120.

In this case, the kiosk 9100 may include a display for displaying at least one piece of information. For example, as shown in FIG. 89, the kiosk 9100 may display date information and weather information, but is not limited thereto.

In addition, the biometric index measuring device 9110 may acquire at least one biometric index. For example, the biometric index measuring device 9110 may obtain a heart rate, but is not limited thereto, and may obtain at least one biometric index among biomarkers such as heart rate, oxygen saturation, blood pressure, and body temperature.

In addition, the biometric index measuring device 9110 may acquire at least one piece of biometric information. For example, the biometric index measurement device 9110 may acquire today's condition information, but is not limited thereto, and acquires at least one of biometric information such as drowsiness information, condition information, concentration information, and health information. can do.

In addition, the kiosk 9100 may display the obtained biometric index. For example, as shown in FIG. 89, the kiosk 9100 may display the acquired heart rate, oxygen saturation, and blood pressure, but is not limited thereto.

Also, the kiosk 9100 may display the obtained biometric information. For example, as shown in FIG. 89, the kiosk 9100 may display the acquired today's condition information, but is not limited thereto.

In addition, the kiosk 9100 may transmit the obtained biometric index and biometric information. For example, the kiosk 9100 may transmit the obtained biometric index and biometric information to the mobile terminal of the subject 9120, but is not limited thereto.

In addition, the kiosk 9100 may transmit the obtained biometric index and biometric information. For example, the kiosk 9100 may transmit the obtained biometric index and biometric information to a terminal that can be checked by an administrator, but is not limited thereto.

In addition, in this case, it is possible to detect a patient with a biometric index abnormality in the building in which the kiosk 9100 is disposed, and thus quarantine and security of the building may be reinforced.

In addition, as described above, the additional use of a biometric index measuring device on the kiosk enables continuous monitoring of individual health, and may be a method of reinforcing quarantine and security of the building in which the kiosk is disposed.

In addition, the above-described kiosk device can be used for manpower management, and for example, it is used for a kiosk for entry and exit of construction site workers so that measures can be taken against construction site workers with abnormal biometric indexes. It may be, but is not limited thereto.

Referring to FIG. 90, a method of operating a biometric index measuring apparatus 9150 according to an embodiment includes obtaining a biometric index (S9160), obtaining biometric information (S9170), and outputting the biometric index and biometric information. It may include at least some of the steps S9180.

In this case, in the step of obtaining the biometric index (S9160), the above-described biometric index obtaining method may be applied, and therefore, a redundant description will be omitted.

In addition, in the step of acquiring the biometric information (S9170), since the above-described biometric information acquisition method may be applied, a redundant description will be omitted.

In addition, the step of outputting the biometric index and biometric information (S9180) may include displaying the biometric index and biometric information. For example, the step of outputting the biometric index and biometric information (S9180) may include displaying heart rate, oxygen saturation, blood pressure, and today's condition information, but is not limited thereto.

In addition, the step of outputting the biometric index and the biometric information (S9180) may include transmitting the biometric index and the biometric information. For example, the step of outputting the biometric index and biometric information (S9180) may include transmitting heart rate, oxygen saturation, blood pressure, and today's condition information to another mobile terminal, but is not limited thereto.

In addition, the step of outputting the biometric index and the biometric information (S9180) may include printing the biometric index and the biometric information. For example, the step of outputting the biometric index and biometric information (S9180) may include printing heart rate, oxygen saturation, blood pressure, and today's condition information, but is not limited thereto.

In addition, the apparatus for obtaining biometric index may perform various operations of obtaining and outputting biometric index and biometric information in addition to the above-described examples.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

As described above, in the best mode for carrying out the invention, related matters have been described.

Claims

As a method of measuring a physiological parameter for a subject in a non-contact manner performed by one or more processors,

Obtaining a plurality of image frames for the subject;

Obtaining a first color channel value, a second color channel value, and a third color channel value for at least one image frame included in the plurality of image frames;

For at least one image frame included in the plurality of image frames, a first difference value and a second difference value are determined based on the first color channel value, the second color channel value, and the third color channel value. Calculating-the first difference value is a difference value between the first color channel value and the second color channel value for the same image frame, and the second difference value is the first color channel value for the same image frame Represents a difference between the value of the third color channel and the value of the third color channel;

The first difference value for at least one image frame included in the first image frame group with respect to the first image frame group acquired during a first preset time period and the first difference value for the first image frame group Obtaining a first characteristic based on an average value of;

Obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference value for the first image frame group; And

Including the step of determining a biometric index for the subject based on the first characteristic value and the second characteristic value,

The first color channel value is an average pixel value of a first color channel for one image frame, the second color channel value is an average pixel value of a second color channel for one image frame, and the third color The channel value represents the average pixel value of the third color channel for one image frame.

Biometric index measurement method.
The method of claim 1,

The biometric index includes at least one of heart rate and blood pressure.

Biometric index measurement method.
The method of claim 1,

The first, second, and third color channels are color channels according to an RGB color space.

Biometric index measurement method.
The method of claim 3,

Considering the absorbance of hemoglobin and oxyhemoglobin, in order to reduce noise,

The first color channel is set to a green channel, the second color channel is set to a red channel, and the third color channel is set to a blue channel.

Biometric index measurement method.
The method of claim 1,

The first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group,

The second characteristic value is obtained based on a second deviation value of the second difference value with respect to at least one image frame included in the first image frame group,

The first deviation value is calculated based on an average value of the first difference value for the at least one image frame and the first difference value for the first image frame group,

The second deviation value is calculated based on an average value of the second difference value for the at least one image frame and the second difference value for the first image frame group.

Biometric index measurement method.
The method of claim 5,

The first characteristic value and the second characteristic value are normalized values.

Biometric index measurement method.
The method of claim 6,

The first characteristic value is a value normalized by a first standard deviation value, and the second characteristic value is a value normalized by a second standard deviation value,

The first standard deviation value is a standard deviation value of the first difference value for the first image frame group, and the second standard deviation value is a standard deviation value of the second difference value for the first image frame group.

Biometric index measurement method.
The method of claim 1,

The biometric index for the subject is determined based on a third characteristic value obtained as a sum of the first characteristic value and the second characteristic value.

Biometric index measurement method.
The method of claim 1,

Further comprising the step of outputting the biometric index,

The determined biometric index includes a first biometric index and a second biometric index,

The output biometric index is determined based on the first biometric index and the second biometric index.

Biometric index measurement method.
The method of claim 9,

The first biometric index is determined based on a second image frame group, and the second biometric index is determined based on a third image frame group,

The number of image frames included in the first image frame group is smaller than the number of image frames included in the second and third image frame groups,

The first image frame group is included in the second image frame group.

Biometric index measurement method.
The method of claim 10,

The number of image frames included in the second image frame group is equal to the number of image frames included in the third image frame group.

Biometric index measurement method.
The method of claim 1,

Further comprising the step of outputting a biometric index based on the determined biometric index,

The determined biomarker includes at least four preliminary biomarkers,

The output biometric index is determined based on the four preliminary biometric indexes.

Biometric index measurement method.
The method of claim 1,

Further comprising the step of outputting a biometric index based on the determined biometric index,

The output biometric index includes a first biometric index and a second biometric index,

The second biometric index is a biometric index of the same type as the first biometric index, and the second biometric index is output after the first biometric index is output,

When the difference between the second biometric index and the first biometric index exceeds a reference value, the second biometric index is corrected and output.

Biometric index measurement method.
As a method of measuring a physiological parameter for a subject in a non-contact manner using an infrared camera,

Acquiring a plurality of image frames for a subject by using an infrared camera;

Obtaining a first region value, a second region value, and a third region value for at least one image frame included in the plurality of image frames;

Calculating a first difference value and a second difference value based on the first area value, the second area value, and the third area value for at least one image frame included in the plurality of image frames;

The first difference value for at least one image frame included in the first image frame group with respect to the first image frame group acquired during a first preset time period and the first difference value for the first image frame group Obtaining a first characteristic value based on an average value of;

Obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference value for the first image frame group; And

Including the step of determining a biometric index for the subject based on the first characteristic value and the second characteristic value,

The first region value is an average pixel value for a first region of interest in one image frame, the second region value is an average pixel value for a second region of interest in one image frame, and the third region value is Is the average pixel value for the third region of interest in one image frame,

The first difference value is a difference value between a first area value and a second area value for the same image frame, and the second difference value is a difference value between a first area value and a third area value for the same image frame.

Biometric index measurement method.
The method of claim 14,

The biometric index includes at least one of heart rate and blood pressure.

Biometric index measurement method.
The method of claim 14,

The first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group,

The second characteristic value is obtained based on a second deviation value of the second difference value with respect to at least one image frame included in the first image frame group,

The first deviation value is calculated based on an average value of the first difference value for the at least one image frame and the first difference value for the first image frame group,

The second deviation value is calculated based on an average value of the second difference value for the at least one image frame and the second difference value for the first image frame group.

Biometric index measurement method.
As a biometric index measuring device for measuring a physiological parameter for a subject in a non-contact manner,

An image acquisition unit for acquiring an image frame for the subject; And

Including a control unit for obtaining a biometric index using the image frame,

The control unit

Obtaining a first color channel value, a second color channel value, and a third color channel value for at least one image frame included in the plurality of acquired image frames,

For at least one image frame included in the plurality of image frames, a first difference value and a second difference value are calculated based on the first color channel value, the second color channel value, and the third color channel value, ,

The first difference value for at least one image frame included in the first image frame group with respect to the first image frame group acquired during a first preset time period and the first difference value for the first image frame group To obtain a first characteristic value based on the average value of,

Obtaining a second characteristic value based on a second difference value for at least one image frame included in the first image frame group and an average value of the second difference value for the first image frame group, and

Determining a biometric index for the subject based on the first characteristic value and the second characteristic value,

The first color channel value is an average pixel value for a first color channel in one image frame, the second color channel value is an average pixel value for a second color channel in one image frame, and the third color The channel value is the average pixel value for the third color channel in one image frame,

The first difference value is a difference value between a first color channel value and a second color channel value for the same image frame, and the second difference value is a value of a first color channel value and a third color channel value for the same image frame. Difference value

Biometric index measuring device.
The method of claim 17,

The first, second, and third color channels are color channels according to an RGB color space,

Considering the absorbance of hemoglobin and oxyhemoglobin, in order to reduce noise,

The first color channel is set to a green channel, the second color channel is set to a red channel, and the third color channel is set to a blue channel.

Biometric index measuring device.
The method of claim 17,

The first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group,

The second characteristic value is obtained based on a second deviation value of the second difference value with respect to at least one image frame included in the first image frame group,

The first deviation value is calculated based on an average value of the first difference value for the at least one image frame and the first difference value for the first image frame group,

The second deviation value is calculated based on an average value of the second difference value for the at least one image frame and the second difference value for the first image frame group.

Biometric index measuring device.
The method of claim 17,

The control unit acquires biometric information based on the determined biometric index,

The biometric information includes at least one of emotion information, drowsiness information, stress information, and excitement level information.

Biometric index measuring device.
As a method of measuring a physiological parameter for a subject in a non-contact manner,

Obtaining a plurality of image frames for the subject;

Obtaining a first color channel value, a second color channel value, and a third color channel value for at least one image frame included in the plurality of image frames;

Obtaining a first characteristic value based on the first, second, and third color channel values for at least one image frame included in the first image frame group acquired during a first preset time;

Obtaining a second characteristic value based on the first, second, and third color channel values for at least one image frame included in a second image frame group acquired during a second preset time; And

Including; determining a biometric index based on the first characteristic value and the second characteristic value;

The first image frame group and the second image frame group at least partially overlap,

To determine the biomarker,

A first characteristic value for a first image frame included in the first image frame group and not included in the second image frame group is used, and included in the first image frame group and the second image frame group. The first characteristic and the second characteristic value of the second image frame are used, and the second characteristic value of the third image frame included in the second image frame group but not included in the first image frame group is used. felled

Biometric index measurement method.
The method of claim 21,

Computing a first difference value based on at least a portion of the first color channel value, the second color channel value, and the third color channel value for at least one image frame included in the plurality of image frames. Including,

The first characteristic value is obtained based on an average value of the first difference value for at least one image frame included in the first image frame group and the first difference value for the first image frame group,

The second characteristic value is obtained based on an average value of the first difference value for at least one image frame included in the second image frame group and the second difference value for the second image frame group,

The first difference value is a difference value between a first color channel value and a second color channel value for the same image frame, and the second difference value is a value of a first color channel value and a third color channel value for the same image frame. Difference value

Biometric index measurement method.
The method of claim 21,

The first characteristic value is obtained based on a first deviation value of the first difference value with respect to at least one image frame included in the first image frame group,

The second characteristic value is obtained based on a second deviation value of the first difference value with respect to at least one image frame included in the second image frame group,

The first deviation value is calculated based on an average value of a first difference value for at least one image frame included in the first image frame group and the first difference value for the first image frame group,

The second deviation value is calculated based on an average value of a first difference value for at least one image frame included in the second image frame group and the first difference value for the second image frame group.

Biometric index measurement method.
As a biometric index output method that obtains and outputs a biometric index in a non-contact manner,

Obtaining a plurality of image frames including a first image frame group and a second image frame group at least partially overlapping with the first image frame group;

Obtaining a biometric index based on the first image frame group;

Outputting a biometric index at a first time point;

Obtaining a first biometric index based on the second image frame group;

Outputting a biometric index at a second time point later than the first time point; Including,

The biometric index output at the first time point is a biometric index obtained based on the first image frame group,

The biometric index output at the second time point is

When the difference between the first biometric index and the biometric index output at the first time point is less than or equal to a reference value, it is the first biometric index,

When the difference between the first biometric index and the biometric index output at the first time point exceeds a reference value, the biometric index corrected from the biometric index output at the first time point

Biometric index output method.
The method of claim 24,

The corrected biometric index is

When the first biometric index is greater than the biometric index output at the first time point, it is a biometric index corrected by adding a preset value to the biometric index output at the first time point,

If the first biometric index is smaller than the biometric index output at the first time point, the biometric index corrected by subtracting a preset value from the biometric index output at the first time point

Biometric index output method.