JP6739038B2 - Human state estimation method and human state estimation system - Google Patents

Description

The present invention relates to a human state estimation method and a human state estimation system.

It is said that a state such as a degree of stress or drowsiness felt by a person (hereinafter, also referred to as a human state) is related to a physiological amount based on the activity of a person's autonomic nerves. The physiological amount based on the activity of the autonomic nerve includes, for example, skin temperature, variation in heartbeat interval, respiratory waveform, and the like. For example, as for the skin temperature as a physiological amount, when a person feels stress, the peripheral blood vessels are contracted and the blood flow is reduced due to the enhancement of the sympathetic nerves of the autonomic nerves. It is explained by the mechanism that the skin temperature decreases. Regarding sleepiness as a physiological amount, by the mechanism that peripheral blood vessels are relaxed by increasing parasympathetic nerves among autonomic nerves and blood flow is increased, the peripheral skin temperature is increased. Explained. A drowsiness detection device utilizing this is disclosed (see Patent Documents 1 and 2).

JP, 9-154835, A JP 2002-120591 A

However, since the activity of the autonomic nerve is affected by other than stress or drowsiness, further studies have been required to accurately estimate the human state such as the degree of stress or drowsiness.

Therefore, the present invention provides a human state estimation method and the like for improving the accuracy of human state estimation.

A human state estimation method according to an aspect of the present invention is a human state estimation method for estimating a human state that is a state of a human being, wherein the estimated thermal sensation of the person is indexed within a prescribed range. Acquiring a feeling index, it is determined whether or not the acquired thermal sensation index is within a predetermined range of the specified range, and the acquired thermal sensation index is determined to be within the predetermined range. The human state is estimated based on the physiological amount acquired when the human body's autonomic nerve activity is reflected.

Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer. It may be realized by any combination of the program and the recording medium.

The human state estimation method of the present invention improves the accuracy of human state estimation.

FIG. 1A is a conceptual diagram of an automobile equipped with the human state estimation device according to the first embodiment. FIG. 1B is a schematic view of an automobile equipped with the human state estimation device according to the first embodiment, as viewed from above. FIG. 2 is an explanatory diagram showing an example of a human face image taken by the thermal image sensor according to the first embodiment. FIG. 3 is a schematic configuration diagram of the thermal sensation estimation unit in the first embodiment. FIG. 4 is an explanatory diagram showing an example of the thermal sensation index in the first embodiment. FIG. 5 is a block diagram showing the function of the thermal sensation estimation unit in the first embodiment. FIG. 6 is a block diagram showing functions related to human state estimation in the first embodiment. FIG. 7A is an explanatory diagram showing an example of the thermal sensation, the outside air temperature, and the time variation of the nose skin temperature in the first embodiment when the thermal sensation is 2 or more. FIG. 7B is an explanatory diagram showing an example of the temporal sensation of the thermal sensation, the outside air temperature, and the nose skin temperature in the first embodiment when the thermal sensation is near zero. FIG. 8 is an explanatory diagram showing an example of the drowsiness index in the first embodiment. FIG. 9A is an explanatory diagram showing an example of the thermal sensation, the outside air temperature, and the time variation of the nose skin temperature in the first embodiment when the thermal sensation is −2 or less. FIG. 9B is an explanatory diagram showing an example when the thermal sensation is near 0 with respect to the temporal variability of the thermal sensation, the outside air temperature, and the nose skin temperature in the first embodiment. FIG. 10 is a block diagram showing functions of the human state estimating device in the first embodiment. FIG. 11 is a flowchart showing a human state estimation method by the human state estimation device according to the first embodiment. FIG. 12A is an explanatory diagram of nose skin temperature measurement according to the first embodiment. FIG. 12B is an explanatory diagram of nostril temperature fluctuations in nose skin temperature measurement according to the first embodiment. FIG. 12C is a schematic diagram showing a pulse wave measuring method according to the first embodiment. FIG. 12D is a schematic diagram showing a pulse wave measuring method according to the first embodiment. FIG. 13A is a conceptual diagram of an automobile equipped with the human state estimation device according to the second embodiment. FIG. 13B is a schematic view of a person wearing the human state estimating device according to the second embodiment, as viewed from the front. FIG. 14A is an explanatory diagram showing an example of the measured pulse wave in the second embodiment. FIG. 14B is an explanatory diagram of time-series fluctuations of heartbeat intervals obtained from the pulse wave shown in FIG. 14A. FIG. 15A is an explanatory diagram showing a frequency component of a heartbeat interval and a low HF component in the second embodiment. FIG. 15B is an explanatory diagram showing frequency components of the heartbeat interval in the second embodiment, which have many HF components. FIG. 16 is a correlation diagram showing an example of the correlation between earlobe skin temperature and thermal sensation in the second embodiment. FIG. 17 is a block diagram showing functions of the human state estimating device in the second embodiment. FIG. 18 is a flowchart showing a human state estimation method by the human state estimation device according to the second embodiment. FIG. 19A is an explanatory diagram of a respiratory frequency according to the processing load reduction method using a respiratory component according to the second embodiment. FIG. 19B is a block diagram showing a wavelet transform according to the processing load reduction method using the respiratory component according to the second embodiment. FIG. 20 is a flowchart showing a human state estimating method to which the processing load reducing method using the respiratory component according to the second embodiment is applied. FIG. 21 is a block diagram showing a function of correcting the human condition determination result according to the second embodiment.

(Findings that form the basis of the present invention)
Human organs are not actively controlled by human intention, but are controlled by the central part of the brain via autonomic nerves based on information from receptors throughout the body. Autonomic nerves consist of two systems, sympathetic nerves and parasympathetic nerves, and both sympathetic nerves and parasympathetic nerves often dominate one organ.

For example, with respect to the heart rate of the heart, when the sympathetic nerve is enhanced, the heart rate is increased and the variation in heartbeat intervals is reduced. On the other hand, when the parasympathetic nerve is enhanced, the heart rate is decreased and the heartbeat interval becomes large. Further, in the peripheral blood vessels, when the sympathetic nerve is enhanced, the blood vessels contract and the blood flow is inhibited, and as a result, the peripheral skin temperature decreases. On the contrary, when the parasympathetic nerve is enhanced, the blood vessels are relaxed and the blood flow is improved, resulting in an increase in the skin temperature in the peripheral part.

Peripheral skin temperature decreases under stress because it stimulates blood flow in the deep part of the body by enhancing sympathetic nerves, and increasing heart rate causes large amounts of blood to flow into the brain and muscles to supply oxygen. However, as a result, it is thought to activate the activity of the brain and muscles. Also, the rise in peripheral skin temperature due to drowsiness requires that the core body temperature during sleep be slightly lower than that during awakening.Therefore, by increasing the parasympathetic nerves and spreading blood flow to the peripheral part, body temperature is increased. It is considered to facilitate heat dissipation.

The enhancement of the sympathetic nerve or the parasympathetic nerve is influenced by a person's thermal sensation (a feeling of heat or cold felt by a person) in addition to the effects of stress and drowsiness described above. For example, when a person feels it is hot, it is necessary to promote heat dissipation from the body, so the parasympathetic nerve is enhanced in order to relax peripheral blood vessels. As a result, the heart rate decreases, and the variation in heartbeat intervals also increases. On the other hand, when a person feels cold, it is necessary to suppress the heat radiation from the body, and therefore the sympathetic nerve is enhanced in order to contract the peripheral blood vessels. As a result, the heart rate increases and the variation in heartbeat intervals also decreases. Therefore, when estimating the human state such as stress or drowsiness based on the physiological amount based on the activity of the autonomic nerve such as the skin temperature or the heartbeat interval, the physiological amount depends on the influence of stress or drowsiness. It is necessary to distinguish whether it is due to fluctuations in thermal sensation.

In order to solve such a problem, a human state estimation method according to an aspect of the present invention is a human state estimation method for estimating a human state that is a human state, and the human estimated thermal sensation of the person. To obtain a thermal sensation index indexed within a specified range, the acquired thermal sensation index is determined whether or not it is within a predetermined range of the specified range, the acquired thermal sensation index Is a physiological amount acquired when is determined to be within the predetermined range, and the human state is estimated based on a physiological amount that reflects the activity of the human autonomic nerve.

According to this, the human state can be estimated with higher accuracy by reducing erroneous detection at the time of estimating the human state. Specifically, in the human state estimation method according to one aspect of the present invention, the human state is estimated from the physiological amount under the condition that the acquired thermal sensation information is within the predetermined range. Therefore, erroneous detection at the time of human state estimation can be reduced as compared with the case of estimating the human state regardless of whether or not this condition is satisfied. Therefore, the human state estimation method can improve the accuracy of human state estimation. Furthermore, by reducing the above-mentioned erroneous detection, it is not necessary to perform the detection process again, and thus it is possible to obtain the effects of reducing the processing amount and the processing load and the power consumption.

For example, the predetermined range is a part of the specified range that includes a thermal neutral point related to thermal sensation.

According to this, when the person's thermal sensation is within a range relatively close to the thermal neutral point at which the person does not feel hot or cold, the human state is estimated. When a person feels hot or warm, the skin temperature rises by promoting heat dissipation from the body. On the other hand, when a person feels cold or cool, suppressing heat radiation from the body lowers the skin temperature. In other words, when the person's thermal sensation is within a range relatively close to the thermal neutral point, there is no or relatively little effect of promoting or suppressing heat radiation from the human body. Therefore, by estimating the human state from the physiological amount in such a case, it is possible to reduce the influence of promotion or suppression of heat radiation from the human body, which is included in the estimation result, and prevent the false detection and the estimation accuracy. Contribute to the improvement of.

For example, the predetermined range is a part of the specified range that does not include the hottest point indicating the hottest point and the coldest point indicating the coldest point.

According to this, when a person's thermal sensation is in a range excluding the time when he/she feels very hot or very cold, the human state is estimated. When a person feels very hot, heat dissipation from the human body is greatly promoted. Further, when a person feels very cold, heat radiation from the human body is greatly suppressed. By excluding such a case from the case of estimating the human state, it is possible to reduce the influence of promotion or suppression of heat radiation from the human body, which is included in the estimation result, and prevent the false detection and the estimation accuracy. Contribute to the improvement of.

For example, the physiological amount is the nose skin temperature of the person, and the person state includes the degree of drowsiness of the person.

According to this, when estimating the drowsiness of a person based on the skin temperature of the nose of the person, it is possible to reduce the influence of promotion or suppression of heat radiation from the body of the person, which is included in the estimation result. .. Further, by using the skin temperature of the nose, it is possible to reduce the influence of disturbance even when there is disturbance.

For example, in the person state estimation method, the Tokushi preparative and thermal sensation index the nasal skin temperature Prefecture, upon said determining, in the each predetermined range of the obtained pre-Symbol thermal sensation index determines whether the time of estimation of the person condition, based on the rise with time of Kihana part skin temperature before acquired, to estimate the degree of the person of sleepiness.

According to this, the degree of drowsiness of a person can be specifically estimated based on the skin temperature of the nose of the person.

For example, the physiological amount includes the heartbeat interval of the person, and the human state estimation method estimates the human state based on the variation of the heartbeat interval as the acquired physiological amount.

According to this, the human state can be specifically estimated based on the heartbeat interval of the person.

For example, the physiological amount includes the respiratory waveform of the person, and the human state estimation method further estimates the human state based on the acquired respiratory waveform as the physiological amount.

According to this, the human state can be specifically estimated based on the respiratory waveform of the human. Further, it is possible to contribute to reduction of processing load and speeding up of processing.

For example, the skin temperature of the earlobe of the person is acquired, and when the thermal sensation index is acquired, the correlation between the skin temperature of the earlobe and the thermal sensation index is used to acquire the temperature of the earlobe. Acquiring the thermal sensation index estimated from skin temperature, when acquiring the physiological amount, the pulse wave measured from the earlobe of the person is acquired as the physiological amount to estimate the human condition. At that time, the human state is estimated based on the frequency analysis of the acquired pulse wave.

According to this, the human condition can be specifically estimated based on the skin temperature and the pulse wave of the human earlobe. The earlobe is characterized in that it is easy to measure the pulse wave. Therefore, the pulse wave is acquired by the earlobe and the skin temperature is also acquired, so that the information necessary for estimating the human state can be collectively acquired from the earlobe.

For example, in the human state estimation method, the thermal sensation index estimated by PMV (Predicted Mean Vote) is acquired.

For example, the specified range is represented by a 7-level evaluation scale of PMV, and the predetermined range is a range in which the PMV value in the specified range is −2 or more and +2 or less.

According to this, it is possible to more accurately estimate the thermal sensation of a person from the temperature, humidity, air flow, radiation, amount of clothing, and amount of activity, which are the six elements of heat and heat, by the PMV estimation.

For example, the physiological amount is the nose skin temperature of the person, the human state includes the degree of stress of the person, in the human state estimation method, the thermal sensation index and the nose skin temperature and It acquires the in the determination, the previous SL thermal sensation indicators acquired to determine whether or not it is within the predetermined range, the time of estimation of the person condition, the obtained pre-Symbol The degree of stress of the person is estimated based on the fall width of the nose skin temperature over time.

According to this, the degree of stress of a person can be specifically estimated based on the skin temperature of the nose of the person.

For example, the physiological amount is skin blood flow, blood pressure, or pulse wave transit time of the person, and the person state includes the degree of drowsiness of the person.

According to this, the degree of drowsiness of a person can be specifically estimated based on the skin blood flow, blood pressure, or pulse wave transit time of the person.

A human state estimation system according to one aspect of the present invention is a human state estimation system that estimates a human state that is a state of a person, and the estimated thermal sensation of the person is indexed within a prescribed range. A thermal sensation estimation unit that acquires a thermal sensation index, a determination unit that determines whether the acquired thermal sensation index is within a predetermined range of the specified range, and the acquired thermal sensation A physiological state obtained when the index is determined to be within the predetermined range, a human state estimation unit that estimates the human state based on a physiological amount that reflects the activity of the autonomic nerve of the person. Prepare

According to this, the same effect as that of the above-mentioned human state estimation method can be obtained.

A program according to one aspect of the present invention is a program that causes a computer to execute the human state estimation method described above.

According to this, the same effect as that of the above-mentioned human state estimation method can be obtained.

Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer. It may be realized by any combination of programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements not described in the independent claim showing the highest concept are described as arbitrary constituent elements.

(Embodiment 1)
In the present embodiment, a human state estimation method and a human state estimation device that improve the accuracy of human state estimation will be described. The human state is a concept including the degree of drowsiness or the degree of stress of a person as described above. The human state estimation device may be referred to as a human state estimation system.

An example of the case where the human state estimating device according to the present embodiment is installed in an automobile will be described with reference to FIGS. 1 to 12D.

FIG. 1A is a conceptual diagram of an automobile equipped with the human state estimating device according to the present embodiment. FIG. 1B is a schematic view of an automobile equipped with the human state estimating device according to the present embodiment, as viewed from above.

A person 102 is in the driver's seat of the automobile 100. The automobile 100 is equipped with a thermal image sensor 101 in front of the driver's seat toward the person 102, and can take a two-dimensional image of the heat distribution between the face of the person 102 and the periphery thereof. In the thermal image sensor 101, elements such as a bolometer or a thermopile, which are generally sensitive to infrared rays, are arranged in a two-dimensional matrix, and the amount of infrared rays emitted according to the temperature distribution on the object plane is arranged by the lens in the matrix shape. By forming a uniform image, the temperature distribution on the object surface can be visualized.

FIG. 2 is an explanatory diagram showing an example of an image of a human face taken by the thermal image sensor 101 according to the present embodiment.

When the thermal image sensor 101 and the person 102 are in the positional relationship as shown in FIG. 1, a thermal image as shown in FIG. 2, for example, is taken. In the thermal image shown in FIG. 2, the higher the temperature of the object (pixel), the higher the density is displayed. That is, in the thermal image shown in FIG. 2, the higher the temperature of the pixel, the closer the color of the pixel is displayed to the black color, and it can be recognized that the temperature around the forehead is higher. The display of the thermal image is not limited to this. Further, the image acquired by the thermal image sensor 101 may be a still image or a moving image.

FIG. 3 is a schematic configuration diagram of the thermal sensation estimation unit 108 in the present embodiment. FIG. 4 is an explanatory diagram showing an example of the thermal sensation index in the present embodiment. FIG. 5 is a block diagram showing functions of the thermal sensation estimation unit 108 in the present embodiment. The thermal sensation estimation unit 108 arranged inside the automobile 100 will be described with reference to these drawings.

A person's thermal sensation can be quantified using, for example, seven-stage indexes as shown in FIG. Here, the thermal sensation 0 is a thermal neutral point that is neither hot nor cold. As the temperature gets higher, the absolute value becomes larger and the absolute value becomes larger, and as the temperature gets colder, the negative absolute value becomes larger. Quantify as you take. The quantified (indexed) thermal sensation is called the thermal sensation index. The thermal sensation index may be simply thermal sensation within a range that does not cause confusion.

PMV is one that associates this thermal sensation with the six thermal elements. Therefore, when the temperature, humidity, radiation, airflow, amount of activity of a person, and the amount of clothing, which are the six heat and heat factors, are known, it is known that the thermal sensation of that person can be estimated using the PMV calculation formula. Has been. Hereinafter, a method for specifically determining the thermal sensation from PMV will be described. It should be noted that the index of human thermal sensation may be 9 levels in which “+4 is very hot” and “−4 is very cold” are added to the 7 levels shown in FIG.

As shown in FIG. 3, the thermal sensation estimation unit 108 includes a camera 103, a glove thermometer 104a, a thermometer 104b, an anemometer 105a provided near the louver 105b, a hygrometer 107, and the like. And a thermal sensation estimation control unit 106 connected to.

The thermal sensation estimation unit 108 is a processing unit that acquires a thermal sensation index in which the estimated thermal sensation of the person 102 is indexed within a specified range. The thermal sensation estimation unit 108 specifically acquires a thermal sensation index estimated by PMV. In this case, the specified range is represented by a 7-level evaluation scale of PMV, and the predetermined range is a range in which the PMV value at the 7 levels is −2 or more and +2 or less. Hereinafter, an example in which the thermal sensation estimation unit 108 acquires the thermal sensation index by estimating the thermal sensation index will be described. However, the thermal sensation estimation unit 108 uses the thermal sensation index estimated by another device or the like. It may be obtained from the device or the like.

The configuration of the thermal sensation estimation unit 108 will be described with reference to FIG.

The air temperature of the six heat factors can be obtained from the thermometer 104b.

Radiation of the six warm heat elements can be obtained from the glove thermometer 104a and the thermometer 104b. The glove thermometer is a thermometer in which a glass thermometer is inserted into a copper ball having a black surface. Since the glove thermometer measures not only the ambient temperature but also the temperature including the ambient radiation by the copper spheres coated in black, the difference between the measured values of the globe thermometer 104a and the thermometer 104b causes the influence of the radiation. Can be estimated.

The airflow of the six elements of warm heat is measured by the anemometer 105. The anemometer 105 corresponds to the anemometer 105a.

The humidity of the six elements of heat and heat is measured by the hygrometer 107.

The clothing amount of the six heat factors can be obtained by analyzing the image of the person 102 taken by the camera 103. That is, the clothing amount estimation unit 106a of the thermal sensation estimation control unit 106 calculates the image of the person 102 captured by the camera 103. The clothing amount estimation unit 106a obtains the area between the clothing portion and the exposed portion, that is, the skin portion, from the image captured by the camera 103, and determines the ratio between the unclothed portion and the clothing portion of the person 102. The amount of clothes may be obtained, the amount of clothes may be obtained from the ratio of the exposed part of the wrist to the arm part having clothes, or the difference in temperature between the clothes part and the exposed part. This is because the temperature of the clothing surface tends to approach the body surface temperature as the amount of clothing decreases. Of course, the amount of clothing may be obtained by other means, or may be entered by the person himself/herself and input.

The activity amount of a person out of the six elements of heat and heat is almost close to sitting on a chair while driving a car, and is usually considered to be about 1.1 mets without any particular measurement. Even when the human state estimating device is used in a place other than an automobile, for example, the camera 103 captures a moving image of the person 102, the human region of each image is specified, and the activity amount is calculated from the variation amount of the human region. It is also possible to estimate.

With the above method, the six heat factors related to the person 102 can be extracted. It should be noted that the method of obtaining the six thermal elements shown here is merely an example, and the present invention is not limited to this.

Next, for the PMV calculation unit 106b of the thermal sensation estimation control unit 106, the amount of clothing obtained by the clothing amount estimation unit 106a, the temperature and radiation obtained from the glove thermometer 104a and the thermometer 104b, and the anemometer 105. The wind speed obtained by the above and the humidity obtained by the hygrometer 107 are input to the PMV calculation unit 106b, and the PMV calculation unit 106b calculates the thermal sensation of the person 102 from the formula of PMV. The above is the operation mechanism of the thermal sensation estimation unit 108.

Next, a method of estimating the human state, here, a method of estimating using the nose skin temperature will be described.

FIG. 6 is a block diagram showing functions relating to human state estimation in the present embodiment.

This function is realized by the thermal image sensor 101, the physiological amount acquisition unit 109A, and the human state estimation unit 110.

The thermal image sensor 101 acquires a thermal image of the person 102.

The physiological amount acquisition unit 109A is a processing unit that acquires a physiological amount from the thermal image acquired by the thermal image sensor 101. The physiological amount acquisition unit 109A includes an image processing unit 109, and acquires the physiological amount using the image processing unit 109. Here, an example in which the skin temperature of the nose of the person 102 is used as the physiological amount will be described.

The image processing unit 109 extracts the nose from the thermal image of the person 102 acquired by the thermal image sensor 101, and calculates the nose skin temperature, which is the temperature of the extracted nose. An example of the method of extracting the nose skin temperature will be described later. The nose skin temperature calculated by the image processing unit 109 is input to the human state estimation unit 110 by the physiological amount acquisition unit 109A, and the human state estimation unit 110 estimates the human state based on the nose skin temperature.

An example of processing when the human state estimation unit 110 detects the degree of drowsiness as a human state will be described.

FIG. 7A is an explanatory diagram showing an example when the thermal sensation is 2 or more with respect to the temporal variability of the thermal sensation, the outside air temperature, and the nose skin temperature in the present embodiment.

7A is calculated by the nose skin temperature calculated by the physiological amount acquisition unit 109A, the ambient temperature (outside air temperature) of the person 102 measured by the thermometer 104b, and the thermal sensation estimation control unit 106 (PMV calculation unit 106b). The change with the thermal sensation of the person 102 who has done this is shown. Now, as shown in FIG. 7A, it is assumed that the outside air temperature is almost constant at 28° C. and the thermal sensation exceeds +2. Further, it is assumed that the nose skin temperature rises by about 1.5° C. within the illustrated time range.

When a person feels warm, he or she tries to promote blood flow by expanding peripheral blood vessels in order to promote heat dissipation from the body. Therefore, peripheral skin temperature rises, but as described above, when a person feels drowsiness, peripheral blood vessels are expanded to try to improve blood flow. It's hard to tell whether it's just because you're feeling, or whether you're feeling drowsy.

FIG. 7B is an explanatory diagram showing an example when the thermal sensation is near 0 with respect to the temporal variability of the thermal sensation, the outside air temperature, and the nose skin temperature in the present embodiment.

As shown in FIG. 7B, for example, the outside air temperature is lowered to about 25° C., which is lower than about 28° C. in the case of FIG. 7A, and the thermal sensation exists near zero, which is a thermal neutral point. Further, it is assumed that the nose skin temperature rises by about 1.5° C. within the illustrated time range (the temperature is different), as in FIG. 7A.

In this case, since it is not necessary to promote heat radiation from the body, no increase in peripheral skin temperature such as the nose due to an increase in peripheral blood flow can be seen. Therefore, at the stage where the thermal sensation exists near the thermal neutral point as in FIG. 7B, it can be estimated that the rise in the nose skin temperature as in FIG. 7B is caused by drowsiness. ..

Here, it is difficult to estimate the thermal sensation based on the temperature range of the outside air temperature. Specifically, for example, even if the outside temperature is the same, due to the difference in the amount of clothing (for example, when wearing only a T-shirt or wearing thick down jacket, etc.), it is natural that a person feels warm and cold. Since it is different, it is difficult to estimate the thermal sensation depending on the temperature range of the outside air temperature. Therefore, it is necessary to judge not by the outside temperature but by the thermal sensation of the person.

It should be noted that the degree of drowsiness may be divided into five levels according to the temperature range of the varied nasal skin temperature as shown in FIG. If the temperature fluctuation range of the nose skin temperature is large, it can be judged that the person is extremely sleepy, and if the temperature fluctuation range of the nose skin temperature is small, it may be a little sleepy. The relationship between the width and the drowsiness degree may be appropriately determined according to the difference in the temperature fluctuation width.

Next, an example of processing when the human state estimation unit 110 detects a stress level as a human state will be described.

FIG. 9A is an explanatory diagram showing an example of a case where the thermal sensation is −2 or less with respect to the temporal variability of the thermal sensation, the outside air temperature, and the nose skin temperature in the present embodiment.

9A is calculated by the nose skin temperature calculated by the physiological amount acquisition unit 109A, the ambient temperature (outside air temperature) of the person 102 measured by the thermometer 104b, and the thermal sensation estimation control unit 106 (PMV calculation unit 106b). The change with the thermal sensation of the person 102 who has done this is shown. Now, as shown in FIG. 9A, it is assumed that the outside air temperature is constant at about 18° C. and the thermal sensation is below −2. When a person feels cold, he or she tries to suppress blood flow by contracting peripheral blood vessels in order to suppress heat dissipation from the body. Therefore, the peripheral skin temperature will decrease, but as described above, when a person feels stress, it tries to contract the peripheral blood vessels to suppress the blood flow, so this decrease in the nose skin temperature is It's hard to tell if it's just because you're feeling cold or if you're also feeling stress.

FIG. 9B is an explanatory diagram showing an example when the thermal sensation is near 0 with respect to the temporal variability of the thermal sensation, the outside air temperature, and the nose skin temperature in the present embodiment.

As shown in FIG. 9B, for example, when the outside air temperature rises to about 22° C., which is higher than about 18° C. in the case of FIG. 9A, and the thermal sensation exists near zero, which is the thermal neutral point, heat radiation from the body Since it is not necessary to suppress the above, no decrease in peripheral skin temperature such as the nose due to a decrease in peripheral blood flow will not be seen. Therefore, at the stage where the thermal sensation exists near the thermal neutral point as shown in FIG. 9B, it can be estimated that the decrease in nasal skin temperature as shown in FIG. 9B is caused by stress. ..

Here, it is difficult to estimate the thermal sensation based on the temperature range of the outside air temperature. Specifically, for example, even if the outside temperature is the same, due to the difference in the amount of clothing (for example, when wearing only a T-shirt or wearing thick down jacket, etc.), it is natural that a person feels warm and cold. Since it is different, it is difficult to estimate the thermal sensation depending on the temperature range of the outside air temperature. Therefore, it is important to judge based on the thermal sensation of a person, not simply the outside temperature.

Note that the degree of stress may be divided into stages according to the temperature range of the nasal skin temperature that has changed, as in the case of drowsiness in FIG. It can be judged that a strong stress is being felt when the temperature fluctuation range of the nose skin temperature is large, and when the temperature fluctuation range of the nose skin temperature is small, it can be classified as feeling light stress. The variation temperature range and the stress level may be appropriately determined depending on the difference in the temperature variation range.

Next, the configuration of the human state estimation device 113 according to the present embodiment will be described.

FIG. 10 is a block diagram showing functions of the human state estimation device 113 according to this embodiment. The human state estimation device 113 can also be referred to as a human state estimation system.

The human state estimation device 113 includes a thermal sensation estimation unit 108, a thermal image sensor 101, and a control unit 112. Further, the human state estimation device 113 is connected to the speaker 114.

Since the thermal sensation estimation unit 108 and the thermal image sensor 101 are the same as the functional blocks having the same names, the description thereof will be omitted.

The control unit 112 includes a physiological amount acquisition unit 109A, a human state estimation unit 110, and a human state estimation execution determination unit 111.

When it is determined that the thermal sensation index acquired by the thermal sensation estimation unit 108 is within a predetermined range, the physiological amount acquisition unit 109A acquires a physiological amount in which the activity of the human autonomic nerve is reflected. Is. The physiological amount acquisition unit 109A acquires the nose skin temperature as the physiological amount by processing the thermal image acquired by the thermal image sensor 101 using the image processing unit 109.

The human state estimation unit 110 is a processing unit that estimates the human state based on the physiological amount acquired by the physiological amount acquisition unit 109A. Specifically, the human state estimation unit 110 is a processing unit that estimates the degree of stress or drowsiness that is a human state from the nose skin temperature calculated by the physiological amount acquisition unit 109A.

The human state estimation execution determination unit 111 is a processing unit that determines whether or not the thermal sensation index acquired by the thermal sensation estimation unit 108 is within a predetermined range of the specified range. This determination is used to determine whether the human state estimation unit 110 estimates the human state. The human state estimation execution determination unit 111 is connected to the thermal sensation estimation control unit 106 (PMV calculation unit 106b) of the thermal sensation estimation unit 108, and the result of the thermal sensation estimation of the person 102 is the human state estimation execution determination. It is input to the section 111. In addition, the human state estimation execution determination unit 111 is connected to the physiological amount acquisition unit 109A. In addition, the human state estimation execution determination unit 111 corresponds to a determination unit.

The human state estimation execution determination unit 111 may be connected to the human state estimation unit 110. In this case, the human state estimation execution determination unit 111 may not be connected to the physiological amount acquisition unit 109A. In this case (when the human state estimation execution determination unit 111 is not connected to the physiological amount acquisition unit 109A but is connected to the human state estimation unit 110), the physiological amount acquisition unit 109A always acquires the physiological amount of the person. Perform processing to On the other hand, the human state estimation unit 110 determines the physiological amount at the timing when the human state estimation execution determination unit 111 determines that “the thermal sensation index acquired by the thermal sensation estimation unit 108 is within a predetermined range of the specified range”. The state of a person is estimated using the physiological amount acquired by the acquisition unit 109A. That is, the human state estimation unit 110 does not determine that the human state estimation execution determination unit 111 determines that “the thermal sensation index acquired by the thermal sensation estimation unit 108 is within a predetermined range of the specified range”. In addition, the physiological amount acquired by the physiological amount acquisition unit 109A is not used for estimating the human state.

The speaker 114 is an output device that notifies the person 102 according to the human state estimation result by the human state estimation unit 110. The speaker 114 may be a part of the human state estimation device 113.

The human state estimation device 113 may be realized as one device by accommodating each of the above functional blocks in one housing, or each of the above functional blocks may be arranged in a distributed manner. It may be realized by transmitting and receiving information via a communication line.

Hereinafter, the flow of processing of the human state estimation device 113 will be described.

FIG. 11 is a flow chart showing a human state estimation method by human state estimation device 113 in the present embodiment.

In step S101, the data obtained by the sensors (camera 103, glove thermometer 104a, thermometer 104b, anemometer 105, hygrometer 107) are input to the PMV calculation unit 106b to estimate the thermal sensation of the person 102. It

In step S102, the thermal sensation of the person 102 estimated in step S101 is input to the human state estimation execution determination unit 111, and the human state estimation implementation determination unit 111 determines the thermal sensation. Specifically, the human state estimation execution determination unit 111 determines whether the thermal sensation is within a predetermined range. The predetermined range may be, for example, a range in which the thermal sensation is −2 or more and +2 or less, and this case will be described below. The predetermined range may be a part of the specified range that includes a thermal neutral point related to the thermal sensation. In addition, the predetermined range may be a part of the specified range that does not include the hottest and coldest points and the coldest and coldest points.

When it is determined in step S102 that the thermal sensation is greater than +2 or less than -2, the process proceeds to step S101, and the PMV calculation unit 106b estimates the thermal sensation of the person 102 again. Done. In this case, the processes of steps S103 and S104 described below are not performed. When it is determined that the thermal sensation is greater than +2 or less than −2, that is, the person 102 having the thermal sensation “very hot” or “very cold” is known to have less drowsiness. Therefore, in this case, the human state estimation unit 110 may estimate the drowsiness level as 1.

When it is determined in step S102 that the thermal sensation is within the range of −2 to +2, the process proceeds to step S103.

In step S103, the human state estimation unit 110 estimates a drowsiness level as a human state based on the nose skin temperature acquired by the physiological amount acquisition unit 109A based on the thermal image acquired by the thermal image sensor 101. Specifically, the human state estimation unit 110 estimates the degree of drowsiness of the person 102 based on the rise width of the nose skin temperature over time. For example, when the nose skin temperature rises by 1° C. within the time range in which the nose skin temperature is measured (for example, the time range on the horizontal axis shown in FIG. 7A etc.), the human state estimation unit 110 sets the drowsiness level. Judge as 2. Further, when the nose skin temperature rises by 2° C. within the above time range, the drowsiness level is determined to be 4.

Further, the estimated sleepiness level is determined, and the following processing is performed according to the determination result.

That is, when it is determined in step S103 that the drowsiness level is 1, the process proceeds to step S101, and then the series of processes shown in the flowchart is performed again. This is because, in this case, the drowsiness level does not interfere with the driving of the automobile by the person 102, and it is considered that it is not necessary to notify the person 102.

On the other hand, if it is determined in step S103 that the drowsiness level is 2 or more, the person 102 is notified. In this case, the drowsiness level is considered to interfere with the driving of the automobile by the person 102, and therefore the above notification is performed to notify the person 102 of the fact. The notification is performed, for example, by informing the person 102 that he is getting sleepy by the speaker 114, or by prompting him to take a rest. After the notification, the process proceeds to step S101, and the series of processes shown in this flowchart is performed again.

When estimating the degree of stress as the human state, the process by the human state estimating unit 110 in step S103 is different. Specifically, the human state estimation unit 110 estimates the degree of stress as a human state based on the nose skin temperature acquired by the physiological amount acquisition unit 109A based on the thermal image acquired by the thermal image sensor 101. .. Specifically, the human state estimation unit 110 estimates the degree of stress on the person 102 based on the fall width of the nose skin temperature over time. Other processing is the same as above.

As described above, it is possible to determine whether the change in the skin temperature is due to a person's thermal sensation or drowsiness, so that it is possible to provide a highly accurate drowsiness degree estimation unit. Of course, the same applies to the case where the human state is the degree of stress, and by doing the same, it is possible to provide a highly accurate stress degree estimating means. That is, it becomes possible to provide a highly accurate human state estimation means. In addition, when detecting the degree of drowsiness, for example, when the thermal sensation is greater than +2 or less than -2, that is, when the person feels heat or cold, the person is less likely to feel drowsiness. Since unnecessary sleepiness degree estimation in the control unit 112 can be omitted, there is also an effect that the processing load can be reduced and the energy consumption can be reduced.

Although it has been described that the human state estimation unit 110 informs the person 102 through the speaker 114 when the drowsiness level is 2 or more as a result of estimating the drowsiness, other means may be used to inform the person 102, for example, a sheet. The user may be notified by urging the user to wake up by tightening the belt, or may be displayed on a display or the like, and the method is not particularly limited.

Further, although it has been described that the physiological amount acquisition unit 109A (image processing unit 109) calculates the nose skin temperature from the thermal image taken by the thermal image sensor 101, this is not limited to the nose skin temperature. It suffices that it is a region corresponding to a similar peripheral region, for example, the back of the hand or the earlobe. However, when the human state is estimated in the automobile 100, the lower the body of the person 102 (in other words, the closer to the foot) is, the more likely it is to be affected by solar radiation. It is preferable to use the above, and by measuring at the nose or earlobe, it is possible to estimate the human state that is not easily disturbed by solar radiation.

Furthermore, in the human state estimation execution determination unit 111, the upper and lower limits of the range of the thermal sensation obtained by the PMV calculation unit 106b are set to +2 and −2, but this value may be different from these values. Well, for example, the human state may be estimated when the thermal sensation is +1 or less and −1 or more. If the range including the thermal neutral point is narrow, the human state can be estimated with higher accuracy. Of course, a decimal value such as +1.5 or −1.5 may be used as the threshold value instead of an integer.

In the above description, the human thermal sensation is automatically estimated by the thermal sensation estimation unit 108. However, for example, the human 102 may directly input the thermal sensation of the person 102. As long as the human state estimation execution determination unit 111 can be provided with a value that can determine the thermal sensation of the person 102, the means is not limited.

Further, when the nose skin temperature is calculated by the physiological amount acquisition unit 109A (image processing unit 109), it is necessary to specify the nose from the thermal image acquired by the thermal image sensor 101. An example will be described. FIG. 12A is an explanatory diagram of nose skin temperature measurement according to the present embodiment. FIG. 12B is an explanatory diagram of nostril temperature fluctuations in nose skin temperature measurement according to the present embodiment.

FIG. 12A is a schematic diagram of a person's nose, and the temperature of the nostril changes with breathing as shown in FIG. 12B in accordance with the respiratory cycle. This temperature fluctuation is due to the temperature rise due to the warming of the nostril part by exhaling the warmed breath when exhaling from the nose due to breathing, and the heat of the nostril part due to inhaling the outside air when breathing in. This is due to the temperature drop due to being deprived. Therefore, the cycle of this temperature fluctuation is approximately equal to the respiratory cycle (usually about 0.2 to 0.3 Hz). Therefore, in the image processing unit 109, if there are two parts that change in a cycle of about 0.2 to 0.3 Hz, it is estimated that the parts are nostril parts. When the nostril part is known, the nose part to be measured can be easily specified from that position. From the above, it becomes possible to specify the nose.

Further, when the nostril portion cannot be specified, the display 102 or the like may notify the person 102 that the nostril portion cannot be extracted and the human state cannot be estimated. At the same time, you may encourage them to wear glasses or the like. Since the glasses do not normally transmit infrared rays, when the person wearing the glasses is photographed by the thermal image sensor 101, the temperature of the glasses is not the temperature of the eyes but the temperature of the glasses itself, and the temperature is closer to the ambient temperature than the skin temperature. , It becomes easier to detect the position. The position of the nose can be accurately estimated by estimating and measuring the position of the nose from the detected eye position.

In addition, you may encourage people to take a deep breath through a display or the like. By taking a deep breath, the amplitude of the temperature of the nostril shown in FIG. 12B becomes large, and it becomes easier to extract the position of the nostril. Of course, the nose may be specified by a method other than this, for example, the face may be extracted and the nose may be estimated from the contour thereof, and the method is not limited.

When measuring the skin temperature of the nose for estimating the human condition, the measurement timing may be synchronized with the phase of respiration. For example, as shown by the arrow in FIG. 12B, the nose skin temperature may be measured at the timing when the temperature of the nostril becomes highest for each breath. The nose skin temperature is affected by the change in the nostril temperature due to breathing. Therefore, by measuring the skin temperature of the nose in synchronization with the phase of breathing, it is possible to perform precise measurement with little variation. In addition, here, the measurement is performed at the timing when the temperature of the nostril becomes highest for each breath, but of course, it may be another phase, the timing when the temperature of the nostril is the lowest, or any other phase. No, it is not limiting here.

Further, in order to measure the skin temperature of the nose, the thermal image sensor 101 is used here, but of course this does not limit the means, and any device capable of measuring the skin temperature may be used. A pyroelectric sensor or a monocular infrared sensor (bolometer sensor, thermopile sensor, or the like) may be used, and the means is not limited.

The human condition may be obtained from the skin blood flow rate. A method of obtaining the human state from the skin blood flow will be described below.

Since there is a correlation that blood flow in the peripheral part (particularly the nose) increases when the drowsiness as a human condition increases, the human condition can be estimated from the skin blood flow based on this correlation. There are various methods for measuring the skin blood flow, but for example, a camera receives light of a specific wavelength (for example, infrared light) and the amount of hemoglobin measured based on the received light is used. The skin blood flow can be calculated.

Further, the human condition may be obtained from blood pressure. A method of obtaining the human state from blood pressure will be described below with reference to FIGS. 12C and 12D.

Since there is a correlation that the blood pressure decreases when the drowsiness as the human state becomes strong, the human state can be estimated from the skin blood flow based on this correlation. The blood pressure may be continuously obtained with a cuff or the like, or may be obtained from the pulse wave transit time. The pulse wave propagation time is the time required for blood flow from the heart to reach a predetermined end portion.

There is a correlation between the blood pressure and the pulse wave transit time that the pulse wave transit time increases as the blood pressure decreases. Therefore, based on these correlations, the human state can be estimated from the pulse wave transit time through the blood pressure.

There may be various methods for measuring the pulse wave transit time. For example, as shown in FIG. 12C, a moving image including a person's face and parts other than the face (neck, hands, etc.) is photographed by a camera, and from this moving image, a part P1 included in the face and a part P2 other than the face are captured. It can be obtained from the amount of deviation T of the peak time of the pulse wave in (FIG. 12D). The deviation amount T is a value that can fluctuate in about 0.2 msec.

Further, instead of using the face and the part other than the face, for example, it may be calculated from the deviation amount of the peak time of the pulse wave of two different parts (for example, jaw and forehead) in the face. In addition to this, the vibration of the heart is measured with a millimeter wave sensor, etc., and the pulse wave of the face is detected from the subtle color change of the face image taken from the camera, and the peak of the vibration of the heart The blood pressure fluctuation may be estimated using the time shift amount as the pulse wave propagation time.

As described above, according to the human state estimation method according to the present embodiment, it is possible to estimate the human state with higher accuracy by reducing erroneous detection during human state estimation. Specifically, in the human state estimation method according to one aspect of the present invention, the human state is estimated from the physiological amount under the condition that the acquired thermal sensation information is within the predetermined range. Therefore, erroneous detection at the time of human state estimation can be reduced as compared with the case of estimating the human state regardless of whether or not this condition is satisfied. Therefore, the human state estimation method can improve the accuracy of human state estimation. Furthermore, by reducing the above-mentioned erroneous detection, it is not necessary to perform the detection process again, and thus it is possible to obtain the effects of reducing the processing amount and the processing load and the power consumption.

In addition, when a person's thermal sensation is within a range relatively close to a thermal neutral point at which the person does not feel hot or cold, the human state is estimated. When a person feels hot or warm, the skin temperature rises by promoting heat dissipation from the body. On the other hand, when a person feels cold or cool, suppressing heat radiation from the body lowers the skin temperature. In other words, when the person's thermal sensation is within a range relatively close to the thermal neutral point, there is no or relatively little effect of promoting or suppressing heat radiation from the human body. Therefore, by estimating the human state from the physiological amount in such a case, it is possible to reduce the influence of promotion or suppression of heat radiation from the human body, which is included in the estimation result, and prevent the false detection and the estimation accuracy. Contribute to the improvement of.

In addition, when a person's thermal sensation falls within a range excluding the time when he/she feels very hot or very cold, the human state is estimated. When a person feels very hot, heat dissipation from the human body is greatly promoted. Further, when a person feels very cold, heat radiation from the human body is greatly suppressed. By excluding such a case from the case of estimating the human state, it is possible to reduce the influence of promotion or suppression of heat radiation from the human body, which is included in the estimation result, and prevent the false detection and the estimation accuracy. Contribute to the improvement of.

Further, when estimating the degree of drowsiness of a person based on the skin temperature of the nose of the person, it is possible to reduce the influence of promotion or suppression of heat radiation from the body of the person included in the estimation result. Further, by using the skin temperature of the nose, it is possible to reduce the influence of disturbance even when there is disturbance.

Further, the degree of drowsiness of a person can be specifically estimated based on the skin temperature of the nose of the person.

Further, the estimation by the PMV makes it possible to more accurately estimate the thermal sensation of a person from the temperature, humidity, air flow, radiation, clothing amount, and activity amount, which are the six heat and heat factors.

Moreover, the degree of stress of a person can be specifically estimated based on the skin temperature of the nose of the person.

Further, the degree of drowsiness of a person can be specifically estimated based on the skin blood flow, blood pressure, or pulse wave transit time of the person.

(Embodiment 2)
An example of the case where the human state estimating device according to the present embodiment is installed in an automobile will be described with reference to FIGS. 13 to 21.

FIG. 13A is a conceptual diagram of an automobile 200 in which the human state estimating device according to the present embodiment is installed. FIG. 13B is a schematic diagram of a person 202 wearing the human state estimating device according to the present embodiment, as viewed from the front. The person 202 shown in FIG. 13B corresponds to the person 202 who is in the automobile 200 shown in FIG. 13A.

As shown in FIG. 13B, the person 202 attaches the skin temperature pulse wave sensor 203 to the earlobe. The skin temperature pulse wave sensor 203 is a combination of a temperature sensor and a pulse wave sensor, and can simultaneously measure the skin temperature and the pulse wave. In the present embodiment, an example is shown in which the skin temperature of the earlobe is used for estimating the thermal sensation and the pulse wave is used for estimating the human state.

Hereinafter, a method of estimating a human state using a pulse wave will be described.

FIG. 14A is an explanatory diagram showing an example of the measured pulse wave in the present embodiment. FIG. 14B is an explanatory diagram of time-series fluctuations of heartbeat intervals obtained from the pulse wave shown in FIG. 14A. FIG. 15A is an explanatory diagram showing a frequency component of a heartbeat interval in the present embodiment, which has a small HF component. FIG. 15B is an explanatory diagram showing frequency components of the heartbeat interval in the present embodiment, which have many HF components.

FIG. 14A shows an example of the pulse wave measured by the skin temperature pulse wave sensor 203. The pulse wave is a waveform of a change in blood pressure or volume in the peripheral vascular system associated with the pulsation of the heart as a waveform from the body surface, and typically shows a sawtooth waveform as shown in FIG. 14A. When heartbeat intervals are extracted from the waveform of this pulse wave and arranged in time series, fluctuations in the heartbeat intervals (changes in heartbeat intervals) over time are observed, as shown in FIG. 14B.

It is known that there are mainly two types of causes of fluctuations in heartbeat intervals, one is blood pressure fluctuation and the other is respiratory fluctuation. The pulsation of the heart is controlled by the center of the brain via the autonomic nerves, and is controlled by the two systems of the sympathetic nerve and the parasympathetic nerve. When the heartbeat is accelerated, the sympathetic nerve is enhanced and the heartbeat is changed. When slowed down, it stimulates parasympathetic nerves. The center of the brain determines whether to accelerate or slow the heartbeat from the state of the body, and the factors include blood pressure and respiration. Regarding blood pressure, it functions to accelerate the heartbeat to activate the activity of the heart when the blood pressure decreases, and to slow the heartbeat to suppress the activity of the heart when the blood pressure rises. With the extension of the lungs, the breathing also functions so that the heart rate becomes faster while inhaling and slows when exhaling.

Further, the blood pressure fluctuates at about 0.1 Hz, and this fluctuation is called a Mayer wave. The breathing is about 0.2 to 0.3 Hz at rest. Therefore, the heartbeat interval has a waveform with a cycle of about 0.2 to 0.3 Hz as shown in FIG. 14B or a waveform with a cycle of about 0.1 Hz.

Next, the speed (response speed) at which the sympathetic nerve and the parasympathetic nerve follow a command regarding the speed of the heartbeat from the center of the brain will be described. It is known that the parasympathetic nerve can follow the above command even with a fluctuation of about 0.2 to 0.3 Hz. On the other hand, it is known that the sympathetic nerve can follow the above command even if the command has a fluctuation of about 0.1 Hz, but cannot follow the command if the command has a high frequency fluctuation of about 0.2 to 0.3 Hz. Therefore, when the frequency component of the waveform of the heartbeat interval is obtained when the sympathetic nerve is enhanced, the frequency component of about 0.2 to 0.3 Hz becomes relatively low as shown in FIG. 15B, the frequency component of about 0.2 to 0.3 Hz increases as compared with the case of FIG. 15A as shown in FIG. 15B.

Therefore, the ratio LF/HF of the low frequency component (Low Frequency: LF) and the high frequency component (High Frequency: HF) can be used as an index indicating which of the sympathetic nerve and the parasympathetic nerve is dominant. That is, the sympathetic nerve is dominant when LF/HF is high, and the parasympathetic nerve is dominant when LF/HF is low. Therefore, when drowsiness or the like as a human state occurs and the parasympathetic nerve is enhanced, the HF component increases and the LF/HF decreases as shown in FIG. 15B. On the other hand, when stress or the like occurs and the sympathetic nerve is enhanced, the HF component decreases and the LF/HF increases as shown in FIG. 15A. Here, for example, LF is a frequency component of 0.15 Hz or less, and HF is a frequency component of 0.2 Hz to 0.3 Hz. These boundary values are examples and are not limited to the above values.

Using the above, it is possible to estimate the human state based on the autonomic nerve from the pulse wave detected by the skin temperature pulse wave sensor 203. Although the pulse wave is detected from the earlobe as the means for detecting the heartbeat interval here, the method is not limited to this. For example, a sensor may be provided on the steering wheel or the like to detect the pulse wave from the fingertip or the like. The electrocardiogram may be measured instead of the pulse wave and the heartbeat interval may be extracted from the electrocardiogram. The method is not limited as long as the heartbeat interval can be measured.

Next, a method of estimating thermal sensation using the skin temperature of a person's peripheral part will be described.

FIG. 16 is a correlation diagram showing an example of the correlation between earlobe skin temperature and thermal sensation according to the present embodiment.

When a person feels cold, the brain constricts blood vessels in the peripheral part to prevent the temperature drop in the deep part where many vital organs are important. Try to reduce the amount. When the amount of blood reaching the peripheral area decreases, the skin temperature in the peripheral area decreases.

On the other hand, when a person feels heat, the brain tries to expand the blood vessels in the peripheral part to increase the amount of blood flow reaching the peripheral part of the body in order to prevent the core body temperature from rising above a certain level. To do. When the amount of blood reaching the peripheral area increases, the skin temperature in the peripheral area increases.

As a result, there is a correlation between the peripheral skin temperature and the thermal sensation. For example, as shown in FIG. 16, there is a linear correlation between the peripheral skin temperature of the earlobe and the thermal sensation. Relationships arise. From this, it is possible to estimate the human thermal sensation by detecting the skin temperature in the peripheral portion. Here, the earlobe is used as the peripheral part, but other parts may of course be used, and other parts such as the palm part and the nose part may be used as long as they are the peripheral part. However, when the human state is estimated in the automobile 200, the lower the body of the person 202 is, the more likely it is to be affected by solar radiation. By measuring at the earlobe, it is possible to estimate the human condition that is less likely to be disturbed by solar radiation.

Further, as shown in FIG. 16, the fluctuation range of the peripheral skin temperature due to the fluctuation of the thermal sensation (-3 to +3) is as large as about 20° C. (about 15° C. to 35° C.), which affects the skin temperature. Since it is larger than the temperature fluctuation width (usually about 1 to 2° C.) due to the influence of stress or drowsiness, which is the factor of (3), the estimated thermal sensation is not significantly affected. Further, for example, the thermal sensation may be estimated not from the skin temperature of the peripheral part alone but from the difference between the skin temperature of the forehead part close to the skin temperature of the trunk part and the skin temperature of the peripheral part. By doing so, it is possible to reduce individual differences when estimating the thermal sensation.

Next, the configuration of the human state estimation device 213 according to the present embodiment will be described.

FIG. 17 is a block diagram showing functions of the human state estimation device 213 in this embodiment.

The human state estimation device 213 includes a skin temperature pulse wave sensor 203, a thermal sensation estimation unit 208, and a control unit 212.

The control unit 212 includes a human state estimation unit 210 and a human state estimation execution determination unit 211.

The human state estimation unit 210 is a processing unit that estimates the degree of stress or drowsiness that is a human state from the pulse wave acquired by the skin temperature pulse wave sensor 203. More specifically, the human state estimation unit 210 acquires the heartbeat interval of the person 202 obtained from the pulse wave as a physiological amount, and estimates the human state of the person 202 based on the acquired fluctuation of the heartbeat interval.

The human state estimation execution determination unit 211 is a processing unit that determines whether or not the human state estimation unit 210 performs human state estimation. The human state estimation execution determination unit 211 is connected to the thermal sensation estimation unit 208, and the result of the thermal sensation estimation of the person 202 is input to the human state estimation execution determination unit 211. The human state estimation execution determination unit 211 is connected to the human state estimation unit 210.

Hereinafter, the flow of processing of the human state estimation device 213 will be described.

FIG. 18 is a flow chart showing a human state estimating method by the human state estimating device 213 in the present embodiment.

In step S201, the skin temperature data obtained by the skin temperature pulse wave sensor 203 is input to the thermal sensation estimation unit 208 to estimate the thermal sensation of the person 202.

In step S202, the thermal sensation of the person 202 estimated in step S201 is input to the human state estimation execution determination unit 211, and the human state estimation execution determination unit 211 determines the thermal sensation.

When it is determined in step S201 that the thermal sensation is greater than +2 or less than −2, the process proceeds to step S201, and the thermal sensation of the person 202 by the thermal sensation estimation unit 208 is determined again. An estimate is made.

When it is determined in step S201 that the thermal sensation is within the range of −2 to +2, the process proceeds to step S203.

In step S203, the human state estimation unit 210 estimates the drowsiness level as a human state based on the pulse wave measured by the skin temperature pulse wave sensor 203. Further, the estimated sleepiness level is determined, and the following processing is performed according to the determination result.

That is, if it is determined in step S203 that the drowsiness level is 1, the process proceeds to step S201, and then the series of processes shown in this flowchart is performed again. This is because, in this case, the drowsiness level does not interfere with the driving of the automobile by the person 202, and it is considered that it is not necessary to notify the person 202.

On the other hand, when it is determined in step S203 that the drowsiness level is 2 or more, the person 202 is notified. In this case, it is considered that the drowsiness level hinders the person 202 from driving the vehicle, and therefore the above notification is performed to notify the person 202 of the fact. The notification is performed, for example, by informing the person 202 that he is getting sleepy by the speaker 114, or by urging him to take a rest. After the notification, the process proceeds to step S201, and the series of processes shown in this flowchart is performed again.

In the case of estimating the degree of stress as the human state, the same process as in the case of drowsiness is performed except that the determination criteria in the human state estimating unit 210 are different, and thus the description thereof will be omitted.

As described above, it is possible to determine whether the fluctuation of the pulse wave is due to a person's thermal sensation or drowsiness, so that it is possible to provide a highly accurate drowsiness degree estimation unit. Of course, the same applies to the case where the human state is the degree of stress, and by doing the same, it is possible to provide a highly accurate stress degree estimating means. That is, it becomes possible to provide a highly accurate human state estimation means. In addition, when detecting the degree of drowsiness, for example, when the thermal sensation is greater than +2 or less than -2, that is, when the person feels heat or cold, the person is less likely to feel drowsiness. Since unnecessary sleepiness degree estimation in the control unit 112 can be omitted, there is also an effect that the processing load can be reduced and the energy consumption can be reduced.

Although it has been described that the human state estimation unit 210 notifies the person 202 through the speaker 114 when the drowsiness level is 2 or more as a result of estimating the drowsiness, the person 202 may be notified by other means, for example, a sheet. The person 202 may be notified by urging the user to wake up by tightening the belt, or by displaying it on a display or the like, and the method is not particularly limited.

Further, in the human state estimation execution determination unit 211, the upper and lower limits of the range of the thermal sensation obtained by the thermal sensation estimation unit 208 are set to +2 and −2, but this value uses a different value from these. Alternatively, for example, the human state may be estimated when the thermal sensation is +1 or less and −1 or more. If the range including the thermal neutral point is narrow, the human state can be estimated with higher accuracy. Of course, a decimal value such as +1.5 or −1.5 may be used as the threshold value instead of an integer.

In the above description, the human thermal sensation is automatically estimated by the thermal sensation estimation unit 208, but the human 202 may directly input the thermal sensation of the person 202, for example. As long as the human state estimation execution determination unit 211 can be provided with a value that can determine the thermal sensation of the person 202, the means is not limited.

Further, although the human state is estimated from the pulse wave up to this point, a method of detecting the human state at a higher speed using the respiration described in the first embodiment together with the pulse wave will be described.

FIG. 19A is an explanatory diagram of the respiratory frequency according to the processing load reduction method using the respiratory component according to the present embodiment. FIG. 19B is a block diagram showing a wavelet transform according to the processing load reduction method using the respiratory component according to the present embodiment.

In the description of FIGS. 15A and 15B, it has been shown that the frequency characteristics of FIGS. 15A and 15B are obtained from the waveform of the heartbeat interval shown in FIG. 14B. Usually, Fourier transform or discrete Fourier transform or the like is used to obtain such frequency characteristics, and it is often executed by fast Fourier transform on a computer. However, when the Fourier transform of the pulse wave is performed, only one data is obtained in about 1 second, and therefore data of about several minutes is usually required. However, in order to analyze LF/HF for analysis of an autonomic nerve, it is important to detect how much the HF component is, and it is not always necessary to detect all frequency components.

Therefore, as shown in FIG. 19B, the respiratory waveform can be used as the physiological amount. That is, it is possible to estimate the human state based on the respiratory waveform in addition to the heartbeat interval. The respiratory waveform may be obtained from the temperature fluctuation of the nostril as described in the first embodiment, or other than that, for example, the expansion and contraction of the abdomen due to respiration may be detected from the tension of the seat belt, or the millimeter wave. For example, the position variation of the abdomen may be detected in a non-contact manner, and any means may be used. The frequency of the respiratory frequency component of the person 202 is obtained by performing the respiratory cycle analysis 1902 on the obtained respiratory waveform, for example, by analyzing the respiratory cycle from the interval of the peak values. Next, with respect to the obtained pulse wave waveform, the intensity of the respiratory frequency component obtained from the respiratory waveform is obtained by the wavelet transform 1901.

In this way, by extracting the respiratory component intensity of the pulse wave based on the respiratory waveform, it is possible to determine how much the respiratory frequency component is included in the pulse wave in less than 1 minute. Since the amount of calculation by this method is smaller than the amount of calculation by Fourier transform, there is also an effect that the processing load of the human state estimation device 213 can be reduced and the energy consumption can be reduced.

A human state estimation method including a respiratory sensor will be described.

FIG. 20 is a flowchart showing a human state estimating method to which the processing load reducing method using the respiratory component according to the present embodiment is applied.

In step S<b>203</b>A, a respiratory waveform obtained from the respiratory sensor is input as an input for estimating the human state by the human state estimating unit 210, in addition to the pulse wave obtained by the skin temperature pulse wave sensor 203.

Other than that, it is the same as the processing step of the same name in FIG. 18, and thus detailed description will be omitted.

13A and 13B, the skin temperature pulse wave sensor 203 is attached to the left ear of the person 202. However, when the automobile 200 is a right-hand drive vehicle, the one attached to the left ear of the person is a disturbance of sunlight or the like. This is preferable because it is less likely to be affected and also when the pulse wave is optically detected, it is less likely to be affected by disturbance such as solar radiation. Of course, if it is a left-hand drive vehicle, it is preferable to attach it to the right ear.

Further, in the present embodiment, the method of detecting the pulse wave from the earlobe is shown, but when detecting the human state of a person who is working on a personal computer or the like in the office or the like from the pulse wave, the earlobe does not Alternatively, the pulse wave may be detected through a mouse that is an input device of a personal computer. Since the part of the mouse on which the finger is placed is almost fixed, by optically reading the pulse wave at that part, the human condition can be detected without actively wearing the sensor. Other than that, as a wearable sensor, for example, an element for detecting a person's pulse wave may be attached to a shirt or the like, and the pulse wave waveform may be wirelessly transferred to a smart phone or the like and processed using a cloud .. By doing so, the human state can be detected even when the person is moving. Further, for example, by treating and analyzing the human condition of a large number of people as big data, it is possible to specify a place or a time zone in which a lot of stress is applied. Therefore, a place or a time zone in which an accident or the like is likely to occur is extracted. You can

Next, a method of reducing the individual difference in the human state will be described.

FIG. 21 is a block diagram showing the function of correcting the human condition determination result in the present embodiment.

When the human state estimation unit 210 analyzes the pulse wave to determine the human state, the pulse wave analysis unit 250 performs the above-described Fourier transform and wavelet transform to obtain LF/HF, and the human state determination unit 251 determines the human state from the result. It is determined at which level 1 to 5 the degree of drowsiness as a state is. A speaker or the like outputs the result of the above judgment as voice.

The person 202 learns by listening to the result of the above determination. Then, when the person 202 feels that the state and the degree of drowsiness that he or she feels are different, the person 202 inputs the degree of drowsiness that he or she feels through the correction value input unit 253. The human state correction unit 252 recognizes and stores an individual difference from the difference between the degree of drowsiness determined by the human state determination unit 251 and the degree of drowsiness of the person received by the correction value input unit 253. After that, by correcting the result determined by the human condition determining unit 251, using the value stored in the human condition correcting unit 252, the corrected human condition for each person 202 is estimated. .. By doing so, it is possible to provide the human state estimation device 213 having a small individual difference.

Further, although the human state detection unit using the skin temperature and the pulse wave has been shown so far, the present invention is not limited to these, and other than the skin temperature and the pulse wave, for example, the skin blood flow may be detected. The means is not limited as long as it is a physiological amount controlled by an autonomic nerve such as the line of sight, blink, cerebral blood flow, or brain wave.

As described above, according to the human state estimation method according to the present embodiment, it is possible to estimate the human state with higher accuracy by reducing erroneous detection during human state estimation. Specifically, in the human state estimation method according to one aspect of the present invention, the human state is estimated from the physiological amount under the condition that the acquired thermal sensation information is within the predetermined range. Therefore, erroneous detection at the time of human state estimation can be reduced as compared with the case of estimating the human state regardless of whether or not this condition is satisfied. Therefore, the human state estimation method can improve the accuracy of human state estimation. Furthermore, by reducing the above-mentioned erroneous detection, it is not necessary to perform the detection process again, and thus it is possible to obtain the effects of reducing the processing amount and the processing load and the power consumption.

Further, it is possible to specifically estimate the human state based on the heartbeat interval of the person.

In addition, the human state can be specifically estimated based on the respiratory waveform of the person. Further, it is possible to contribute to reduction of processing load and speeding up of processing.

In addition, the human condition can be specifically estimated based on the skin temperature and the pulse wave of the human earlobe. The earlobe is characterized in that it is easy to measure the pulse wave. Therefore, the pulse wave is acquired by the earlobe and the skin temperature is also acquired, so that the information necessary for estimating the human state can be collectively acquired from the earlobe.

In addition, in each of the above-described embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the human state estimation device of each of the above-described embodiments is the following program.

That is, this program is a human state estimation method for estimating a human state that is a human state in a computer, and acquires a thermal sensation index that is an index of the estimated thermal sensation of the person within a specified range. Then, it is determined whether or not the acquired thermal sensation index is within a predetermined range of the specified range, and acquired when it is determined that the acquired thermal sensation index is within the predetermined range. The human state estimation method for estimating the human state is executed based on the physiological amount that reflects the activity of the human autonomic nerve.

Although the human state estimation device and the like according to one or more aspects have been described above based on the embodiment, the present invention is not limited to this embodiment. As long as it does not depart from the gist of the present invention, various modifications that a person skilled in the art can think of in the present embodiment, and forms constructed by combining the components in different embodiments are also within the scope of one or more aspects. May be included within.

For example, the following cases are also included in the present invention.

(1) In the above embodiment, the human state estimation method including at least the sensor, the thermal sensation estimation unit, and the control unit has been described. However, a part of the components of the sensor, the thermal sensation estimation unit, the control unit, and the human state estimation method can be separately configured as software. In this case, the main body that processes the software may be a calculation unit of a human state estimation method, a calculation unit included in a PC (personal computer), a smart phone, or the like. It may be a cloud server or the like connected to the device via a network.

Further, the arrangement or configuration of each device is not limited to the arrangement or configuration of devices as shown in FIG. A part or all of each sensor (camera, thermometer, glove thermometer, anemometer, hygrometer) may be incorporated in an integrated module, or may be arranged alone (as a separate body). Further, as shown in FIG. 10, the thermal sensation estimation unit 108 may be incorporated as an integrated configuration. Further, the thermal sensation estimation control unit 106 (the process included therein) may be separately provided as software. Also, the thermal sensation estimation unit 108 and the control unit 112 (the processing included in them) may be provided as integrated software. Further, the thermal image sensor 101, the thermal sensation estimation control unit 106, and the control unit 112 may be provided as an integrated module. The arrangement or configuration of each device is not limited to this, and any form of providing each configuration and providing it as software or a module is also included.

(2) Each of the above devices is specifically a computer system including a microprocessor, a ROM, a RAM, a hard disk unit, a display unit, a keyboard, a mouse, and the like. A computer program is stored in the RAM or the hard disk unit. Each device achieves its function by the microprocessor operating according to the computer program. Here, the computer program is configured by combining a plurality of instruction codes indicating instructions to the computer in order to achieve a predetermined function.

(3) Part or all of the constituent elements of each of the above devices may be composed of one system LSI (Large Scale Integration). The system LSI is a super-multifunctional LSI manufactured by integrating a plurality of constituent parts on one chip, and specifically, is a computer system including a microprocessor, ROM, RAM and the like. .. A computer program is stored in the RAM. The system LSI achieves its function by the microprocessor operating according to the computer program.

(4) Part or all of the constituent elements of each of the above devices may be configured with an IC card that can be attached to and detached from each device or a single module. The IC card or the module is a computer system including a microprocessor, ROM, RAM and the like. The IC card or the module may include the above super-multifunctional LSI. The IC card or the module achieves its function by the microprocessor operating according to the computer program. This IC card or this module may be tamper resistant.

(5) The present disclosure may be the methods described above. Further, it may be a computer program that realizes these methods by a computer, or may be a digital signal including the computer program.

The present disclosure also discloses a computer-readable recording medium that can read the computer program or the digital signal, for example, a flexible disk, a hard disk, a CD-ROM, a MO, a DVD, a DVD-ROM, a DVD-RAM, a BD (Blu-ray). (Registered trademark) Disc), or may be recorded in a semiconductor memory or the like. Further, the digital signal recorded on these recording media may be used.

Further, the present disclosure may transmit the computer program or the digital signal via an electric communication line, a wireless or wired communication line, a network represented by the Internet, a data broadcast, or the like.

Further, the present disclosure may be a computer system including a microprocessor and a memory, wherein the memory stores the computer program and the microprocessor operates according to the computer program.

In addition, by recording the program or the digital signal in the recording medium and transferring the program or by transferring the program or the digital signal via the network or the like, an independent computer system is implemented. You may.

(6) The above embodiment and the above modifications may be combined.

INDUSTRIAL APPLICABILITY The present disclosure can be used for a human state estimation device, and particularly, can be used for a human state estimation device that estimates a human state in a driver's seat of an automobile or a train, an office, or the like.

100, 200 automobile 101 thermal image sensor 102, 202 person 103 camera 104a glove thermometer 104b thermometer 105, 105a anemometer 105b louver 106 thermal sensation estimation control unit 106a clothing amount estimation unit 106b PMV calculation unit 107 hygrometer 108, 208 Thermal sensation estimation unit 109 Image processing unit 109A Physiological amount acquisition unit 110, 210 Human state estimation unit 111, 211 Human state estimation execution determination unit 112, 212 Control unit 113, 213 Human state estimation device 114 Speaker 203 Skin temperature pulse wave sensor 250 pulse wave analysis unit 251 human condition determination unit 252 human condition correction unit 253 correction value input unit P1, P2 parts

Claims (14)

A human state estimation method for estimating a human state, which is a human state, comprising:
At least, it was estimated based on the skin temperature in the peripheral part of the person, or (1) the measurement data of the environment around the person and (2) the area ratio or the temperature difference between the clothing part and the exposed part of the person . A thermal sensation index, which is a thermal sensation index obtained by indexing thermal sensation within a specified range,
The acquired thermal sensation index determines whether or not it is within a predetermined range of the specified range,
Estimating the human state based on the physiological amount acquired when the acquired thermal sensation index is determined to be within the predetermined range, and which reflects the activity of the human autonomic nerve. Human state estimation method. The human state estimation method according to claim 1, wherein the predetermined range is a part of the specified range that includes a thermal neutral point related to thermal sensation. The said predetermined range is a part of range which does not include the point which shows the hottest as a thermal sensation, and the point which shows the coldest as a thermal sensation among said specified range. Human state estimation method. The physiological amount is the nose skin temperature of the person,
The human state estimation method according to claim 1, wherein the human state includes a degree of drowsiness of the person. In the human state estimation method,
Obtaining the thermal sensation index and the nose skin temperature,
At the time of the determination, it is determined whether or not the acquired thermal sensation index is within the predetermined range,
The human state estimation method according to claim 4, wherein, when estimating the human state, the degree of sleepiness of the person is estimated based on the obtained rise width of the nose skin temperature over time. The physiological amount includes the heartbeat interval of the person,
In the human state estimation method,
The human condition estimation method according to claim 1, wherein the human condition is estimated based on the fluctuation of the heartbeat interval as the acquired physiological amount. The physiological amount includes the respiratory waveform of the person,
In the human state estimation method,
The human state estimation method according to claim 6, wherein the human state is estimated further based on the acquired respiratory waveform as the physiological amount. Obtaining the skin temperature of the earlobe of the person,
When acquiring the thermal sensation index, using the correlation between the skin temperature of the earlobe and the thermal sensation index, the thermal sensation index estimated from the acquired skin temperature of the earlobe is acquired. ,
When acquiring the physiological amount, the pulse wave measured from the earlobe of the person is acquired as the physiological amount,
The human state estimation method according to claim 1, wherein when estimating the human state, the human state is estimated based on frequency analysis of the acquired pulse wave. The human condition estimation method according to claim 1, wherein the human condition estimation method acquires the thermal sensation index estimated by PMV (Predicted Mean Vote). The specified range is expressed by a 7-level rating scale of PMV,
The human state estimation method according to claim 9, wherein the predetermined range is a range in which the PMV value in the specified range is −2 or more and +2 or less. The physiological amount is the nose skin temperature of the person,
The human condition includes the degree of stress of the person,
In the human state estimation method,
Obtaining the thermal sensation index and the nose skin temperature,
At the time of the determination, it is determined whether or not the acquired thermal sensation index is within the predetermined range,
When estimating the said human state, the degree of stress of the said person is estimated based on the fall width with time passage of the acquired said nose skin temperature. Human state estimation method. The physiological amount is skin blood flow of the person, blood pressure, or pulse wave transit time,
The human state estimation method according to claim 1, wherein the human state includes a degree of drowsiness of the person. A human state estimation system for estimating a human state, which is a human state,
At least, it was estimated based on the skin temperature in the peripheral part of the person, or (1) the measurement data of the environment around the person and (2) the area ratio or the temperature difference between the clothing part and the exposed part of the person. It is a thermal sensation index,
A thermal sensation estimation unit that acquires a thermal sensation index in which thermal sensation is indexed within a specified range,
The determination unit that determines whether the acquired thermal sensation index is within a predetermined range of the specified range,
Estimating the human state based on the physiological amount acquired when the acquired thermal sensation index is determined to be within the predetermined range, and which reflects the activity of the human autonomic nerve. A human state estimation system including a human state estimation unit. A program that causes a computer to execute the human state estimation method according to any one of claims 1 to 12.