JP2019111092A - Biological state estimation device, method, and program - Google Patents

Abstract

To acquire a reliable estimation result by preventing a wrong estimation result from being output as a correct one when a measurement state of an object person changes by a movement of the object person.SOLUTION: A biological state estimation device 20 takes in facial image data of a driver output from a camera 40 at a fixed acquisition period, and takes in electrocardiographic data transmitted from a wearable terminal 50 at the acquisition period. First to third drowsiness estimation units 213-215 detect a feature amount indicating vestibulo-ocular reflex (VOR), a feature amount indicating a blinking movement of eyes or an opening/closing movement of eyelids, and a feature amount indicating a facial expression of the driver from the facial image data, and estimate the user's drowsiness based on these feature amounts respectively. Also, a fourth drowsiness estimation unit 216 detects a feature amount related to a human autonomic nerve activity from the electrocardiographic data, and estimates the user's drowsiness based on this feature amount.SELECTED DRAWING: Figure 4

Description

  An embodiment of the present invention relates to, for example, a biological condition estimation apparatus, method and program for estimating a biological condition of a driver of a vehicle.

  Conventionally, a technology has been proposed for estimating the living body condition of the driver of the vehicle, particularly the living body condition that may interfere with driving such as sleepiness, based on a plurality of feature quantities detected from face images, biological information, etc. ing. For example, Patent Document 1 describes a technique for estimating the presence or absence of drowsiness from a face image or brain waves of a driver, and outputting an alarm or the like when it is estimated that the driver feels drowsiness.

  Further, according to Patent Document 1, the presence or absence and the type of a mounted article such as glasses are detected from the driver's face image, weights are set according to the detection result, and the drowsiness estimated value based on the face image is corrected. It is considered that the driver is unlikely to feel drowsiness until a preset time elapses from the time of start of driving, and the drowsiness estimation value in the driving start period is weighted to reduce the estimation value. The technology is also described. Adopting these techniques can increase the reliability of the drowsiness estimates.

International Publication No. 2011/111206

  However, according to the technology described in Patent Document 1, the weight is set at the start of driving based on the presence and the type of the wearing product in the driver's face, and the value is maintained thereafter or until a predetermined time elapses from the start of driving. The weight is set only for the period of. That is, the setting of the weight for the estimated value of drowsiness is limitedly performed only at the driving start time or the driving start period.

  For this reason, in the technology described in Patent Document 1, for example, the driver's face image and the acquisition condition of the biological information change due to the driving operation such as steering wheel operation and the change in posture when checking the surroundings during driving. In such a case, although the driver's drowsiness may not be accurately estimated, the drowsiness may be determined as the estimation result being correct.

  The present invention has been made focusing on the above circumstances, and as one aspect, when the measurement state of the subject changes due to the movement of the subject, the erroneous estimation result is prevented from being output as the correct one, and the trust is provided. It is intended to provide a technology that can obtain highly accurate estimation results.

  In order to solve the above problems, a first aspect of the biological condition estimation apparatus, method or program according to the present invention acquires measurement information including at least one of captured image information and vital information of a subject from a sensor unit A feature parameter related to a biological condition affecting the work of the subject is detected from the measurement information, and the biological condition is estimated based on the feature parameter. At the same time, a change in the measurement state of the measurement information that occurs due to the movement of the subject including the work and affects the estimation accuracy of the biological state is determined, and the change in the measurement state is The weighting factor for the estimated value of the biological condition is variably set based on the determination result, and the estimated value of the biological condition is weighted according to the variably set weighting factor, and the weighted biological condition is estimated It is intended to output a value.

  According to the first aspect, for example, when the measurement state of the target person changes due to the movement of the target person, the weighting factor is variably set according to the change of the measurement state, and the variably set weighting factor The estimate of the biological condition of the subject is weighted. For this reason, the estimated value of the biological condition of the subject takes into consideration the influence of the change in the measurement condition caused by the movement of the subject, that is, the possibility of erroneous estimation, thereby obtaining highly reliable estimation results. It becomes possible.

  According to a second aspect of the biological condition estimation apparatus of the present invention, in the first aspect, the information processing apparatus further comprises a notification unit, and the estimated value of the weighted biological condition output from the weighting unit by the notification unit The degree of influence on the work by the biological condition of the subject person is determined based on the above, and the determination result is output.

  According to the second aspect, for example, based on the highly reliable estimation result in which the influence of the change in the measurement state caused by the movement of the subject is considered, the degree of the influence on the work by the biological state of the subject Is determined, and the determination result is output. For this reason, it is possible to perform highly reliable determination as to the degree of influence on the work by the biological condition of the subject, and as a result, highly reliable determination results can be output.

  In a third aspect of the biological condition estimation apparatus according to the present invention, in the first aspect, the measurement information acquisition unit acquires face image information of the subject as the measurement information, and the estimation unit The movement of the face or eyes of the subject is detected from the face image information as the feature parameter, and the degree of drowsiness is estimated based on the movement of the face or eyes as the biological condition. Further, the measurement state determination unit determines whether the face image information includes information capable of detecting the movement of the face or the eye of the subject as the measurement state, and the weight control unit determines the face image The weighting factor is set to a first value when it is determined that the information includes information capable of detecting the movement of the face or eye of the subject, and the weighting is determined when the information is determined not to be included. The coefficient is set to a second value smaller than the first value.

  According to the third aspect, for example, the motion of the subject's face or eye is detected from the face image information of the subject, and the subject's physical condition is sleepiness as the subject's biological state based on the motion of the subject's face or eye. In the estimation, it is determined whether the face image information includes information capable of detecting the movement of the face or eye of the subject. Then, if it is determined not to be included, the weighting factor for the estimated value of sleepiness is set to a second value smaller than the first value set when it is determined to be included. That is, when the measurement state of the face image information of the object person can not detect the movement of the face or eye of the object person, the weighting factor for the estimated value of sleepiness is set to a small value.

  For example, with the movement of the subject, the face or eyes of the subject being worked may be temporarily shielded from the sensor unit by peripheral devices or the subject's own hands or arms, etc. When the measurement state of the face image information of the subject changes due to a change in the direction etc. of the subject, the change of the measurement state is taken into consideration to change the weighting factor, and as a result, the estimated value of sleepiness is corrected.

  Therefore, when the measurement state of the face image information changes to a state that adversely affects the determination of drowsiness due to the movement of the subject, the erroneous estimation result can be prevented from being output as the correct one. Can improve the reliability of the estimation result.

  In a fourth aspect of the device according to the present invention, in the third aspect, the estimation unit detects, from the face image information, a vestibular oculoocular reflection that represents a movement of a pupil of the eye of the subject, A first estimation unit for estimating the degree of drowsiness based on vestibular moving eye reflection, and information representing the degree of eye blink or eyelid opening and closing of the subject detected from the face image information and detected A second estimation unit for estimating the degree of drowsiness based on information representing the degree of opening and closing of blinks or eyebrows, and the change of a specific part of the face of the subject from the face image information as information representing a change in expression A detection unit is provided with at least one of a third estimation unit that estimates the degree of sleepiness based on information representing a change in the detected facial expression.

  According to the fourth aspect, for example, a vestibular oculoocular reflex representing the movement of the pupil of the subject's eye, information representing the degree of blinking or eyelid opening and closing of the subject's eye, and a change in facial expression of the subject's face It is possible to estimate the sleepiness of the subject using at least one of the information representing.

  In a fifth aspect of the device according to the present invention, in the fourth aspect, the measurement state determination unit determines whether the face image information includes an image representing the face of the subject as the measurement state. Determining that the weight control unit determines that the face image information does not include an image representing the face of the subject, at least one of the first, second, and third estimation units. Setting the weighting factor for the estimated value obtained by the second method to the second value.

  According to the fifth aspect, for example, the face image information does not include an image representing the face of the subject such as when the face of the subject is facing downward or deviates from the imaging field of view of the sensor unit. The weighting coefficients for the estimation results of the first, second, and third estimation units that estimate drowsiness based on the movement of the face or eye are smaller than in the case where an image representing the face is obtained. It is set. For this reason, when the face image information does not include the image representing the face of the subject, the possibility of erroneous estimation can be taken into consideration so that the incorrect estimation result can be prevented from being output as the correct one. Can improve the reliability of the estimation result.

  In a sixth aspect of the device according to the present invention, in the fourth aspect, the measurement state determination unit determines that the orientation of the face of the subject is the sensor portion based on the face image information as the measurement state. The weight control unit determines whether the orientation of the face of the subject is within the range of the predetermined angle with respect to the direction of the sensor unit. The weighting factor for the estimated value obtained by at least one of the first and third estimating units is set to the second value when it is determined that the second value does not hold.

  According to the sixth aspect, for example, even when the face image information includes the face of the target person, as in the case where the target person faces sideways to the sensor portion, the direction of the face is the direction of the sensor portion On the other hand, if it does not fall within the predetermined angle range, the frontal image of the face can not be used to estimate drowsiness, so drowsiness is determined based on changes in the vestibular moving eye reflection or facial expression of the subject. The weighting factors for the estimation results of the first and third estimation units to be estimated are set to smaller values as compared with the case where the front image of the face is obtained. For this reason, when the subject is facing sideways, it is possible to prevent the erroneous estimation result from being output as the correct one in consideration of the possibility of the erroneous estimation, thereby making the reliability of the estimation result It can be enhanced.

  In a seventh aspect of the device according to the present invention, in the fourth aspect, whether the measurement state determination unit includes an image representing at least one of the eyes of the subject as the measurement state It is determined whether or not the weight control unit determines that the face image information does not include an image representing at least one of the subject's eyes, at least the first and second estimation units. The weighting factor for the estimated value obtained by one is set to the second value.

  According to the seventh aspect, for example, when the subject changes the position or the orientation of the head with respect to the sensor unit, and thereby both eyes of the subject are not reflected in the face image information, based on the eye movement. The weighting coefficients for the estimation results of the first and second estimation units that estimate drowsiness are set to smaller values as compared to the case where at least one eye image appears in the face image information. For this reason, when no image of the subject's eye is included in the face image information, it is possible to prevent an erroneous estimation result from being output as a correct one, in consideration of the possibility of erroneous estimation. This can improve the reliability of the estimation result.

  In an eighth aspect of the device according to the present invention, in the fourth aspect, the measurement state determination unit determines whether the face image information includes an image representing the mouth of the subject as the measurement state. When it is determined that the weight control unit does not include the image representing the mouth of the subject, the weight coefficient for the estimated value by the third estimation unit is set to the second value. The

  According to the eighth aspect, when, for example, the face image information does not include an image representing the mouth as in the case where the subject is masking, sleepiness is determined based on the subject's expression. The weighting factor for the estimation value by the third estimation unit to be estimated is set to a smaller value than in the case where the face image information includes an image representing a mouth. For this reason, in the case where the subject is, for example, masking, it is possible to prevent the erroneous estimation result from being output as the correct one in consideration of the possibility of erroneous estimation. Reliability can be improved.

  In a ninth aspect of the device according to the present invention, in the fourth aspect, the measurement state determination unit determines that the direction of the line of sight of the subject is the sensor portion based on the face image information as the measurement state. The weight control unit determines whether the direction of the line of sight of the subject is within a range of a predetermined angle with respect to the direction of the sensor unit. The weighting factor for the estimated value by the first estimation unit is set to the second value when it is determined that the second value is not determined.

  According to the ninth aspect, in the case where the pupil can not be detected in the face image information, for example, as in the case where the subject turns the direction of the line of sight from the front to the side, drowsiness is estimated based on vestibular moving eye reflection. The weighting factor for the estimation result of the first estimation unit is set to a smaller value than when the pupil is detected. For this reason, for example, when the target person turns the line of sight in the horizontal direction from the front, it is possible to prevent an incorrect estimation result from being output as a correct one in consideration of the possibility of erroneous estimation. Can improve the reliability of the estimation result.

  In a tenth aspect of the device according to the present invention, in the first aspect, the measurement information acquisition unit acquires heartbeat information of the subject as the measurement information, and the estimation unit performs the feature parameter as the feature parameter. The feature quantity having relevance to the autonomic nerve activity is detected from the heart rate information, the drowsiness degree is estimated based on the feature quantity having relevance to the autonomic nerve activity as the biological condition of the subject, and the measurement state determination is performed And determining, as the measurement state, whether or not the heartbeat information includes an information component capable of detecting a feature amount having relevance to the autonomic nerve activity, and the weight control unit determines that the heartbeat information is autonomous. The weighting factor is set to a third value when it is determined that the feature amount having relevance to the neural activity includes the detectable information component, and the weighting factor is determined when it is determined not to include the information component. Before It is obtained so as to set the third value smaller than the fourth value.

  According to the tenth aspect, for example, the mounting state of the sensor unit changes due to a change in the posture of the subject, which makes it impossible to detect the feature amount necessary to estimate the sleepiness of the subject from the electrocardiogram information In this case, the weighting factor for the estimated value of drowsiness is set to a smaller value than in the case where the feature value can be detected. That is, when the electrocardiographic information of the subject can not be detected with high quality, the weighting factor for the estimated value of sleepiness is set to a small value. Therefore, when the feature amount necessary for sleepiness estimation can not be detected from the electrocardiogram information, it is possible to prevent the erroneous estimation result from being output as the correct one in consideration of the possibility of the erroneous estimation. It is possible to improve the reliability of the estimation result.

  That is, according to each aspect of the present invention, when the measurement state of the subject changes due to the movement of the subject, the erroneous estimation result is not output as the correct one, and the highly reliable estimation result is output. Can provide technology that can

FIG. 1 is a block diagram for explaining an application example of the biological condition estimation apparatus according to the present invention. FIG. 2 is a schematic configuration diagram of a vehicle equipped with a biological condition estimation apparatus according to an embodiment of the present invention. FIG. 3 is a block diagram showing the hardware configuration of the biological condition estimation apparatus according to the embodiment of the present invention. FIG. 4 is a block diagram showing a software configuration added to the hardware configuration of the biological condition estimation apparatus according to the embodiment of the present invention. FIG. 5 is a flowchart showing the processing procedure and the processing content by the biological condition estimation apparatus shown in FIG. FIG. 6 is a flowchart showing the first half of the process procedure of the weight coefficient control process and the process content in the process procedure shown in FIG. 5; FIG. 7 is a flowchart showing the second half of the processing procedure of the weighting factor control processing and the processing content in the processing procedure shown in FIG. 5;

  Hereinafter, embodiments according to the present invention will be described with reference to the drawings.

[Example of application]
First, an application example of a biological condition estimation apparatus according to an embodiment of the present invention will be described. The present application example is an application of the biological condition estimation apparatus to the case of estimating the biological condition of a driver of a vehicle, for example, sleepiness. The living condition is not limited to sleepiness, but may be fatigue, physical condition, excitement, or the like.

FIG. 1 is a block diagram showing a functional configuration of a biological condition estimation apparatus in the present application example.
The biological condition estimation apparatus 1 includes a measurement information acquisition unit 2, a plurality of biological condition estimation units 3a to 3n, a measurement condition determination unit 4, a weight control unit 5, a weighting addition unit 6, and a biological condition determination unit 7. Is equipped.

  The measurement information acquisition unit 2 acquires, as measurement information, face image information and vital information of a driver as a target person measured by a sensor unit (not shown). The face image information is obtained, for example, by a camera included in the sensor unit. Vital information is obtained by a vital sensor included in the sensor unit. An example of the vital sensor is an electrocardiographic sensor. The measurement information may be either face image information or vital information.

  The living body condition estimation units 3a to 3n respectively detect feature amounts related to a plurality of different feature parameters necessary to estimate the driver's sleepiness from the face image information and vital information. The characteristic parameters include, for example, pupil movement of the driver's eyes, eye blink movement or eyelid opening and closing movement, facial expression of the driver's face, heartbeat, if it can estimate the drowsiness of the driver It may be other than.

In addition, the biological condition estimation units 3a to 3n estimate the drowsiness degree of the driver based on the feature quantities of the detected feature parameters.
For example, the living body state estimation unit 3a calculates a value representing the vestibular oculoocular reflex (VOR) based on the information indicating the pupil movement of the driver's eye and the information indicating the movement of the driver's head. For example, the value representing the vestibular oculoocular reflex is compared with a threshold value, and the result is output as an estimated value of drowsiness. The vestibular oculomotor reflex is a function that moves the eyeball in the opposite direction when the head moves to prevent blurring of the external image seen on the retina, and more specifically, the correlation between the angular velocity of the head and the angular velocity of the pupil Detected as a value. The drowsiness estimation using VOR can also be obtained by using the standard deviation value of VOR or the phase delay, in addition to comparing the VOR value (gain) with the threshold value.

  For example, the living body state estimation unit 3b calculates the number or frequency of blinks per unit time, or the total time during which the eyelids are closed per unit time, based on the information indicating blink motion of the eye or eyelid opening and closing motion. Then, these calculated results are compared with a threshold value, and the result is output as an estimated value of drowsiness.

  For example, the living body condition estimation unit 3c (not shown) is an information indicating the distance between feature points of the face such as eyes, eyebrows, and mouth, or the edge of the face surface from the driver's face image. As detected. Then, the calculated value is compared with a threshold value, and the result is output as an estimated value of sleepiness.

  For example, the living body state estimation unit 3n detects a heartbeat interval (RRI) as a feature quantity related to human autonomic nerve activity from heartbeat information of the driver, and analyzes the heartbeat interval by spectrum to obtain high frequency components (HF) And detect low frequency components (LF). Then, the degree of drowsiness is estimated based on the detected values of the high frequency component (HF) and the low frequency component (LF).

  Based on the face image information acquired by the measurement information acquisition unit 2, the measurement state determination unit 4 is generated due to the driver's movement and estimates the sleepiness in each of the biological condition estimation units 3a to 3n. A change in measurement state (a change in face image information) that affects the accuracy is determined. As the assumed driver's movement, for example, an operation of steering wheel, an operation of swinging a face or a line of sight to confirm surroundings, an operation of changing a driving posture to reduce fatigue, and the like are assumed. Examples of changes in the face image information to be judged include the presence / absence of face detection, face orientation (face angle with respect to camera), presence / absence of eye detection, presence / absence of mouth detection, direction of sight line (camera The amount of deviation of the line of sight with respect to the direction can be considered.

  Further, based on the heartbeat information acquired by the measurement information acquisition unit 2, the measurement state determination unit 4 determines whether or not the heartbeat information contains information necessary for estimating sleepiness. For example, it is determined whether the signal strength or S / N of the heartbeat waveform in the heartbeat data is equal to or greater than a threshold.

  The weight control unit 5 determines the change in the measurement state that affects the drowsiness estimation accuracy obtained by the measurement state determination unit 4, that is, based on the determination result of the change in the face image information or the heart rate information. A weighting factor is variably set for the sleepiness estimated value output from each of the living body state estimation units 3a to 3n for each of the living body state estimation units 3a to 3n.

  For example, in the case where the determination result of the change in the face image information or the heart rate information indicates that there is no possibility of affecting the drowsiness estimation accuracy or less than a predetermined value, all biological condition estimation units 3a A first weighting factor is set for the estimated value of to 3n.

  On the other hand, if the determination result of the change in the face image information or the heart rate information indicates that the drowsiness estimation accuracy may be affected, the type of change in the face image information or the heart rate information is In response, a living body state estimation unit that requires a response is selected. Then, a second weighting factor smaller than the first weighting factor is set for the estimated value of the selected living body state estimation unit. The second weighting factor may set a preset value uniformly, but the above estimation is made for each of the biological condition estimation units 3a to 3n according to the type of change in the face image information or the heart rate information. It may be set variably according to the degree of influence on the value.

  The weighting addition unit 6 weights the first or second weighting coefficient set by the weight control unit 5 with respect to the sleepiness estimated value output from the living body state estimation units 3a to 3n. Then, the weighted estimated values are summed and output to the living body condition determination unit 7. In addition, prior to the above-mentioned weighted addition, it is preferable to perform a normalization process according to a predetermined rule on the weight coefficient set for each of the living body state estimation units 3a to 3n.

  For example, the living body condition determining unit 7 compares the drowsiness estimated value after the weighting addition output from the weighting addition unit 6 with a threshold set in advance, and determines whether or not a problem may occur in the driving operation. Do. Then, when it is determined that the estimated value of drowsiness may cause a problem in driving operation, the living body condition determination unit 7 outputs notification control information to a notification unit (not shown), for example, to reduce drowsiness for the driver or The notification operation for suppressing is performed.

Since it is a structure as mentioned above, the biological condition estimation apparatus 1 operate | moves as follows.
That is, the driver operates the steering wheel, shakes his / her face or line of sight to check the surroundings, and changes the driving posture to reduce fatigue, etc. As a result, the driver's face image information or heartbeat It is assumed that the information does not contain the information necessary to estimate the driver's drowsiness. For example, the driver's face is not displayed, the face is displayed but the face direction is not directed to the camera, the eyes are not displayed, or the eyes can not be detected. It is assumed that the line-of-sight can not be detected, or that the mouth is not shown and the information can not be determined. The state of the face image information or the heart rate information is determined by the measurement state determination unit 4, and the determination result is given to the weight control unit 5.

  Then, in the weight control unit 5, when it is shown that the determination result of the change in the face image information or the heart rate information has no possibility of affecting the estimation accuracy of sleepiness or is less than a predetermined value, all living bodies A first weighting factor is set for the estimated values of the state estimation units 3a to 3n. On the other hand, when the determination result of the change in the face image information or the heart rate information indicates that the drowsiness estimation accuracy may be affected, the biological condition estimation in which the influence is likely to occur is A second weighting factor smaller than the first weighting factor is set for the estimated value of the part. Then, the weight control unit 5 supplies the weight addition unit 6 with the first or second weighting coefficient set for each of the living body condition estimation units 3a to 3n.

  As a result, the weighting addition unit 6 performs weighting processing on the sleepiness estimated value output from the living body state estimation units 3a to 3n according to the given weighting coefficient. Then, after the weighted estimated values are summed and added, the weighted addition unit 6 passes it to the living body condition determination unit 7.

  In the living body condition determination unit 7, the estimated value of sleepiness after weighting addition passed from the weighting addition unit 6 is compared with a preset threshold value. Based on the comparison result, the living body condition determination unit 7 determines whether the drowsiness estimated value after the weighting addition is an undesirable value for the driver to drive and that it is not preferable. When it is determined, notification control information is output. As a result, in the notification unit, a notification operation for reducing or suppressing drowsiness is performed to the driver.

  Therefore, according to the living body condition estimation apparatus 1, for example, the face or the eye is temporarily shielded from the camera by the steering wheel or the driver's own hand or arm by the movement of the driver's driving operation or the like. When the driver's face direction or the like changes with respect to, and the measurement state of the driver's face image information changes, the estimated state of sleepiness is corrected in consideration of the measurement state. Therefore, when the measurement state changes so as to adversely affect the determination of drowsiness due to the movement of the driver, it is possible to prevent the erroneous estimation result from being output as the correct one, and thereby the reliability of the estimation result Can be enhanced.

  Also, for example, even if the mounting state of the electrocardiogram sensor with respect to the driver is changed due to a change in the driver's driving posture, the possibility of erroneous estimation can be obtained even when the feature amount necessary for sleepiness estimation can not be detected from the electrocardiogram information. Therefore, it is possible to prevent an erroneous estimation result from being output as a correct one, thereby enhancing the reliability of the estimation result.

[One embodiment]
(Example of configuration)
FIG. 2 is a block diagram showing an example of a system configuration of a vehicle equipped with a biological condition estimation apparatus according to an embodiment of the present invention.
(1) System In the vehicle 10, a biological condition estimation device 20, a display control device 30, and a camera 40 are installed. Further, a wristwatch-type wearable terminal 50 is attached to the wrist of the driver 60. The camera 40 and the wearable terminal 50 constitute a sensor unit.

  The camera 40 is disposed, for example, at a position facing the driver 60 of the dashboard. The camera 40 uses, for example, a CMOS (Complementary MOS) image sensor capable of receiving near-infrared light as an imaging device. Then, the face of the driver 60 is imaged, and the face image data thereof is output to the biological condition estimation apparatus 20 via the signal cable. Note that as the imaging device, another solid-state imaging device such as a CCD (Charge Coupled Device) may be used.

  The wearable terminal 50 incorporates an electrocardiogram sensor. Then, an electrocardiogram signal of the driver 60 measured by the electrocardiogram sensor is sampled at a constant cycle and converted into electrocardiogram data, and then transmitted to the biological condition estimation apparatus 20 via the wireless interface.

(2) Biological state estimation device 20
The biological condition estimation apparatus 20 has a function of estimating sleepiness as one of the biological conditions of the driver 60 based on a plurality of feature parameters of the face image data and the electrocardiogram data having relevance to the sleepiness. And a function of weighting the drowsiness estimate. The weighting function determines a change in the measurement state of the face image data and the electrocardiogram data caused by the movement of the driver 60, variably sets a weighting factor for the drowsiness estimated value based on the determination result, The estimated value of drowsiness is weighted and added.

(2-1) Hardware Configuration FIG. 3 is a block diagram showing an example of the hardware configuration of the biological condition estimation apparatus 20. As shown in FIG.
The biological condition estimation apparatus 20 includes, as hardware, a hardware processor 21A such as a CPU (Central Processing Unit), and the hardware processor 21A includes a program memory 21B, a data memory 25, and a sensor interface (sensor I / I). F) An external interface (external I / F) 24 is connected via a bus 22.

  The sensor I / F 23 receives face image data output from the camera 40 via a signal cable. Further, the sensor I / F 23 receives the electrocardiogram data transmitted from the wearable terminal 50. The external I / F 24 outputs notification control data to the display control device 30 according to the drowsiness determination result.

  When the in-car wired network such as LAN (Local Area Network) or the in-car wireless network adopting low power wireless data communication standards such as Bluetooth (registered trademark) is provided in the car, the sensor I / F 23 Transmission of data between the camera 40 and the wearable terminal 50 and between the external I / F 24 and the display control device 30 may be performed using the above network.

  The program memory 21B uses, as a storage medium, for example, a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) that can be written and read as needed, or a non-volatile memory such as a ROM. Programs necessary to execute various control processes according to the present embodiment are stored.

  The data memory 25 includes, for example, a combination of a non-volatile memory such as an HDD or an SSD that can be written and read as needed, and a volatile memory such as a RAM as a storage medium, and relates to the present embodiment. It is used to store various data acquired, detected and calculated in the process of executing various processes.

(2-2) Software Configuration FIG. 4 is a block diagram showing the hardware configuration of the biological condition estimation apparatus 20 shown in FIG. 3 with the addition of a software configuration.
In the storage area of the data memory 25, an image information storage unit 251 and an electrocardiogram information storage unit 252 are provided. The image information storage unit 251 is used to store face image data acquired from the camera 40. The electrocardiogram information storage unit 252 is used to store the electrocardiogram data acquired from the wearable terminal 50. The data memory 25 also includes a storage area for storing various data detected or calculated in the process of the drowsiness estimation process and the weighted addition to the estimated value by the control unit 21.

  The control unit 21 includes the hardware processor 21A and the program memory 21B, and as a processing function unit by software, an image acquisition unit 211, an electrocardiogram information acquisition unit 212, and a first sleepiness estimation unit 213. , Second sleepiness estimation unit 214, third sleepiness estimation unit 215, fourth sleepiness estimation unit 216, measurement state determination unit 217, weight control unit 218, weighting addition unit 219, sleepiness determination unit And 220. Among these, the image acquisition unit 211 and the electrocardiogram information acquisition unit 212 constitute a measurement information acquisition unit. Further, the first sleepiness estimation unit 213, the second sleepiness estimation unit 214, the third sleepiness estimation unit 215, and the fourth sleepiness estimation unit 216 constitute a living state estimation unit.

  Each of the acquisition units 211 and 212, estimation units 213 to 216, determination units 217 and 220, control unit 218 and addition unit 219 causes the hardware processor 21A to execute the program stored in the program memory 21B. Is realized by

  The image acquisition unit 211 fetches face image data of the driver 60 output from the camera 40 through the sensor I / F 23 at a predetermined acquisition cycle, adds a time stamp, and stores the image data in the image information storage unit 251 of the data memory 25. Perform processing to store.

  The electrocardiogram information acquisition unit 212 acquires the electrocardiogram data transmitted from the wearable terminal 50 through the sensor I / F 23 at the predetermined acquisition cycle, and corresponds the time stamp indicating the acquired time to the acquired electrocardiogram data. A process of adding a time stamp and storing the time stamp in the electrocardiogram information storage unit 252 of the data memory 25 is performed. Each time stamp has a role of temporally correlating the face image data and the electrocardiogram data.

The first sleepiness estimation unit 213 performs the following processing.
(1) Based on the image of the head or face and the image of the eye included in the face image data stored in the image information storage unit 251, predetermined detection as a first feature parameter for estimating drowsiness A process of detecting vestibular oculoocular reflex (VOR) in a cycle. As described above, the vestibular oculomotor reflex is a function that moves the eye in the opposite direction when the head moves to prevent blurring of the external image reflected on the retina, specifically, the head or It is detected as a correlation value between the angular velocity of the face and the angular velocity of the pupil.

  (2) A process of calculating the VOR gain and the standard deviation based on the detected value representing the vestibular moving eye reflex (VOR), and calculating the drowsiness estimated value E1 based on these calculated values. The threshold may be one or more, and can be selected in accordance with the required estimated value identification accuracy.

The second sleepiness estimation unit 214 performs the following processing.
(1) Blinking of the eye at the predetermined detection cycle as a second feature parameter for estimating drowsiness based on the image of the eye included in the face image data stored in the image information storage unit 251 A process of detecting a feature that indicates the movement or the opening and closing movement of the eyelid. Specifically, the number or frequency of blinks per unit time, or the total value of the eyelid closing time per unit time, that is, the eye open time ratio (PERCLOS) is calculated.

  (2) A process of comparing the calculated number or frequency of blinks or the total value of the eyelid closing time per unit time with a threshold value and outputting the result as a sleepiness estimated value E2. The threshold may be one or more, and can be selected in accordance with the required estimated value identification accuracy.

The third sleepiness estimation unit 215 performs the following processing.
(1) Information indicating the facial expression of the driver's face in the predetermined detection cycle as a third feature parameter for estimating drowsiness based on the face image data stored in the image information storage unit 251 Process to detect Specifically, a feature point of each part such as eyes, eyebrows and mouth and face feature points such as an edge of a face are detected from the face image, and a change in distance between the plurality of detected feature points or Detect changes in edge strength of the face surface.

  (2) The detected change in the distance between facial feature points such as eyes, eyebrows, and mouth, or the detected value of the change in edge strength of the face, for example, is compared with a threshold value and the result is an estimated value of sleepiness E3 Process to output as. The above threshold value may be one or more, and can be arbitrarily selected according to the required estimated value identification accuracy.

The fourth sleepiness estimation unit 216 performs the following processing.
(1) As the fourth feature parameter for estimating drowsiness based on the electrocardiogram data stored in the above-mentioned electrocardiogram information storage unit 252, the autonomic nerve activity and the relativity in the above-mentioned predetermined detection cycle A process of detecting a heartbeat interval (RR Interval: RRI), which is a feature quantity having the following feature, and spectrally analyzing the heartbeat interval (RRI) to detect a high-frequency component (HF).

  (2) A process of estimating drowsiness based on the value of the detected high frequency component (HF) and outputting the estimated value E4. The degree of drowsiness can also be estimated by calculating the heart rate from the electrocardiogram data. In general, when the heart rate is low, it indicates a state of high sleepiness, and when it is high, it indicates a state of low sleepiness.

The measurement state determination unit 217 performs the following processing.
(1) Based on the face image data stored in the image information storage unit 251, a face image is generated due to the movement of the driver 60 and the first to third drowsiness estimation units 213 to 215 The process of determining whether or not the state affects the estimation accuracy of sleepiness in For example, a state in which no face image is displayed, a state in which the direction of the face with respect to the camera 40 is out of a predetermined angle range, a state in which both eyes are not shown, a state without a mouth, and a direction of a line of sight with respect to the camera 40 Is equivalent to a state in which the angle is out of the predetermined angle. As the movement of the driver 60, for example, an operation of steering wheel, an operation of swinging a face or a line of sight to confirm surroundings, an operation of changing a driving posture to reduce fatigue, and the like are assumed.

  (2) A state in which the electrocardiogram data stored in the electrocardiogram information storage unit 252 is generated due to the movement of the driver 60 and affects the sleepiness estimation accuracy in the fourth sleepiness estimation unit 216 Processing to determine whether or not For example, it is determined whether the signal strength or S / N of the heartbeat waveform represented by the electrocardiogram data is below a threshold.

  The weight control unit 218 controls each of the sleepiness estimating units 213 to 216 based on the determination result of the state of the face image data and the electrocardiogram data which affect the drowsiness estimation accuracy, which is obtained by the measurement state determining unit 217. Then, a process of variably setting weighting factors for the sleepiness estimated values E1 to E4 output from the sleepiness estimation units 213 to 216 is performed. Detailed examples of variable settings will be described later.

  The weighting addition unit 219 normalizes the drowsiness estimated values E1 to E4 output from the drowsiness estimation units 213 to 216, and then, the drowsiness estimated values after the normalization are variably set by the weight control unit 218, respectively. The weighting factor is weighted. Then, the weighted sleepiness estimated values are added, and the weighted added sleepiness estimated value E is passed to the sleepiness determination unit 220.

  The drowsiness determination unit 220 compares, for example, the drowsiness estimated value E after weighting addition passed from the weighting addition unit 219 with a determination threshold set in advance, which may cause a problem in the driver's 60 driving operation. It is determined whether or not there is a state. Then, when it is determined that the driver is in a state in which a problem may occur in the driving operation, the external I / F 24 outputs notification control information to the display control device 30.

(3) Display control device 30
The display control device 30 has a hardware processor and a memory, and implements the alarm control function by causing the hardware processor to execute the program stored in the memory. The alarm control function is a function to generate an alarm for sleepiness from an output unit (not shown) based on the control information when the notification control information output from the biological condition estimation apparatus 20 is received.

(Operation example)
Next, an operation example of the biological condition estimation apparatus 20 will be described using FIGS. 5 and 6.
FIG. 5 and FIG. 6 are flowcharts showing an example of the processing procedure and the processing content by the biological condition estimation apparatus 20.

(1) Initial Setting of Weighting Coefficient The biological condition estimating apparatus 20 performs initial setting processing of the weighting coefficient, triggered by, for example, that the driver 60 is seated at the driver's seat and the start key of the engine is operated. For example, under the control of the weight control unit 218, in step S10, with respect to the estimated values E1 to E4 obtained by the first to fourth sleepiness estimating units 213 to 216, weighting coefficients are calculated from the initial data area in the data memory 25. The initial values (for example, the same value) of W1 to W4 are read out, and the initial values of the weighting factors are uniformly set. The initial values of the weighting factors W1 to W4 for the estimated values E1 to E4 may be set to different values according to the degree of influence of the estimated values E1 to E4 given to the comprehensive sleepiness estimated value E. Good.

(2) Acquisition of Measurement Information In response to the operation of the start key of the engine, the camera 40 starts imaging of the face of the driver 60 at a constant frame cycle, and the imaged face image data is sent to the biological condition estimation device 20 Output. Under the control of the image acquisition unit 211, in step S11, the biological condition estimation apparatus 20 captures face image data of the driver 60 captured by the camera 40 via the sensor I / F 23, and captures the captured face image data. Are stored in the image information storage unit 251 in association with the time stamp indicating the acquisition time.

  On the other hand, in the wearable terminal 50, measurement of an electrocardiogram signal is constantly performed by the built-in electrocardiogram sensor. Then, an electrocardiogram signal obtained by the electrocardiogram sensor is sampled at a constant sampling cycle and converted into electrocardiogram data, and the electrocardiogram data is transmitted to the biological state estimation device 20. On the other hand, the biological condition estimation apparatus 20 captures the electrocardiogram data of the driver 60 captured by the camera 40 through the sensor I / F 23 in step S12 under the control of the electrocardiogram information acquisition unit 212. The acquired electrocardiogram data is stored in the image information storage unit 251 in association with a time stamp indicating the acquisition time.

  The acquisition process of the face image data and the electrocardiogram data described above is repeatedly performed in a predetermined detection cycle until the engine of the vehicle 10 is stopped.

(3) Estimating drowsiness When acquisition processing of the measurement information is started, the biological condition estimating device 20 causes the first to fourth sleepiness estimating units 213 to 216 to generate face image data and heart having the same acquisition time. A process of estimating the drowsiness of the driver 60 is executed based on the telegraphic data.

(3-1) Estimation of Drowsiness Based on VOR The biological state estimation device 20 performs processing of face image data from the image information storage unit 251 of the data memory 25 in step S13 under the control of the first sleepiness estimation unit 213. Read sequentially according to the series. Then, based on the image of the head or face and the image of the eye included in each face image data, the vestibular moving eye reflection (VOR) is detected as a first feature parameter having relevance to drowsiness. VOR is calculated, for example, as a correlation value between the angular velocity of the head or face of the driver 60 and the angular velocity of the pupil.

  Subsequently, the first sleepiness estimation unit 213 calculates the VOR gain and the standard deviation from the detection value of the VOR, and estimates the drowsiness level of the driver 60 based on the VOR gain and the standard deviation. Then, the estimated value E 1 of drowsiness is output to the weighted addition unit 219.

  For a method of estimating drowsiness based on the VOR gain and standard deviation, for example, “Nishiyama et al.,“ Drowsiness precursor detection by vestibular moving eye reflex ”, Biomedical Engineering, Vol. 48 (2010) No. 1 p1 It is described in detail in -10.

(3-2) Estimating drowsiness based on blinks The biological state estimation device 20 controls the face image data from the image information storage unit 251 of the data memory 25 in step S14 under the control of the second sleepiness estimation unit 214. Read sequentially according to time series. And based on the image of the eye contained in the said face image data, the feature-value which shows the blink motion of eye, or eyelid open-close motion is detected as 2nd feature parameter which has relevance with drowsiness. As the feature amount, for example, the number or frequency of blinks per unit time, or a total value of time during which the eyelids per unit time are closed, that is, an eye open time ratio (PERCLOS) is calculated.

  Then, the estimated value is determined by comparing the calculated number or frequency of blinks or the total value of the time during which the eyelids per unit time are closed with a preset threshold, and the determination result is blinked. It outputs to the weighting addition part 219 as the estimated value E2 of drowsiness based.

  The method of estimating drowsiness by the above-mentioned percentage of eye opening time (PERCLOS) is described, for example, “WW Wierwille et al.“ Research on vehicle-based driver status / performance monitoring; development, validation and refinement of algorithms for detection of driver drowsiness, ”National Highway Traffic Administration Final Report: DOT HS 808 247 (1994).

(3-3) Estimation of Drowsiness Based on Facial Expression The biological state estimation device 20 controls the face image data from the image information storage unit 251 of the data memory 25 in step S15 under the control of the third sleepiness estimation unit 215. Read sequentially according to the series. Then, from the respective face image data, feature points of each part such as eyes, eyebrows and mouth and face feature points such as an edge of face are detected as third feature parameters having relevance to drowsiness, A change in distance per unit time between a plurality of detected feature points or a change in intensity per unit time of an edge of a face surface is detected.

  Then, the third sleepiness estimation unit 215 compares, for example, the detected value of the change in the distance between the face feature points such as the eye, eyelid and mouth or the change in the edge strength of the face with the threshold value. The result is output to the weighted addition unit 219 as the sleepiness estimated value E3.

  In addition, about the method of estimating drowsiness based on the change of the distance between the feature points of the face and the change of face surface edge strength, for example, “Sato et al.,“ Drowsiness detection of driver based on correlation of face image features and drowsiness ” , The Information Processing Society of Japan 78th National Conference Draft "is described in detail.

(3-4) Estimation of Drowsiness Based on Heart Rate Information The biological condition estimation device 20 controls the electrocardiogram data from the electrocardiogram information storage unit 252 of the data memory 25 in step S16 under the control of the fourth sleepiness estimation unit 216. Are sequentially read in chronological order. Then, based on the above-mentioned electrocardiogram data, a heart rate interval (RRI) is detected as a fourth characteristic parameter related to drowsiness, and this heart rate interval (RRI) is subjected to spectrum analysis to obtain a high frequency component (HF). To detect. Then, drowsiness is estimated based on the value of the detected high frequency component (HF), and the estimated value E4 is output to the weighted addition unit 219.

  In addition, about the estimation method of the drowsiness using the spectrum analysis of the said heart rate interval, for example, it is detailed in "Nakano et al.," Wake level detection technique of driver ", FUJITSU, Vol. 59 No. 4, p 416-420, 2008". Have been described.

(4) Determination of Change in Measurement State Due to Movement of Driver 60 The biological state estimation device 20 determines the measurement state in parallel with the sleepiness estimation process in the first to fourth sleepiness estimation units 213 to 216 described above. Under the control of the unit 217, in step S17, the measurement state of the face image data and the electrocardiogram data adversely affects the sleepiness estimation accuracy in the first to fourth sleepiness estimation units 213 to 216. For each item below, it is determined whether there is any.

(4-1) Presence / absence of detection of face image For example, the measurement state determination unit 217 reads face image data from the image information storage unit 251 and tries to detect an image representing a face from the face image data. Pattern recognition or the like can be used to detect the image representing the face. If the face image of the driver 60 can not be recognized as a result of the detection processing, it is determined that the measurement state of the face image data is a state that affects the estimation accuracy of the sleepiness.

(4-2) Face Orientation If, for example, the face image of the driver 60 can be detected from the face image data, the measurement state determination unit 217 determines an angle indicating the face orientation with respect to the camera 40 based on the face image. Calculate The angle indicating the orientation of the face can be obtained, for example, by detecting feature quantities (a plurality of parameters) representing the position of each part of the face from face image data and comparing the feature quantities with a three-dimensional face shape model. it can. The measurement state determination unit 217 determines whether or not the face orientation angle with respect to the camera is equal to or greater than a predetermined angle (for example, 20 degrees), and in the case of 20 degrees or more, the measurement state of the face image data is It is determined that the state affects the estimation accuracy of the sleepiness. In addition, the method of calculating | requiring the direction of a face is illustrated by international publication 2006/051607 specification or patent 4093273, for example.

(4-3) Presence of Detection of Image Representing Eye When the face image of the driver is detected from the face image data, the measurement state determination unit 217 detects one or more images representing the eye from the face image. To determine whether the Then, if it is determined that both eyes are not detected as a result of this determination, it is determined that the measurement state of the face image data at this time is a state that affects the estimation accuracy of the sleepiness.

(4-4) Presence / absence of detection of image representing mouth The measurement state determination unit 217 determines whether an image representing the mouth is detected from the face image when the driver's face image is recognized from the face image data. Determine if Then, if it is difficult to determine the expression if the image representing the mouth is not detected as a result of the determination, it is assumed that the measurement state of the face image data at this time is the state that influences the estimation accuracy of the sleepiness. judge.

(4-5) Direction of Line of sight When one or more images representing an eye are detected from the face image data, the measurement state determination unit 217 detects the direction of the line of sight based on the image representing the eye. The direction of the line of sight can be calculated, for example, from the relative relationship between the position of the bright spot of the eyeball and the position of the pupil. Then, it is determined whether the angle in the direction of the line of sight with respect to the forward direction of the vehicle, that is, the direction of 0 degrees with respect to the vehicle coordinate system is less than a predetermined angle (for example 5 degrees). In the above case, it is determined that the measurement state of the face image data affects the estimation accuracy of the sleepiness. The conversion between the camera coordinate system and the vehicle coordinate system can be performed based on the mounting angle of the camera with respect to the vehicle.

(4-6) Quality Decrease of Electrocardiogram Data The measurement state determination unit 217 reads the electrocardiogram data from the electrocardiogram information storage unit 252 of the data memory 25. Then, based on the electrocardiogram data, it is determined whether the signal strength or S / N of the electrocardiogram signal waveform is below the threshold. As a result of this determination, if the signal intensity or S / N of the heartbeat waveform is below the threshold, it is difficult to detect the feature amount used for sleepiness estimation, so the measurement state of the electrocardiogram data at this time is the above. It is determined that it is a state that affects the estimation accuracy of sleepiness.

(5) Setting Control of Weighting Factors The biological condition estimating apparatus 20 controls setting of weighting factors as follows based on the determination result of the measurement state determining unit 217 in step S18 under the control of the weighting coefficient control unit 218. I do.

6 and 7 are flowcharts showing the processing procedure and the processing content of the weight coefficient control unit 218.
The weighting factor control unit 218 first refers to the determination result of the measurement state determination unit 217 in step S181 to determine whether an image representing a face can not be detected from the face image data. If the face image can not be detected, the process proceeds to step S182. If the face image can not be detected, VOR, blinks and expressions can not be detected. For this reason, in step S182, the first, second and third sleepiness estimation units 213, 214 and 215 that perform estimation processing based on the VOR, blinks and facial expression are selected as targets of setting change of weighting coefficients, and these are The weighting factors W1, W2, and W3 for the estimated values E1, E2, and E3 obtained by the sleepiness estimating unit 213, 214, and 215 are set to a value (for example, "0.1") smaller than the initial value (for example, "1"). Do. When the setting processing is completed, the weight coefficient control unit 218 proceeds to step S191.

  On the other hand, when the face image can be detected from the face image data in step S181, the weight coefficient control unit 218 proceeds to step S183. Then, in step S183, with reference to the determination result by the measurement state determination unit 217, it is determined whether or not the angle of the face orientation with respect to the camera 40 is, for example, 20 degrees or more. As a result of the determination, when the angle of the face direction is, for example, 20 degrees or more, the process proceeds to step S184. If the face orientation angle is 20 degrees or more, only oblique images or side-face images of the face can be obtained, and degradation in detection accuracy of VOR and expression can not be avoided. For this reason, in step S184, the first and third sleepiness estimation units 213 and 215 that estimate drowsiness based on VOR or facial expression are selected as the target of setting change of the weighting factor, and these sleepiness estimation units 213 and 215 are selected. The weighting factors W1 and W3 for the estimated values E1 and E3 obtained by the above are set to a value (for example, "0.5") smaller than the initial value (for example, "1").

  Next, in step S185, the weighting factor control unit 218 refers to the determination result by the measurement state determination unit 217, and determines whether at least one eye is detected or not detected from the face image data. . And when both eyes are not detected, it shifts to Step S186. It is difficult to detect VOR and blinks if the eye image is not detected in both eyes. Therefore, in step S186, the first and second drowsiness estimation units 213 and 214 that estimate drowsiness based on VOR or blinks are selected as the target of setting change of the weighting coefficient, and these drowsiness estimation units 213 and 214 are selected. The weighting factors W1 and W2 for the estimated values E1 and E2 obtained by the above are set to a value (for example, "0.1") smaller than the initial value (for example, "1").

  Subsequently, in step S187, the weighting factor control unit 218 refers to the determination result by the measurement state determination unit 217, and determines whether or not the image of the mouth is detected from the face image data. Then, when the image of the mouth is not detected, the process proceeds to step S188. If the image of the mouth is not detected, a decrease in the detection accuracy of the expression can not be avoided. For this reason, in step S188, the third sleepiness estimation unit 215 that uses a facial expression in the drowsiness estimation process is selected as a target of setting change of the weighting coefficient, and the weighting coefficient for the estimated value E3 obtained by the third sleepiness estimation unit 215 W3 is set to a value (e.g. "0.3") smaller than the initial value (e.g. "1").

  Furthermore, in step S189, the weighting factor control unit 218 refers to the determination result by the measurement state determination unit 217, and determines whether the direction of the line of sight of the driver 60 with respect to the camera direction is, for example, 5 degrees or more. If the direction of the line of sight deviates by 5 degrees or more from the direction of the camera 40, the process proceeds to step S190. If the direction of the line of sight is deviated by 5 degrees or more with respect to the direction of the camera 40, it is difficult to detect the pupil, which hinders the detection of VOR. For this reason, in step S190, the first sleepiness estimation unit 213 that uses VOR to estimate sleepiness is selected, and the weighting factor W1 for the estimated value E1 obtained by the first sleepiness estimation unit 213 is set to an initial value (for example, 1 ′ ′) set to a smaller value (eg “0.1”).

  Next, in step S191, the weighting factor control unit 218 refers to the determination result by the measurement state determination unit 217, and determines whether the signal intensity or S / N of the heartbeat waveform, that is, the quality of the heartbeat signal is below a threshold. judge. Then, if it is determined that the quality of the heartbeat signal is lower than the threshold value, the process proceeds to step S192. If the quality of the heartbeat signal is below the threshold, it is difficult to detect feature quantities used for sleepiness estimation. Therefore, in step S192, the fourth drowsiness estimation unit 216 that uses the heart beat feature amount for drowsiness estimation is selected as the target of setting change of the weighting coefficient, and the estimated value E4 obtained by the fourth drowsiness estimation unit 216 is selected. The weighting factor W4 is set to a value (for example, "0.1") smaller than the initial value (for example, "1").

  The weighting factor control unit 218 sets an initial value (for example, “1”) previously set for the weighting factor for which the setting of the weighting factor has not been changed in steps S182, S184, S186, S188, S190, and S192 in step S193. Keep the

  Then, when setting control of all the weighting factors W1 to W4 ends, the weighting factor control unit 218 performs normalization processing on the set weighting factors W1 to W4 in step S194. For example, the values of the weighting factors W1 to W4 are normalized so that the total value of the weighting factors W1 to W4 becomes "1".

(6) Weighted Addition When setting control of the weighting factors W1 to W4 is completed, the living body condition estimation apparatus 20 subsequently proceeds to step S19. Under the control of the weighting addition unit 219, weighting factors set by the weighting factor control unit 218 with respect to the estimated sleepiness values E1 to E4 obtained by the first to fourth sleepiness estimating units 213 to 216, respectively. W1 to W4 are weighted, and the drowsiness estimated values E1 to E4 after the weighting are added. That is, the drowsiness estimated values E1 to E4 are weighted and added. Then, the overall drowsiness estimated value E obtained by the above-described weighting addition is passed to the drowsiness determination unit 220.

(7) Determination of Drowsiness Under the control of the drowsiness determination unit 220, the biological state estimation device 20 first compares the comprehensive drowsiness estimated value E with the determination threshold set in advance in step S20, and calculates the drowsiness estimated value. It is determined whether E exceeds a threshold. That is, it is determined whether the drowsiness estimated value E is a value that may cause trouble in the driving operation. As a result, if the drowsiness estimated value E is a value that may cause a problem in the driving operation, notification control information (alarm information) is output from the external I / F 24 to the display control device 30 in step S21.

  In the display control device 30, when the notification control information (alarm information) is received from the biological condition estimation device 20, control is performed to generate an alarm for sleepiness from an output unit (not shown) based on the control information.

  For example, an alarm message for sleepiness is generated and displayed on a display such as a navigation device. Also, an alarm sound or a chime is output from a speaker of a navigation device or a car stereo. Furthermore, when the seat or the like is provided with a vibrator, a vibrator drive signal is output, whereby the vibrator vibrates.

(effect)
As described above, in one embodiment, the biological condition estimation apparatus 20 captures the face image data of the driver 60 output from the camera 40 at a constant acquisition cycle and also transmits the electrocardiogram data transmitted from the wearable terminal 50. Are captured in the above acquisition cycle. Then, in each of the first to third drowsiness estimation units 213 to 215, from the face image data, a feature indicative of vestibular moving eye reflection (VOR), a feature indicative of eyelid blinking motion or eyelid opening and closing motion, and driving The feature quantity representing the facial expression of the person is detected, and the drowsiness of the user is estimated based on these feature quantities. Further, the fourth sleepiness estimation unit 216 detects a feature amount that is related to the autonomic nerve activity of a person from the above-mentioned electrocardiogram data, and estimates the sleepiness of the user based on this feature amount.

  On the other hand, in parallel with the above estimation processes, the measurement state determination unit 217 determines whether or not the face image data and the electrocardiogram data can correctly detect the feature quantities, and the determination result is Based on the weighting factor control unit 218, the weighting factors W1 to W4 for the estimated values of the estimation units 213 to 216 are variably set. Then, in the weighting addition unit 219, the above estimated values are added by weighting based on the weighting factors W1 to W4, and based on the result, the sleepiness estimation unit 220 determines the drowsiness of the user.

  Therefore, according to one embodiment, for example, the face or eyes are temporarily shielded from the camera 40 by the steering wheel, the driver's own hand or arm, etc. by the driver's driving operation or the like, or the driver drives the camera 40. When the measurement state of the driver's face image data changes due to a change in the face direction etc. of the person, the weighting factors W1 to W4 are variably set according to the measurement state, whereby the drowsiness estimated value is corrected. Ru. Therefore, when the measurement state changes so as to adversely affect the determination of drowsiness due to the movement of the driver, it is possible to prevent the erroneous estimation result from being output as the correct one, and thereby the reliability of the estimation result Can be enhanced.

  In addition, for example, even when the wearing state of the wearable terminal 50 with respect to the driver changes due to a change in the driving posture of the driver, and thus the feature amount necessary for sleepiness estimation can not be detected from the electrocardiogram data, the possibility of erroneous estimation. Therefore, it is possible to prevent an erroneous estimation result from being output as a correct one, thereby enhancing the reliability of the estimation result.

[Modification]
While the embodiments of the present invention have been described in detail, the above description is merely illustrative of the present invention in all respects. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. That is, in the implementation of the present invention, a specific configuration according to the embodiment may be appropriately adopted. Some specific modifications are shown below.

(1) In the above embodiment, it is binary determined whether the face image data and the electrocardiogram data can correctly detect each feature amount, and the weighting factors W1 to W4 are determined based on the determination results. Was set. However, it is determined how correctly each feature can be detected, that is, the detection accuracy of the feature is determined in multiple stages, and the weighting factors W1 to W4 are variably set in multiple stages according to the determination value. May be This makes it possible to estimate drowsiness more accurately.
Further, in the above-described embodiment, when the change in the measurement state of the measurement information is determined, it is determined whether the face image data and the electrocardiogram data are in the state in which each feature amount can be correctly detected. I explained to you. However, the present invention is not limited to this, and the change in the measurement state of the measurement information may be determined based on the position and the posture of the object person with respect to the sensor. For example, when the subject wears a wearable terminal incorporating a motion sensor such as a three-axis acceleration sensor or an angular velocity sensor, the change in the measurement state of the measurement information indirectly based on the sensing data of the motion sensor Determine For example, when it is detected from the sensing data of the motion sensor that the position of the subject has moved to a position where it deviates from the imageable range of the camera, or when the posture of the subject changes to a posture where imaging by the camera is not possible. It can be determined that the measurement of the measurement information is impossible.

(2) In the embodiment, the drowsiness estimation process is performed by detecting a total of four types of feature amounts from face image data and one type from electrocardiogram data. However, the present invention is not limited thereto, and drowsiness may be estimated based on four or more types of feature amounts, and conversely, drowsiness may be determined based on three or less types of feature amounts. In this case, when only one type of feature quantity is used, any one of the four types of feature quantities or more can be selected.
For example, among the above three types of feature quantities obtained from face image data and one type of feature quantities obtained from electrocardiogram data, estimation of drowsiness is performed using only the above three types of feature quantities obtained from face image data. You may do so. In this way, it is possible to estimate sleepiness based on only face image data without using electrocardiogram data, and therefore it is possible to eliminate the need for an electrocardiogram sensor simply by using a camera. In this case, all the above three types of feature quantities obtained from face image data may be used, or at least one of them may be used, and further, extraction is possible from captured image information of the above target person Other than the above three types of feature amounts may be used.
In addition, sleepiness may be estimated using only feature amounts obtained from a biological sensor such as an electrocardiogram sensor without using feature amounts extracted from image data obtained by the camera. In this case, the camera can be eliminated.

  (3) In the above-described embodiment, the case of estimating sleepiness for a driver of a vehicle has been described as an example. However, the present invention is not limited thereto, and the drowsiness state of the auxiliary driver at rest in the front passenger seat, the rear seat or the nap room of the vehicle may be estimated. By doing this, it can be determined whether the auxiliary driver is in a state where it is possible to alternate driving.

  (4) Furthermore, the sleepiness estimation target may be, for example, not only people who are involved in driving a vehicle but also workers of a business place such as a factory. In this way, it is possible to give workers a stimulus in accordance with the estimated value of drowsiness, which makes it possible to more appropriately manage the efficiency of production or production.

  (5) Furthermore, in one embodiment, although the case where sleepiness is estimated as the biological condition of the subject is described as an example, in addition to sleepiness, for example, a decrease in arousal level, fatigue level, mental level such as excitement level or complaint. In this case, it may be possible to estimate non-calm condition, headache, back pain, poor physical condition such as fever, etc.

  (6) In addition, specific types of feature amounts, specific values of weighting coefficients W1 to W4 set for respective estimated values obtained from the respective feature amounts, specific determination methods in the sleepiness determination unit, determination results of drowsiness The method of presenting the stimulus to the user according to the above may be variously modified without departing from the scope of the present invention.

  In short, the present invention is not limited to the above embodiment as it is, and at the implementation stage, the constituent elements can be modified and embodied without departing from the scope of the invention. In addition, various inventions can be formed by appropriate combinations of a plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components in different embodiments may be combined as appropriate.

[Supplementary note]
A part or all of each embodiment described above can be described as shown in the following appendices besides the claims, but is not limited thereto.
(Supplementary Note 1)
A biological condition estimation apparatus comprising a hardware processor (21A) and a memory (21B), comprising:
The hardware processor (21A) executes the program stored in the memory (21B),
Measurement information including at least one of captured image information of a subject and information serving as a physiological index is acquired from the sensor unit (211).
Detecting a characteristic parameter related to a biological condition affecting the work of the subject from the measurement information, and estimating the biological condition based on the characteristic parameter (213 to 216);
A change in the measurement state of the measurement information, which is caused due to the movement of the subject including the work and affects the estimation accuracy of the biological state, is determined (217).
The weighting factor for the estimated value of the biological condition is variably set based on the determination result of the change of the measurement condition (218).
Weighting is performed on the estimated value of the biological condition in accordance with the variably set weighting coefficient, and the estimated value of the weighted biological condition is output (219).
Biological condition estimation device.

(Supplementary Note 2)
A biological condition estimation method to be executed by an apparatus having a hardware processor (21A) and a memory (21B) storing a program for executing the hardware processor,
The hardware processor (21A) acquiring from the sensor unit measurement information including at least one of captured image information of a subject and information representing a physiological index (S11, S12);
The hardware processor (21A) detects a feature parameter related to a biological condition affecting the work of the subject from the measurement information, and estimates the biological condition based on the feature parameter (S13 to S16) ),
Determining the change in the measurement state of the measurement information generated due to the movement of the subject including the work and affecting the estimation accuracy of the biological state by the hardware processor (21A); (S17),
And (S18) the hardware processor (21A) variably sets a weighting factor for the estimated value of the biological condition based on the determination result of the change in the measurement condition.
And (S19) the hardware processor (21A) weights the estimated value of the biological condition according to the variably set weighting coefficient, and outputs the weighted estimated value of the biological condition (S19)
A biological condition estimation method comprising.

1 ... biological state estimation device, 2 ... measurement information acquisition unit, 3a to 3n ... biological state estimation unit,
4 ... measurement state determination unit, 5 ... weight control unit, 6 ... weighted addition unit, 7 ... living body condition determination unit,
DESCRIPTION OF SYMBOLS 10 ... Vehicle, 20 ... Biological-state estimation apparatus 30, 30 ... Display control apparatus, 40 ... Camera,
50: wearable terminal, 21: control unit, 21A: CPU,
21B: Program memory, 22: Bus, 23: Sensor I / F,
24 External I / F 25 Data memory 211 Image acquisition unit
212: electrocardiogram information acquisition unit 213: first sleepiness estimation unit 214: second sleepiness estimation unit
215 ... third sleepiness estimation unit, 216 ... fourth sleepiness estimation unit, 217 ... measurement state determination unit,
218: weighting factor control unit, 219: weighting addition unit, 220: sleepiness determination unit,
251 ... image information storage unit, 252 ... electrocardiogram information storage unit.

Claims (13)

A measurement information acquisition unit that acquires measurement information including at least one of captured image information of a subject and vital information from a sensor unit;
An estimation unit configured to detect feature parameters related to a biological condition affecting the work of the target person from the measurement information, and estimate the biological condition based on the feature parameters;
A measurement state determination unit that determines a change in the measurement state of the measurement information that is generated due to the movement of the target person including the work and that affects the estimation accuracy of the biological state;
A weight control unit that variably sets a weighting factor for the estimated value of the biological state based on the determination result of the change in the measurement state;
And a weighting unit that weights the estimated value of the biological condition according to the variably set weighting coefficient, and outputs the weighted estimated value of the biological condition.   An output unit configured to determine the degree of influence on the work by the biological condition of the subject based on the weighted estimated biological condition output from the weighting unit; The biological condition estimation apparatus according to claim 1 comprising. The measurement information acquisition unit acquires face image information of the subject as the measurement information,
The estimation unit detects the movement of the face or eye of the subject from the face image information as the feature parameter, and estimates the degree of drowsiness based on the movement of the face or eye as the biological condition.
The measurement state determination unit determines, as the measurement state, whether or not the face image information includes information capable of detecting the movement of the face or eye of the subject.
The weight control unit sets the weight coefficient to a first value when it is determined that the face image information includes information capable of detecting the movement of the face or eye of the subject, and does not include the weight coefficient. Setting the weighting factor to a second value smaller than the first value when it is determined that
The biological condition estimation apparatus according to claim 1. The estimation unit
A first estimation unit that detects, from the face image information, a vestibular vestibular reflex that represents movement of a pupil of the subject's eye, and estimates the degree of sleepiness based on the detected vestibular vestibular reflex;
Information representing the degree of blinking or eyelid opening and closing of the subject's eyes is detected from the face image information, and the degree of drowsiness is estimated based on the detected information representing the degree of eyelid opening or closing. A second estimation unit,
Third, a change in a specific part of the face of the subject is detected as information representing a change in expression from the face image information, and the drowsiness degree is estimated based on the information representing the change in the detected expression. The biological condition estimation device according to claim 3, comprising at least one of: an estimation unit. The measurement state determination unit determines whether the face image information includes an image representing the face of the subject as the measurement state,
The weight control unit is acquired by at least one of the first, second, and third estimation units when it is determined that the face image information does not include an image representing the face of the subject. Setting a weighting factor for the estimated value to be calculated to the second value,
The biological condition estimation apparatus according to claim 4. The measurement state determination unit determines, as the measurement state, based on the face image information, whether or not the orientation of the face of the subject is included within a predetermined angle range with respect to the direction of the sensor unit. And
When it is determined that the orientation of the face of the target person is not included in the range of the predetermined angle with respect to the direction of the sensor unit, the weight control unit is configured to operate the first and third estimation units. Setting a weighting factor for the estimated value obtained by at least one of them to the second value;
The biological condition estimation apparatus according to claim 4. The measurement state determination unit determines whether the face image information includes an image representing at least one of the subject's eyes as the measurement state.
The weight control unit estimates that the face image information does not include an image representing at least one of the subject's eyes, an estimation obtained by at least one of the first and second estimation units. Setting a weighting factor for the value to the second value;
The biological condition estimation apparatus according to claim 4. The measurement state determination unit determines whether the face image information includes an image representing the subject's mouth as the measurement state,
When it is determined that the weight control unit does not include an image representing the mouth of the subject, the weight control unit sets a weighting factor for the estimated value by the third estimation unit to the second value.
The biological condition estimation apparatus according to claim 4. The measurement state determination unit determines, as the measurement state, based on the face image information, whether or not the direction of the line of sight of the subject is included within a predetermined angle range with respect to the direction of the sensor unit. And
The weight control unit is configured to weight the estimated value by the first estimation unit when it is determined that the direction of the line of sight of the subject is not included in the range of a predetermined angle with respect to the direction of the sensor unit. Setting the factor to said second value,
The biological condition estimation apparatus according to claim 4. The measurement information acquisition unit acquires heart rate information of the subject as the measurement information,
The estimation unit detects, as the feature parameter, a feature amount that is related to the autonomic nerve activity from the heart rate information, and based on the feature amount that is related to the autonomic nerve activity as the biological condition of the subject. Estimate the degree of sleepiness,
The measurement state determination unit determines, as the measurement state, whether or not the heart rate information includes an information component capable of detecting a feature amount related to the autonomic nerve activity.
The weight control unit sets the weight coefficient to a third value when it is determined that the heart rate information includes an information component capable of detecting a feature amount that is related to the autonomic nerve activity. If it is determined that the information component is not included, the weighting factor is set to a fourth value smaller than the third value.
The biological condition estimation apparatus according to claim 1. A measurement information acquisition unit that acquires captured image information of a target person from an imaging unit;
An estimation unit configured to detect feature parameters related to a biological condition affecting the work of the subject from the captured image information, and estimate the biological condition based on the feature parameters;
A measurement state determination unit that determines a change in the measurement state of the captured image information that is generated due to the movement of the subject including the work and that affects the estimation accuracy of the biological state;
A weight control unit that variably sets a weighting factor for the estimated value of the biological state based on the determination result of the change in the measurement state;
And a weighting unit that weights the estimated value of the biological condition according to the variably set weighting coefficient, and outputs the weighted estimated value of the biological condition. A biological condition estimation method executed by a biological condition estimation apparatus, comprising:
A process in which the biological condition estimation apparatus acquires measurement information including at least one of captured image information and vital information of a target person from a sensor unit;
The biological state estimation apparatus detects a characteristic parameter related to a biological state affecting the work of the subject from the measurement information, and estimates the biological state based on the characteristic parameter;
Determining the change in the measurement state of the measurement information which is generated due to the movement of the subject including the work and affects the estimation accuracy of the biological state;
A step of variably setting a weighting coefficient for the estimated value of the biological condition based on the determination result of the change of the measurement condition, the biological condition estimation apparatus;
The biological state estimation apparatus performs weighting on the estimated value of the biological state according to the variably set weighting coefficient, and outputting the weighted estimated value of the biological state. Method.   A program that causes a processor to execute the processing of each unit included in the biological condition estimation device according to any one of claims 1 to 11.