JP6926636B2 - State estimator - Google Patents

Description

The present invention relates to a state estimation device that estimates the state of a monitored person based on the eye movements of the monitored person.

Patent Document 1 states that an abnormality occurs when the total value of the normal saccade frequency indicating the occurrence frequency of saccades of the monitored person and the microsaccade frequency indicating the occurrence frequency of microsaccades of the monitored person is less than the threshold value. The technology presumed to be is disclosed. That is, a technique for presuming an abnormality when the frequency of occurrence of saccade and microsaccade is less than the threshold value is disclosed.

Japanese Unexamined Patent Publication No. 2017-23519

However, even when the frequency of saccades and microsaccades is low, the monitored person may not be abnormal. For example, the frequency of occurrence of saccades and microsaccades is also reduced by blinks and / or follow-up eye movements caused by the surrounding environment of the monitored person. Therefore, depending on the surrounding environment of the monitored person, the frequency of occurrence of saccades and microsaccades may decrease even if the monitored person is normal, and the accuracy of estimating the state of the monitored person may decrease.

Even if the frequency of saccades and microsaccades is low, if they tend to increase over time, it is considered that the condition of the monitored person has already recovered to normal. Therefore, even in such a case, it is considered that the estimation accuracy of the state of the monitored person is lowered.

One object of this disclosure is to provide a state estimation device capable of more accurately estimating the state of a monitored person based on the eye movements of the monitored person.

The above object is achieved by a combination of the features described in the independent claims, and the sub-claims define further advantageous specific examples of the invention. The reference numerals in parentheses described in the claims indicate, as one embodiment, the correspondence with the specific means described in the embodiments described later, and do not limit the technical scope of the present invention. ..

In order to achieve the above object, the first state estimation device of the present invention is a state estimation device that estimates the state of the monitored person based on the eye movements of the monitored person, and is a saccade of the monitored person and a saccade of the monitored person. A frequency acquisition unit (132) that acquires the frequency of rapid eye movements, which is at least one of the occurrence frequencies of microsaccades, an eye blink acquisition unit (131) that acquires the frequency of blinks of the monitored person, and a monitored person. Reinforcement information that is at least one of the follow-up eye movement acquisition unit (133) that acquires the occurrence time of the follow-up eye movement and the frequency change amount acquisition unit (134) that acquires the change amount of the rapid eye movement frequency of the monitored subject. State estimation unit (140, 140b, 140c) that estimates the state of the monitored person using the information acquired by the augmentation information acquisition unit in addition to the rapid eye movement frequency acquired by the acquisition unit (135) and the frequency acquisition unit. The reinforcement information acquisition unit is at least the blink acquisition unit, and the state estimation unit is monitored when the frequency of rapid eye movements acquired by the frequency acquisition unit is larger than the threshold value for the frequency of rapid eye movements. While presuming that the person's condition is normal, even if the frequency of rapid eye movements acquired by the frequency acquisition unit is less than or equal to the threshold for the frequency of rapid eye movements, the blinks acquired by the blink acquisition unit If the occurrence frequency is greater than the threshold value for that frequency, the state of the monitored person estimated to be normal.
In order to achieve the above object, the second state estimation device of the present invention is a state estimation device that estimates the state of the monitored person based on the eye movements of the monitored person, and is a saccade of the monitored person and a saccade of the monitored person. A frequency acquisition unit (132) that acquires the frequency of rapid eye movements, which is at least one of the occurrence frequencies of microsaccades, a blink acquisition unit (131) that acquires the frequency of blinks of the monitored person, and a monitored person. Reinforcement information that is at least one of the follow-up eye movement acquisition unit (133) that acquires the occurrence time of the follow-up eye movement and the frequency change amount acquisition unit (134) that acquires the change amount of the rapid eye movement frequency of the monitored subject. State estimation unit (140, 140b, 140c) that estimates the state of the monitored person using the information acquired by the augmentation information acquisition unit in addition to the rapid eye movement frequency acquired by the acquisition unit (135) and the frequency acquisition unit. The reinforcement information acquisition unit is at least a follow-up eye movement acquisition unit, and the state estimation unit monitors when the rapid eye movement frequency acquired by the frequency acquisition unit is larger than the threshold value for the rapid eye movement frequency. While presuming that the subject's condition is normal, even if the frequency of rapid eye movements acquired by the frequency acquisition unit is less than or equal to the threshold for the frequency of rapid eye movements, the follow-up eye movement acquisition unit acquires follow-up. When the occurrence time of eye movement is larger than the threshold value for the occurrence time, it is estimated that the condition of the monitored subject is normal.
In order to achieve the above object, the third state estimation device of the present invention is a state estimation device that estimates the state of the monitored person based on the eye movements of the monitored person, and is a saccade of the monitored person and a saccade of the monitored person. A frequency acquisition unit (132) that acquires the frequency of rapid eye movements, which is at least one of the occurrence frequencies of microsaccades, a blink acquisition unit (131) that acquires the frequency of blinks of the monitored person, and a monitored person. Reinforcement information that is at least one of the follow-up eye movement acquisition unit (133) that acquires the occurrence time of the follow-up eye movement and the frequency change amount acquisition unit (134) that acquires the change amount of the rapid eye movement frequency of the monitored subject. State estimation unit (140, 140b, 140c) that estimates the state of the monitored person using the information acquired by the augmentation information acquisition unit in addition to the rapid eye movement frequency acquired by the acquisition unit (135) and the frequency acquisition unit. The supplementary information acquisition unit is at least a frequency change acquisition unit, and the state estimation unit monitors when the frequency of rapid eye movements acquired by the frequency acquisition unit is greater than the threshold value for the frequency of rapid eye movements. While presuming that the subject's condition is normal, even if the frequency of rapid eye movements acquired by the frequency acquisition unit is less than or equal to the threshold for the frequency of rapid eye movements, the monitoring acquired by the frequency change acquisition unit When the amount of change in the frequency of rapid eye movements of the subject is an increase larger than the threshold for the amount of change, it is presumed that the condition of the monitored subject is normal.

According to this, at least one of the frequency of blinks of the monitored person, the time of occurrence of the follow-up eye movement of the monitored person, and the amount of change in the frequency of occurrence of saccade and microsaccade of the monitored person are determined. It becomes possible to estimate the state of the monitored person by using it. When the frequency of blinks of the monitored person is used, it is possible to prevent the decrease in the frequency of rapid eye movements due to the blinks from being estimated as an abnormality in the state of the monitored person. When the occurrence time of the follow-up eye movement of the monitored person is used, it is possible to prevent the decrease in the frequency of rapid eye movement due to the follow-up eye movement from being estimated as an abnormality in the state of the monitored person. When the amount of change in the rapid eye movement frequency of the monitored person is used, the decrease in the rapid eye movement frequency when the rapid eye movement frequency tends to increase should not be estimated as an abnormality in the state of the monitored person. Becomes possible. Therefore, the state of the monitored person can be estimated more accurately than the case where only the occurrence frequency of the saccade and the microsaccade of the monitored person is used. As a result, it becomes possible to more accurately estimate the state of the monitored person based on the eye movements of the monitored person.

It is a figure which shows an example of the schematic structure of the vehicle system 1. It is a figure which shows an example of the schematic structure of HCU10. It is a figure for demonstrating the determination of the determination threshold value about the rapid eye movement frequency. It is a flowchart which shows an example of the flow of the state estimation related processing in HCU10. It is a flowchart which shows an example of the flow of the eye movement extraction processing in HCU10. It is a flowchart which shows an example of the flow of the determination threshold value determination process in HCU10. It is a flowchart which shows an example of the flow of the state estimation process in HCU10. It is a figure which shows an example of the schematic structure of HCU10a. It is a figure which shows an example of the schematic structure of HCU10b. It is a figure which shows an example of the schematic structure of HCU10c.

A plurality of embodiments for disclosure will be described with reference to the drawings. For convenience of explanation, the parts having the same functions as the parts shown in the drawings used in the explanations up to that point may be designated by the same reference numerals and the description thereof may be omitted. be. For the parts with the same reference numerals, the description in other embodiments can be referred to.

(Embodiment 1)
<Outline configuration of vehicle system 1>
The vehicle system 1 shown in FIG. 1 is used in a moving body such as a vehicle, and has an HCU (Human Machine Interface Control Unit) 10, a vehicle sensor 20, a peripheral monitoring sensor 30, a peripheral monitoring ECU 40, a driver imaging unit 50, and a notification. Includes device 60. It is assumed that the HCU 10, the vehicle sensor 20, and the peripheral monitoring ECU 40 are connected to, for example, an in-vehicle LAN. In the following, the description will be continued by taking the case where the vehicle system 1 is used in an automobile as an example.

The vehicle sensor 20 is a group of sensors for detecting the state of the own vehicle. The vehicle sensor 20 includes a humidity sensor that detects the humidity inside the vehicle interior of the vehicle, an external light sensor that detects the brightness in front of the vehicle, a steering angle sensor that detects the steering angle of the vehicle, and the like. As an example, an illuminance sensor that detects illuminance may be used as the external light sensor. The vehicle sensor 20 outputs the sensing result to the in-vehicle LAN. The sensing result of the vehicle sensor 20 may be output to the vehicle LAN via the ECU mounted on the own vehicle.

The peripheral monitoring sensor 30 detects moving objects such as pedestrians, animals other than humans, vehicles, and stationary objects such as signs, guardrails, curbs, and trees around the vehicle. In addition, it detects road markings such as lane markings and stop lines around the vehicle. The peripheral monitoring sensor 30 includes, for example, a peripheral monitoring camera that captures a predetermined range around the vehicle, a millimeter wave radar that transmits exploration waves to a predetermined range around the vehicle, a sonar, and a LIDAR (Light Detection and Ranging / Laser Imaging Detection and). Ranging) and other sensors. The peripheral monitoring camera sequentially outputs the captured images to be sequentially captured as sensing information to the peripheral monitoring ECU 40. Sensors that transmit exploration waves such as sonar, millimeter-wave radar, and LIDAR sequentially output scanning results based on the received signal obtained when the reflected wave reflected by an obstacle is received to the peripheral monitoring ECU 40 as sensing information.

The peripheral monitoring ECU 40 includes a microcomputer including a processor, a volatile memory, a non-volatile memory, and a bus connecting these to each other, and executes various processes by executing a control program stored in the non-volatile memory. do. The peripheral monitoring ECU 40 recognizes the surrounding environment of the own vehicle from the sensing result acquired from the peripheral monitoring sensor 30.

The peripheral monitoring ECU 40 detects an object around the own vehicle as the surrounding environment of the own vehicle from the sensing result of the peripheral monitoring camera. As an example, it is preferable to detect an object to be gazed by the driver separately from an object not to be gazed by a well-known image recognition process such as template matching. Examples of objects to be watched by the driver (hereinafter referred to as “objects to be watched”) include pedestrians, other vehicles, signs, and the like. Further, the peripheral monitoring ECU 40 detects the relative position and relative speed of an object existing around the own vehicle with respect to the own vehicle by a known method from the sensing result of the peripheral monitoring sensor. The peripheral monitoring ECU 40 uses information on the position and speed of objects such as other vehicles and pedestrians obtained by vehicle-to-vehicle communication and / or road-to-vehicle communication via a communication module used in the own vehicle. For objects existing in the vicinity, the relative position and relative speed with respect to the own vehicle may be detected.

The driver imaging unit 50 includes an imaging device for imaging the head of the driver. As an example, the configuration may include a lighting device and an imaging device having sensitivity in the same band as the lighting by the lighting device. As a specific example, a configuration including a near-infrared camera and a near-infrared light source may be used. In the following, a case where the near-infrared camera and the near-infrared light source are used will be described as an example. When extracting the microsaccade described later, a high-performance camera having a spatial resolution of 0.01 degrees and a temporal resolution of 220 Hz may be used, for example.

The near-infrared camera is arranged, for example, on the steering column cover in a posture toward the driver's seat side of the own vehicle. The near-infrared camera may be arranged at another position as long as it can capture the face of the driver seated in the driver's seat of the own vehicle, and is arranged on the upper surface of the instrument panel or the like. May be configured. The driver imaging unit uses a near-infrared camera to photograph the head of the driver irradiated with near-infrared light by a near-infrared light source. The image captured by the near-infrared camera (hereinafter referred to as the face image) is output to the HCU 10.

The notification device 60 notifies the driver of the own vehicle. The notification device 60 includes a display device, a voice output device, a vibrator, and the like. Further, the notification by the notification device 60 includes a display by a display device, a voice output by a voice output device, a vibration by a vibrator, and the like.

The HCU 10 is mainly composed of a processor, a volatile memory, a non-transitory tangible storage medium, a non-volatile memory, an I / O, and a microcomputer equipped with a bus connecting them, and is a driver. It is connected to the image pickup unit 50, the notification device 60, and the in-vehicle LAN. By executing the control program stored in the non-volatile memory, the HCU 10 estimates the state of the driver of the own vehicle and causes the driver of the own vehicle to be notified. The HCU 10 corresponds to the state estimation device of the claim, and the driver corresponds to the monitored person of the claim.

<Outline configuration of HCU10>
Here, the schematic configuration of the HCU 10 will be described with reference to FIG. As shown in FIG. 2, the HCU 10 has an image recognition processing unit 100, an eye movement extraction unit 110, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140, and a notification processing unit 150 as functional blocks. It has. A part or all of the functions executed by the HCU 10 may be configured in hardware by one or a plurality of ICs or the like. Further, a part or all of the functional blocks included in the HCU 10 may be realized by a combination of software execution by a processor and hardware members.

The image recognition processing unit 100 extracts the driver's face part, extracts the shape feature of the face part, calculates the line-of-sight angle, etc. from the face image sequentially output from the driver image pickup unit 50 by a known image recognition process. Perform sequentially. The line-of-sight angle can be rephrased as the line-of-sight direction. An example of the image recognition processing performed by the image recognition processing unit 100 will be described below.

First, the image recognition processing unit 100 extracts the eyes in the face image and extracts the black circles in the eyes as pupils. Subsequently, the white spots in the eyes in the facial image are extracted as a corneal reflex image. Then, the line-of-sight angle with respect to the reference position in the vehicle interior is detected from the positional relationship between the pupil and the corneal reflex. The reference position may be, for example, the installation position of the near-infrared camera. The line-of-sight angle referred to here is assumed to be a line-of-sight angle in both the horizontal direction and the vertical direction, which substantially coincides with the left-right direction of the own vehicle. The calculated line-of-sight angle may be stored in a memory such as a volatile memory.

The eye movement extraction unit 110 performs an eye movement extraction process for extracting the eye movement of the driver of the own vehicle based on the result of the image recognition process in the image recognition processing unit 100. As shown in FIG. 2, the eye movement extraction unit 110 includes a blinking eye movement extraction unit 111, a rapid eye movement extraction unit 112, and a follow-up eye movement extraction unit 113 as sub-functional blocks.

The blink extraction unit 111 extracts the blink of the driver. Blinks can also be rephrased as blinks. The blink extraction unit 111 may be configured to extract, for example, a case where the pupil cannot be extracted as an blink in the image recognition process sequentially performed by the image recognition processing unit 100. Alternatively, the blinking eye may be extracted based on the radius of curvature of the upper eyelid of the driver extracted by the image recognition processing unit 100. As the result of extracting the blink, the blink extraction unit 111 stores the time at which the blink is extracted in the eye movement information storage unit 120 until, for example, the effective time T elapses. The eye movement information storage unit 120 may be configured to use, for example, a volatile memory. The effective time T is a time required for accurately calculating the frequency of occurrence of rapid eye movements such as saccade and microsaccade, which will be described later, and may be, for example, about 1 to several minutes. The same shall apply thereafter.

The rapid eye movement extraction unit 112 extracts rapid eye movements such as a driver's saccade and a microsaccade. Saccade is a short, high-speed eye movement that occurs when a person intentionally changes his or her line of sight. Microsaccades are the finest and most rapid eye movements of involuntary fixation tremors that occur when a person is gazing. The rapid eye movement extraction unit 112 may be configured to extract only one of the saccade and the microsaccade, or may be configured to extract both.

The rapid eye movement extraction unit 112 calculates the amount of change in the line-of-sight angle from the line-of-sight angles of the past several frames (for example, 10 frames) calculated by the image recognition processing unit 100, and divides this amount of change by the time. Calculate the line-of-sight angular velocity. Further, the rapid eye movement extraction unit 112 calculates the line-of-sight angular acceleration by dividing the amount of change in the line-of-sight angular velocity calculated as described above by time.

Then, the rapid eye movement extraction unit 112 extracts the saccade when the line-of-sight angular acceleration is equal to or greater than the reference angular acceleration and the line-of-sight movement distance is equal to or greater than the reference distance. On the other hand, the rapid eye movement extraction unit 112 extracts as a microsaccade when the line-of-sight angular acceleration is equal to or greater than the reference angular acceleration and the line-of-sight movement distance is less than the reference distance. The reference angular acceleration may be, for example, 1200 deg / s ² , and the reference distance may be, for example, 2 deg. In addition, instead of the line-of-sight angular acceleration, the determination may be made based on whether or not the line-of-sight angular velocity is equal to or higher than the reference angular velocity. The rapid eye movement extraction unit 112 may store the time at which the saccade and the microsaccade are extracted as the extraction result of the saccade and the microsaccade in the eye movement information storage unit 120 until, for example, the effective time T elapses.

The follow-up eye movement extraction unit 113 extracts the follow-up eye movement of the driver. The follow-up eye movement is an eye movement that causes the subject to reflexively follow the line of sight. The follow-up eye movement extraction unit 113 extracts the follow-up eye movement when the line-of-sight angular velocity per fixed time is within a certain range. As an example, when the line-of-sight angular velocity per ^{second is 0.5 to 2.0 deg / s 2,} it is extracted as a follow-up eye movement. The following eye movement extraction unit 113 may be configured to make a determination based on whether or not the line-of-sight angular acceleration is within a certain range, instead of the line-of-sight angular velocity. The following eye movement extraction unit 113 may store, as a result of extracting the following eye movement, whether or not the following eye movement is extracted in the eye movement information storage unit 120 for each time until, for example, the effective time T elapses.

The determination data calculation unit 130 acquires the determination data by calculating the determination data used for estimating the state of the driver. As shown in FIG. 2, the determination data calculation unit 130 includes the blink frequency calculation unit 131, the rapid eye movement frequency calculation unit 132, the follow-up eye movement time calculation unit 133, and the change amount calculation unit 134 as sub-functional blocks. Be prepared. The blink frequency calculation unit 131, the follow-up eye movement time calculation unit 133, and the change amount calculation unit 134 correspond to the reinforcement information acquisition unit 135.

The blink frequency calculation unit 131 is based on the time when the blink is extracted, which is the extraction result of the blink extraction unit 111 stored in the eye movement information storage unit 120, and the driver's blink frequency (hereinafter referred to as “blink frequency”). The blink frequency) is calculated to obtain the blink frequency. The blink frequency calculation unit 131 corresponds to the blink acquisition unit of the claim. As a specific example, the blink frequency calculation unit 131 may calculate the blink frequency per predetermined time t from the extraction result of the blink extraction unit 111 within the predetermined time t for each predetermined time t. The predetermined time t is, for example, about 10 seconds. The same shall apply thereafter. The blink frequency calculation unit 131 accumulates the blink frequency calculated sequentially in the eye movement information storage unit 120 in chronological order.

The rapid eye movement frequency calculation unit 132 extracts the saccade and microsaccade, which are the extraction results of the rapid eye movement extraction unit 112 accumulated in the eye movement information storage unit 120, and the driver's saccade. , The frequency of rapid eye movements is obtained by calculating the frequency of occurrence of microsaccades (hereinafter referred to as the frequency of rapid eye movements). The rapid eye movement frequency calculation unit 132 corresponds to the frequency acquisition unit according to the claim.

As a specific example, if the rapid eye movement frequency calculation unit 132 calculates the rapid eye movement frequency per predetermined time t from the extraction result of the rapid eye movement extraction unit 112 within the predetermined time t every predetermined time t. good. The rapid eye movement frequency calculation unit 132 stores the rapidly calculated rapid eye movement frequency in the eye movement information storage unit 120 in chronological order. It is preferable to improve the calculation accuracy of the rapid eye movement frequency by excluding the data at the time when the blink extraction unit 111 is extracting the blink from the process of calculating the rapid eye movement frequency.

The follow-up eye movement time calculation unit 133 determines the follow-up eye movement of the driver based on the presence or absence of the follow-up eye movement for each time, which is the extraction result of the follow-up eye movement extraction unit 113 accumulated in the eye movement information storage unit 120. The follow-up eye movement time is acquired by calculating the occurrence time (hereinafter referred to as the follow-up eye movement time). The time required for the follow-up eye movement can be rephrased as the time required for the follow-up eye movement. The following eye movement time calculation unit 133 corresponds to the following eye movement acquisition unit according to the claim. As a specific example, if the follow-up eye movement time calculation unit 133 calculates the follow-up eye movement time per predetermined time t from the extraction result of the follow-up eye movement extraction unit 113 within the predetermined time t for each predetermined time t. good. The follow-up eye movement time calculation unit 133 stores the follow-up eye movement time calculated sequentially in the eye movement information storage unit 120 in chronological order.

The change amount calculation unit 134 changes the rapid eye movement frequency based on the time when the saccade and the microsaccade, which are the extraction results of the rapid eye movement extraction unit 112 accumulated in the eye movement information storage unit 120, are extracted. By calculating the amount (hereinafter, the amount of change in the frequency of rapid eye movements), the amount of change in the frequency of rapid eye movements is obtained. This change amount calculation unit 134 corresponds to the frequency change amount acquisition unit of the claim. As a specific example, when the rapid eye movement frequency calculation unit 132 calculates the rapid eye movement frequency for each predetermined time t, the number of occurrences of saccade and microsaccade for s seconds immediately before the predetermined time t is calculated. The s seconds referred to here is a time shorter than the predetermined time t, and may be, for example, 2 to 3 seconds. Then, the ratio of the number of occurrences of saccade and microsaccade in the immediately preceding s seconds to the total number of occurrences within the predetermined time t is calculated as the amount of change in the frequency of rapid eye movements. The change amount calculation unit 134 stores the rapidly calculated rapid eye movement frequency change amount in the eye movement information storage unit 120 in chronological order.

The state estimation unit 140 performs a determination threshold value determination process for determining a determination threshold value used for estimating the driver's state from the data group of the determination data sequentially calculated by the determination data calculation unit 130. Then, a state estimation process for estimating the driver's state is performed from the determination data sequentially calculated by the determination data calculation unit 130 and the determination threshold value. As shown in FIG. 2, the state estimation unit 140 includes a determination threshold value determination unit 141 as a sub-function block.

The judgment threshold value determination unit 141 determines a judgment threshold value for the blink frequency, a judgment threshold value for the rapid eye movement frequency, a judgment threshold value for the follow-up eye movement time, and a judgment threshold value for the amount of change in the rapid eye movement frequency. I do.

As an example, the determination threshold value determination unit 141 determines the determination threshold value for the blink frequency as follows. The determination threshold value determination unit 141 is an average value of the blink frequency for each predetermined time t within the effective time T calculated by the blink frequency calculation unit 131, which is stored in the eye movement information storage unit 120 (hereinafter, average blink frequency). Frequency) is calculated. Then, the determination threshold value for the blink frequency is determined from the calculated average blink frequency. In other words, the determination threshold value for the blink frequency is determined from the data group of the actual blink frequency occurrence frequency in the past fixed period calculated by the blink frequency calculation unit 131.

For example, if the average blink frequency is less than 1 time / second, the judgment threshold is 0.5 times / second, and if the average blink frequency is 1 to 2 times, the judgment threshold is 1 time / second, and the average. When the blink frequency is 2 times or more, the judgment threshold value is set to 2 times / second. Further, the variation in the blink frequency for each predetermined time t may be calculated, and the determination threshold value may be determined to be smaller as the degree of variation increases.

The determination threshold value determination unit 141 determines the determination threshold value for the rapid eye movement frequency as follows, for example. The determination threshold value determination unit 141 is based on the rapid eye movement frequency for each predetermined time t within the effective time T calculated by the rapid eye movement frequency calculation unit 132 stored in the eye movement information storage unit 120. Determine the judgment threshold for exercise frequency. In other words, the determination threshold value for the blink frequency is determined from the data group of the actual rapid eye movement frequency in the past fixed period calculated by the rapid eye movement frequency calculation unit 132.

When only one of the saccade frequency and the microsaccade frequency is used as the rapid eye movement frequency, a value obtained by lowering the average value of the saccade frequency by a predetermined shift amount is determined. It may be a threshold value. This shift amount may be configured in advance by an experiment or the like so that the estimation accuracy of the driver's state becomes a desired accuracy. Further, the determination threshold value may be determined to be smaller as the degree of variation in the occurrence frequency for each predetermined time t increases.

On the other hand, when both the frequency of occurrence of saccade and the frequency of occurrence of microsaccade are used as the frequency of rapid eye movement, the determination is made using a regression equation obtained from the frequency of occurrence of saccade and the frequency of occurrence of microsaccade. The threshold value may be determined. The regression equation may be obtained as follows. First, on a graph in which the frequency of occurrence of saccade and the frequency of occurrence of microsaccade are on the horizontal axis and the vertical axis, respectively, the frequency of occurrence of saccade and the frequency of occurrence of microsaccade at predetermined time t within the effective time T, respectively. Is plotted (see FIG. 3). Then, the line A such as the most probable straight line or curve showing the tendency of these plots is obtained. For example, the most probable line A (see A in FIG. 3) may be obtained by the least squares method. Then, by shifting this line A toward the origin of the graph, the regression equation B (see B in FIG. 3) is obtained. The boundary represented by this regression equation B is the determination threshold value.

The shift amount when shifting the line A toward the origin may be configured in advance by an experiment or the like so that the estimation accuracy of the driver's state becomes a desired accuracy. Further, the shift amount may be increased as the degree of variation in the frequency of occurrence of saccades and microsaccades at predetermined time t increases.

The determination threshold value for the follow-up eye movement time in the determination threshold value determination unit 141 is determined as follows as an example. The determination threshold determination unit 141 is the average value of the follow-up eye movement time for each predetermined time t within the effective time T calculated by the follow-up eye movement time calculation unit 133 stored in the eye movement information storage unit 120 (hereinafter, average). Follow-up eye movement time) is calculated. Then, the determination threshold value for the following eye movement time is determined from the calculated average following eye movement time. In other words, the determination threshold value for the following eye movement time is determined from the data group of the actual occurrence time of the following eye movement in the past fixed period calculated by the following eye movement time calculation unit 133. For example, the determination threshold value determination unit 141 may set a value obtained by lowering the average follow-up eye movement time by a predetermined shift amount as the determination threshold value. This shift amount may be configured in advance by an experiment or the like so that the estimation accuracy of the driver's state becomes a desired accuracy. Further, the determination threshold value may be determined to be smaller as the degree of variation in the follow-up eye movement time for each predetermined time t increases.

As an example, the determination threshold value determination unit 141 determines the determination threshold value for the amount of rapid eye movement frequency change as follows. The determination threshold determination unit 141 is an average value (hereinafter, average) of the rapid eye movement frequency change amount for each predetermined time t within the effective time T calculated by the change amount calculation unit 134, which is stored in the eye movement information storage unit 120. Change amount) is calculated. Then, the determination threshold value for the rapid eye movement frequency change amount is determined from the calculated average change amount. In other words, the determination threshold value for the rapid eye movement frequency change amount is determined from the data group of the actual rapid eye movement frequency change amount in the past fixed period calculated by the change amount calculation unit 134. For example, the determination threshold value determination unit 141 may set a value obtained by lowering the average change amount by a predetermined shift amount as the determination threshold value. This shift amount may be configured in advance by an experiment or the like so that the estimation accuracy of the driver's state becomes a desired accuracy. Further, the determination threshold value may be determined to be smaller as the degree of variation in the average change amount for each predetermined time t increases.

In the state estimation process of the state estimation unit 140, when the rapid eye movement frequency sequentially calculated by the rapid eye movement frequency calculation unit 132 is equal to or greater than the judgment threshold value for the rapid eye movement frequency determined by the judgment threshold value determination unit 141, Judge that the driver status is normal. Further, in the state estimation process of the state estimation unit 140, the blink frequency is calculated even when the rapid eye movement frequency sequentially calculated by the rapid eye movement frequency calculation unit 132 is less than the determination threshold value for the rapid eye movement frequency. Each judgment threshold is one of the blink frequency sequentially calculated by the unit 131, the following eye movement time sequentially calculated by the following eye movement time calculation unit 133, and the rapid eye movement frequency change amount sequentially calculated by the change amount calculation unit 134. If it is the above, it is determined that the state of the driver is normal.

On the other hand, the state estimation unit 140 is a case where the rapid eye movement frequency sequentially calculated by the rapid eye movement frequency calculation unit 132 is less than the determination threshold value for the rapid eye movement frequency, and the blink frequency calculation unit 131. When the blink frequency calculated sequentially, the following eye movement time calculated sequentially by the following eye movement time calculation unit 133, and the rapid eye movement frequency change amount sequentially calculated by the change amount calculation unit 134 are all less than the respective judgment thresholds. In addition, it is determined that the state of the driver is abnormal. In the present embodiment, as an example, it is assumed that the concentrated state in which the driver is concentrated is normal, and the careless state in which the driver's attention is reduced is abnormal.

When the state estimation unit 140 estimates that the driver's state is inadvertent, the notification processing unit 150 instructs the notification device 60 to perform notification. The notification may be any as long as it causes the driver to recognize that the driver is inadvertent, for example, an alarm or the like.

<State estimation related processing in HCU10>
Subsequently, an example of the flow of the process related to the estimation of the driver state (hereinafter, the state estimation related process) in the HCU 10 will be described with reference to the flowcharts of FIGS. 4 to 7. The flowchart of FIG. 4 may be configured to start when the power of the HCU 10 is turned on and end when the power of the HCU 10 is turned off. When the power of the HCU 10 is turned on and off according to the on / off of the switch for starting the internal combustion engine or the motor generator of the own vehicle (hereinafter referred to as the power switch), it starts when the power switch is turned on and ends when the power switch is turned off. It may be configured to be used.

First, in step S1, the image recognition processing unit 100 performs image recognition processing from the face image acquired from the driver imaging unit 50. The image recognition processing unit 100 extracts the driver's face portion, extracts the shape feature of the face portion, calculates the line-of-sight angle, and the like by this image recognition processing.

In step S2, the eye movement extraction unit 110 performs the eye movement extraction process and proceeds to step S3. In the eye movement extraction process, when the eye movement to be extracted occurs, this eye movement is extracted. Here, the outline of the eye movement extraction process will be described with reference to the flowchart of FIG.

First, in step S21, the blink extraction unit 111 extracts the blink of the driver. In step S22, the rapid eye movement extraction unit 112 calculates the line-of-sight angular velocity based on the line-of-sight angles for the past several frames calculated by the image recognition processing unit 100. In step S23, the rapid eye movement extraction unit 112 calculates the line-of-sight angular acceleration based on the above-mentioned line-of-sight angular velocity.

In step S24, the rapid eye movement extraction unit 112 extracts the saccade when the line-of-sight angular acceleration calculated in S23 is equal to or greater than the reference angular acceleration and the line-of-sight movement distance is equal to or greater than the reference distance. In step S25, the rapid eye movement extraction unit 112 extracts the microsaccade when the line-of-sight angular acceleration is equal to or greater than the reference angular acceleration and the line-of-sight movement distance is less than the reference distance. In step S26, the following eye movement extraction unit 113 extracts the following eye movement of the driver, and proceeds to step S3. The order of the processes of S24 to S26 may be changed.

Returning to FIG. 4, in step S3, the eye movement data extracted by the eye movement extraction process of S2 is stored in the eye movement information storage unit 120 until the effective time T elapses. In step S4, when the effective time T has elapsed since the state estimation-related processing was started (YES in S4), the process proceeds to step S5. If the effective time T has not elapsed since the state estimation-related processing was started (NO in S4), the process returns to S1 and the processing is repeated. Whether or not the effective time T has elapsed since the state estimation-related processing was started may be determined by the state estimation unit 140 based on, for example, counting by the timer circuit or the like.

In step S5, if the determination threshold value determination unit 141 has already determined the determination threshold value, the process proceeds to step S7. On the other hand, if the determination threshold value determination unit 141 has not determined the determination threshold value, the process proceeds to step S6. In step S6, the determination threshold value determination unit 141 performs the determination threshold value determination process, and proceeds to step S7. Here, the outline of the determination threshold value determination process will be described with reference to the flowchart of FIG.

First, in step S61, the determination threshold value determination unit 141 determines the determination threshold value for the rapid eye movement frequency. In step S62, the determination threshold value determination unit 141 determines the determination threshold value for the blink frequency. In step S63, the determination threshold value determination unit 141 determines the determination threshold value for the follow-up eye movement. In step S64, the determination threshold value determination unit 141 determines the determination threshold value for the amount of rapid eye movement change, and the process proceeds to step S7. The order of the processes of S61 to S64 may be changed.

In the state estimation-related processing, the determination threshold value determination process may be performed only once, and after the determination threshold value is determined, the state estimation process may be performed using the determined determination threshold value. In the present embodiment, the determination threshold value determination process is performed once after the power of the HCU 10 is turned on to determine the determination threshold value, and the state is estimated using the determined determination threshold value until the power of the HCU 10 is turned off. conduct. According to this, the judgment threshold value is newly determined for each trigger of the start of the state estimation related processing such as turning on the power switch of the own vehicle and turning on the power of the HCU10, so that the judgment threshold value suitable for the driver is determined for each ride. It will be possible to redo. Therefore, even if the physical condition of the driver differs from day to day, it is possible to determine the determination threshold value according to the physical condition of the driver on that day and improve the estimation accuracy of the driver's state. Therefore, it is possible to improve the estimation accuracy of the driver's state while suppressing the processing load of redetermining the determination threshold value many times while the vehicle is traveling.

Returning to FIG. 4, in step S7, the state estimation unit 140 performs a state estimation process to estimate the driver's state, and proceeds to step S8. Here, the outline of the state estimation process will be described with reference to the flowchart of FIG. 7. In the state estimation process, the determination data for each predetermined time t is calculated, and the driver state is estimated by comparing the determination data with the determination threshold value determined in S6. In this embodiment, the driver's attention is estimated.

First, in S71, the state estimation unit 140 acquires the frequency of occurrence of the driver's saccade and microsaccade (that is, the frequency of rapid eye movement) calculated by the rapid eye movement frequency calculation unit 132 at the latest predetermined time t. The frequency of rapid eye movement may be any frequency of occurrence of saccade or microsaccade, but here, it is the total value of the frequency of occurrence of saccade and the frequency of occurrence of microsaccade. Subsequently, the state estimation unit 140 compares the rapid eye movement frequency at the latest predetermined time t with the determination threshold value for the rapid eye movement frequency determined in S6. Then, when the rapid eye movement frequency is higher than this determination threshold value (YES in S71), the process proceeds to step S75. On the other hand, when the rapid eye movement frequency is equal to or less than this determination threshold value (NO in S71), the process proceeds to step S72.

In step S72, the state estimation unit 140 acquires the driver's blink frequency at the latest predetermined time t calculated by the blink frequency calculation unit 131, and determines the threshold for the blink frequency and the blink frequency determined in S6. Compare with. Then, when the blink frequency is higher than this determination threshold value (YES in S72), the process proceeds to step S75. On the other hand, when the blink frequency is equal to or less than this determination threshold value (NO in S72), the process proceeds to step S73.

In step S73, the state estimation unit 140 acquires the following eye movement time of the driver at the latest predetermined time t calculated by the following eye movement time calculation unit 133, and the following eye movement time and the following eye movement time determined in S6. Compare with the judgment threshold for. Then, when the following eye movement time is larger than this determination threshold value (YES in S73), the process proceeds to step S75. On the other hand, when the following eye movement time is equal to or less than this determination threshold value (NO in S73), the process proceeds to step S74.

In step S74, the state estimation unit 140 acquires the driver's rapid eye movement frequency change amount for the latest predetermined time t calculated by the change amount calculation unit 134, and the rapid eye movement frequency change amount and the rapid eye movement frequency change amount determined in S6 are determined. Compare with the judgment threshold value for the amount of change in eye movement frequency. Then, when the amount of change in the frequency of rapid eye movements is larger than this determination threshold value (YES in S74), the process proceeds to step S75. On the other hand, when the amount of rapid eye movement frequency change is equal to or less than this determination threshold value (NO in S74), the process proceeds to step S76.

In step S75, the state estimation unit 140 estimates that the driver's state is normal, that is, a concentrated state, and proceeds to step S8. That is, the state estimation unit 140 concentrates the driver's state when any of the rapid eye movement frequency, the blink frequency, the follow-up eye movement time, and the rapid eye movement frequency change amount is larger than the respective determination threshold values. Presumed to be.

On the other hand, in step S76, the state estimation unit 140 estimates that the driver's state is abnormal, that is, a careless state, and proceeds to step S8. That is, the state estimation unit 140 is inattentive to the driver's state when all of the rapid eye movement frequency, the blink frequency, the follow-up eye movement time, and the rapid eye movement frequency change amount are equal to or less than the respective determination threshold values. Presumed to be. The order of processing of S71 to S74 may be changed.

Returning to FIG. 4, in step S8, when the driver's state is estimated to be a careless state in S7 (YES in S8), the process proceeds to step S9. On the other hand, when the driver state is estimated to be the concentrated state in S7 (NO in S8), the process proceeds to step S10. In step S9, the notification processing unit 150 gives an instruction to the notification device 60 to give an alarm, and the process proceeds to step S10.

In step S10, when it is the end timing of the state estimation-related processing (YES in S10), the state estimation-related processing is ended. On the other hand, if it is not the end timing of the state estimation related processing (NO in S10), the process returns to S1 and the processing is repeated. The end timing of the state estimation-related processing includes, for example, the case where the power of the HCU 10 is turned off, the case where the power switch of the own vehicle is turned off, and the like.

<Summary of Embodiment 1>
According to the configuration of the first embodiment, not only the rapid eye movement frequency which is the occurrence frequency of at least one of the driver's saccade and the microsaccade, but also the driver's blink frequency, the follow-up eye movement time, and the rapid eye movement frequency. By estimating the driver state using the amount of change, it becomes possible to estimate the driver state more accurately.

Specifically, by using the blink frequency, it is possible to prevent the decrease in the rapid eye movement frequency due to the blink from being estimated as an abnormality in the driver's condition. Further, by using the follow-up eye movement time, it is possible to prevent the decrease in the frequency of rapid eye movement due to the follow-up eye movement from being estimated as an abnormality in the driver's state. Furthermore, when the amount of change in the frequency of rapid eye movements is used, it becomes possible not to presume that the decrease in the frequency of rapid eye movements when the frequency of rapid eye movements tends to increase is an abnormality in the driver's condition. Therefore, the driver's state can be estimated more accurately than when only the rapid eye movement frequency is used. In addition, the driver's state can be estimated more accurately than when a part of the driver's blink frequency, the following eye movement time, and the rapid eye movement frequency change amount is used.

(Embodiment 2)
In the first embodiment, the configuration in which the driver's follow-up eye movement is extracted based on the line-of-sight angular velocity or the line-of-sight angular acceleration is shown, but the present invention is not necessarily limited to this. For example, the configuration may be such that the driver's follow-up eye movement is extracted by estimating the occurrence of the driver's follow-up eye movement based on the position and / or speed of the gaze object with respect to the own vehicle (hereinafter, embodiment 2). Hereinafter, the configuration of the second embodiment will be described.

The vehicle system 1 of the second embodiment is the same as the vehicle system 1 of the first embodiment except that the HCU 10a is included instead of the HCU 10. Here, a schematic configuration of the HCU 10a will be described with reference to FIG.

As shown in FIG. 8, as functional blocks, the HCU 10a includes an image recognition processing unit 100, an eye movement extraction unit 110a, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140, and a notification processing unit 150. And the object information acquisition unit 160 is provided. The HCU 10a is the same as the HCU 10 except that the object information acquisition unit 160 is provided and the eye movement extraction unit 110a is provided instead of the eye movement extraction unit 110. This HCU 10a also corresponds to the state estimation device of the claim.

The object information acquisition unit 160 acquires object information that is at least one of the position and speed of the gaze object around the vehicle with respect to the vehicle. As an example, the position and / or speed of the gaze object with respect to the own vehicle detected by the peripheral monitoring ECU 40 may be acquired as the object information. When there are a plurality of gaze objects, the object information about the plurality of gaze objects may be acquired.

The eye movement extraction unit 110a includes a blinking eye movement extraction unit 111, a rapid eye movement extraction unit 112, and a follow-up eye movement extraction unit 113a as sub-functional blocks. The eye movement extraction unit 110a is the same as the eye movement extraction unit 110 except that the follow-up eye movement extraction unit 113a is provided instead of the follow-up eye movement extraction unit 113.

The following eye movement extraction unit 113a extracts the driver's following eye movement by estimating the occurrence of the driver's following eye movement based on the object information acquired by the object information acquisition unit 160. Therefore, the following eye movement extraction unit 113a corresponds to the occurrence estimation unit of the claim. When the gaze object is located within the visual field range and the speed difference with respect to the own vehicle is larger than a certain level, the driver causes a follow-up eye movement with respect to the gaze object. Therefore, when the gaze object is located in front of the own vehicle or the speed difference of the gaze object with respect to the own vehicle is larger than a certain level, there is a high probability that the following eye movement will occur. Therefore, the driver's follow-up eye movement may be extracted by estimating the occurrence of the driver's follow-up eye movement under such a case.

The configuration of the second embodiment is the same as that of the first embodiment except that the mode of extracting the follow-up eye movement is different, so that the same effect as that of the first embodiment can be obtained.

(Embodiment 3)
In the first embodiment, the determination threshold value for the follow-up eye movement time is determined from the average value or the degree of variation of the follow-up eye movement time for each predetermined time t within the effective time T, but the present invention is not necessarily limited to this. .. For example, it may be configured to determine the determination threshold value for the following eye movement time based on the position and / or speed of the gaze object with respect to the own vehicle (hereinafter, embodiment 3). Hereinafter, the configuration of the third embodiment will be described.

The vehicle system 1 of the third embodiment is the same as the vehicle system 1 of the first embodiment except that the HCU 10b is included instead of the HCU 10. Here, the schematic configuration of the HCU 10b will be described with reference to FIG.

As shown in FIG. 9, the HCU 10b includes an image recognition processing unit 100, an eye movement extraction unit 110, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140b, and a notification processing unit 150 as functional blocks. And the object information acquisition unit 160 is provided. The HCU 10b is the same as the HCU 10 except that the object information acquisition unit 160 is provided and the state estimation unit 140b is provided instead of the state estimation unit 140. This HCU10b also corresponds to the state estimation device of the claim.

The object information acquisition unit 160 is the same as the object information acquisition unit 160 of the second embodiment. The state estimation unit 140b includes a determination threshold value determination unit 141b as a sub-function block. The state estimation unit 140b is the same as the state estimation unit 140 except that the determination threshold value determination unit 141b is provided instead of the determination threshold value determination unit 141.

The determination threshold value determination unit 141b determines the determination threshold value for the follow-up eye movement time according to the object information acquired by the object information acquisition unit 160. Further, the determination threshold value determination unit 141b changes the determination threshold value for the follow-up eye movement time determined in the same manner as the determination threshold value determination unit 141 of the first embodiment according to the object information acquired by the object information acquisition unit 160. The configuration may be determined by doing so.

As an example, the faster the speed of the gaze object with respect to the own vehicle, the higher the probability that the follow-up eye movement time becomes shorter. The configuration may be such that the threshold value is determined to be small. In addition, when the gaze object is far away from the own vehicle, there is a high probability that the follow-up eye movement time will be longer. The determination threshold value for the follow-up eye movement time may be largely determined according to the decrease in the value. In addition, as the number of gaze objects around the vehicle increases, there is a high probability that the tracking eye movement time for one gaze object will become shorter. As the number increases, the determination threshold value for the follow-up eye movement time may be determined to be smaller.

Depending on the position and / or speed of the gaze object with respect to the own vehicle, the tendency of the length of time during which the follow-up eye movement occurs when the driver of the own vehicle gazes at the gaze object changes. According to the configuration of the third embodiment, by determining the determination threshold value for the follow-up eye movement time according to the object information acquired by the object information acquisition unit 160, the follow-up eye movement that changes according to the object information It becomes possible to determine the determination threshold value for the follow-up eye movement time according to the tendency of the length of the occurrence time. Therefore, it is possible to further improve the accuracy of the determination threshold value for the follow-up eye movement time and further improve the estimation accuracy of the driver's state.

The configuration of the third embodiment is the same as that of the first embodiment except that the determination threshold value for the following eye movement time is changed, so that the same effect as that of the first embodiment can be obtained. .. Further, the configuration of the third embodiment may be combined with the first embodiment or the second embodiment.

(Embodiment 4)
In the first embodiment, the determination threshold value for the blink frequency is determined from the average value or the degree of variation of the blink frequency for each predetermined time t within the effective time T, but the present invention is not necessarily limited to this. For example, the configuration may be such that the determination threshold value for the blink frequency is determined based on the humidity and / or brightness in the interior of the own vehicle (hereinafter referred to as the fourth embodiment). Hereinafter, the configuration of the fourth embodiment will be described.

The vehicle system 1 of the fourth embodiment is the same as the vehicle system 1 of the first embodiment except that the HCU 10c is included instead of the HCU 10. Here, the schematic configuration of the HCU 10c will be described with reference to FIG.

As shown in FIG. 10, as functional blocks, the HCU 10b includes an image recognition processing unit 100, an eye movement extraction unit 110, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140c, and a notification processing unit 150. It also has an indoor environment information acquisition unit 170. The HCU 10c is the same as the HCU 10 except that it includes an indoor environment information acquisition unit 170 and a state estimation unit 140c instead of the state estimation unit 140. This HCU 10c also corresponds to the state estimation device of the claim.

The indoor environment information acquisition unit 170 acquires indoor environment information which is the humidity and / or brightness of the interior of the own vehicle. As an example, the illuminance detected by the external light sensor of the vehicle sensor 20 and / or the humidity detected by the humidity sensor of the vehicle sensor 20 may be acquired as indoor environment information.

The state estimation unit 140c includes a determination threshold value determination unit 141c as a sub-function block. The state estimation unit 140c is the same as the state estimation unit 140 except that the determination threshold value determination unit 141c is provided instead of the determination threshold value determination unit 141.

The determination threshold value determination unit 141c determines the determination threshold value for the blink frequency according to the indoor environment information acquired by the indoor environment information acquisition unit 170. Further, the determination threshold value determination unit 141c changes the determination threshold value for the blink frequency determined in the same manner as the determination threshold value determination unit 141 of the first embodiment according to the indoor environment information acquired by the indoor environment information acquisition unit 170. The configuration may be determined by the above. As an example, the determination threshold value for the blink frequency may be largely determined as the illuminance detected by the external light sensor increases. Further, the determination threshold value for the blink frequency may be significantly changed according to the decrease in the humidity detected by the humidity sensor.

The tendency of the blink frequency of the driver of the own vehicle changes depending on the humidity and / or the brightness of the interior of the own vehicle. For example, the lower the humidity, the more blinks due to drying, and the higher the frequency of blinks. In addition, the greater the brightness, the greater the number of blinks, and the higher the frequency of blinks. According to the configuration of the fourth embodiment, by changing the determination threshold value for the blink frequency according to the indoor environment information acquired by the indoor environment information acquisition unit 170, the tendency of the blink frequency to change according to the indoor environment information. It is possible to change the determination threshold value for the blink frequency according to the above. Therefore, it is possible to further improve the accuracy of the determination threshold value for the blink frequency and further improve the estimation accuracy of the driver's state.

The configuration of the fourth embodiment is the same as that of the first embodiment except that the determination threshold value for the blink frequency is changed, so that the same effect as that of the first embodiment can be obtained. Further, the configuration of the fourth embodiment may be combined with the configurations of the first to third embodiments.

(Embodiment 5)
In the above-described embodiment, the configuration obtained by calculating the rapid eye movement frequency, the blink frequency, the follow-up eye movement time, and the amount of change in the rapid eye movement frequency by HCU10, 10a, 10b, 10c is shown, but this is not necessarily the case. Not limited to. For example, the HCU 10, 10a, 10b, 10c acquires the rapid eye movement frequency, the blink frequency, the follow-up eye movement time, and / or the amount of change in the rapid eye movement frequency calculated by a device other than the HCU 10, 10a, 10b, 10c. It may be configured. Further, the functions of the HCUs 10, 10a, 10b, and 10c may be provided by a plurality of devices.

(Embodiment 6)
In the above-described embodiment, the state estimation units 140, 140b, 140c show a configuration in which the driver's state is estimated using the blink frequency, the follow-up eye movement time, and the rapid eye movement frequency change amount, but this is not necessarily the case. Not exclusively. For example, only a part of the blink frequency, the follow-up eye movement time, and the rapid eye movement frequency change amount may be used for estimating the driver's state. Of the blink frequency, follow-up eye movement time, and rapid eye movement frequency change amount, the rapid eye movement frequency change amount can be calculated using the calculation result of the rapid eye movement frequency, so that the rapid eye movement frequency change amount is also included. Even if the configuration is used to estimate the state of the driver, there is an advantage that the increase in new arithmetic processing can be suppressed and the processing load can be suppressed.

(Embodiment 7)
In the above-described embodiment, the configuration in which the state estimation-related processing is executed every time the power of the HCUs 10, 10a, 10b, and 10c is turned on is shown, but the present invention is not necessarily limited to this. For example, the state estimation-related processing may be executed every time the user inputs an operation input for executing the state estimation-related processing to the operation input unit provided in the own vehicle. According to this, by accepting this operation input from the driver every time the driver of the own vehicle changes, the judgment threshold value corresponding to each driver is determined, and the state of the driver is estimated accurately according to each driver. Will be possible.

Further, by registering identification information such as an electronic key ID different for each driver in the memory of the vehicle system 1, the identification information for each driver obtained from the electronic key ID by short-range wireless communication can be used. It may be configured to execute the state estimation related processing every time the driver is changed. Even in this case, it is possible to determine the determination threshold value according to each driver and accurately estimate the state of the driver according to each driver.

(Embodiment 8)
In the above-described embodiment, the case where the present invention is applied to a vehicle has been described as an example, but the present invention is not necessarily limited to this. For example, it may be applied to a moving body other than a vehicle to estimate the state of an occupant of the moving body other than the vehicle. Further, the configuration may be applied to other than the moving body. For example, it may be configured to estimate the state of workers on a factory line or the like.

The present invention is not limited to the above-described embodiment, and various modifications can be made within the scope of the claims, and the present invention can be obtained by appropriately combining the technical means disclosed in the different embodiments. Embodiments are also included in the technical scope of the present invention.

1 Vehicle system, 10, 10a, 10b, 10c HCU (state estimation device), 20 vehicle sensor, 30 peripheral monitoring sensor, 40 peripheral monitoring ECU, 50 driver imaging unit, 100 image recognition processing unit, 110 eye movement extraction unit, 111 Instantaneous eye movement extraction unit, 112 Rapid eye movement extraction unit, 113 Follow-up eye movement extraction unit, 113a Follow-up eye movement extraction unit (occurrence estimation unit), 120 Eye movement information storage unit, 130 Judgment data calculation unit, 131 Instantaneous frequency calculation Unit (flash acquisition unit), 132 Rapid eye movement frequency calculation unit (frequency acquisition unit), 133 Follow-up eye movement time calculation unit (following eye movement acquisition unit), 134 Change amount calculation unit (frequency change acquisition unit), 135 Reinforcement information acquisition unit, 140, 140b, 140c state estimation unit, 141, 141b, 141c judgment threshold determination unit, 150 notification processing unit, 160 object information acquisition unit, 170 indoor environment information acquisition unit

Claims (13)

A state estimation device that estimates the state of the monitored person based on the eye movements of the monitored person.
A frequency acquisition unit (132) for acquiring the frequency of rapid eye movement, which is the frequency of occurrence of at least one of the saccade and microsaccade of the monitored person, and
The blink acquisition unit (131) for acquiring the frequency of occurrence of the blink of the monitored person, the following eye movement acquisition unit (133) for acquiring the occurrence time of the follow-up eye movement of the monitored person, and the monitoring target person. The reinforcement information acquisition unit (135), which is at least one of the frequency change acquisition units (134) for acquiring the change amount of the rapid eye movement frequency,
In addition to the rapid eye movement frequency acquired by the frequency acquisition unit, a state estimation unit (140, 140b, 140c) that estimates the state of the monitored person by using the information acquired by the reinforcement information acquisition unit is provided. ,
The reinforcement information acquisition unit is at least the blink acquisition unit, and is
The state estimation unit estimates that the state of the monitored subject is normal when the rapid eye movement frequency acquired by the frequency acquisition unit is larger than the threshold value for the rapid eye movement frequency, while the frequency is estimated. Even when the rapid eye movement frequency acquired by the acquisition unit is equal to or less than the threshold value for the rapid eye movement frequency, the occurrence frequency of the blink acquired by the blink acquisition unit is greater than the threshold value for the occurrence frequency. If even larger, the monitoring subject of the state state estimating device you presumed to be normal. The state estimation unit, wherein the threshold for frequency of occurrence of the blink, to claim 1 for determining from the data set of the actual frequency of the blink in a certain period in the past which were sequentially acquired by the blink acquisition unit State estimator. It is used in a vehicle and estimates the state of the driver of the vehicle as the person to be monitored.
The vehicle is provided with an indoor environment information acquisition unit (170) that acquires indoor environment information that is at least one of the humidity and brightness of the interior of the vehicle.
The state estimation unit (140c) according to claim 1 or 2 , wherein the state estimation unit (140c) sequentially determines and updates the threshold value for the occurrence frequency of the blinks according to the indoor environment information sequentially acquired by the indoor environment information acquisition unit. State estimator. The reinforcement information acquisition unit is at least the follow-up eye movement acquisition unit, and is
Even when the rapid eye movement frequency acquired by the frequency acquisition unit is equal to or less than the threshold value for the rapid eye movement frequency, the state estimation unit obtains the follow-up eye movement of the follow-up eye movement acquisition unit. The state estimation device according to any one of claims 1 to 3 , wherein when the occurrence time is larger than the threshold value for the occurrence time, the state of the monitored person is estimated to be normal. A state estimation device that estimates the state of the monitored person based on the eye movements of the monitored person.
A frequency acquisition unit (132) for acquiring the frequency of rapid eye movement, which is the frequency of occurrence of at least one of the saccade and microsaccade of the monitored person, and
The blink acquisition unit (131) for acquiring the frequency of occurrence of the blink of the monitored person, the following eye movement acquisition unit (133) for acquiring the occurrence time of the follow-up eye movement of the monitored person, and the monitoring target person. The reinforcement information acquisition unit (135), which is at least one of the frequency change acquisition units (134) for acquiring the change amount of the rapid eye movement frequency,
In addition to the rapid eye movement frequency acquired by the frequency acquisition unit, a state estimation unit (140, 140b, 140c) that estimates the state of the monitored person by using the information acquired by the reinforcement information acquisition unit is provided. ,
The reinforcement information acquisition unit is at least the follow-up eye movement acquisition unit, and is
The state estimation unit estimates that the state of the monitored subject is normal when the rapid eye movement frequency acquired by the frequency acquisition unit is larger than the threshold value for the rapid eye movement frequency, while the frequency is estimated. Even when the rapid eye movement frequency acquired by the acquisition unit is equal to or less than the threshold value for the rapid eye movement frequency, the occurrence time of the follow-up eye movement acquired by the follow-up eye movement acquisition unit is related to the occurrence time. If greater than the threshold, the state estimating device status of the monitored person estimated to be normal. The state estimation unit determines a threshold value for the occurrence time of the follow-up eye movement from a data group of the actual occurrence time of the follow-up eye movement in a certain past period sequentially acquired by the follow-up eye movement acquisition unit. The state estimation device according to 4 or 5. It is used in a vehicle and estimates the state of the driver of the vehicle as the person to be monitored.
An object information acquisition unit (160) that acquires object information that is at least one of the position and speed of the gaze object around the vehicle with respect to the vehicle.
It is provided with an occurrence estimation unit (113a) that estimates the occurrence of follow-up eye movements of the monitored subject based on the object information acquired by the object information acquisition unit.
The state estimation device according to any one of claims 4 to 6, wherein the following eye movement acquisition unit acquires the occurrence time of the following eye movement of the monitored person based on the estimation result of the occurrence estimation unit. .. It is used in a vehicle and estimates the state of the driver of the vehicle as the person to be monitored.
It is provided with an object information acquisition unit (160) for acquiring object information which is at least one of the position and speed of the gaze object around the vehicle with respect to the vehicle.
The state estimation unit (140b) determines the threshold value for the occurrence time of the follow-up eye movement according to the object information acquired by the object information acquisition unit according to any one of claims 4 to 7. The described state estimator. The reinforcement information acquisition unit is at least the frequency change amount acquisition unit.
Even when the rapid eye movement frequency acquired by the frequency acquisition unit is equal to or less than the threshold value for the rapid eye movement frequency, the state estimation unit acquires the frequency change amount acquisition unit of the monitored person. wherein when the change amount of the rapid eye movement frequency is increased greater than the threshold for the amount of change is according to any one of claims 1 to 8, the state of the monitored person is estimated to be normal State estimator. A state estimation device that estimates the state of the monitored person based on the eye movements of the monitored person.
A frequency acquisition unit (132) for acquiring the frequency of rapid eye movement, which is the frequency of occurrence of at least one of the saccade and microsaccade of the monitored person, and
The blink acquisition unit (131) for acquiring the frequency of occurrence of the blink of the monitored person, the following eye movement acquisition unit (133) for acquiring the occurrence time of the follow-up eye movement of the monitored person, and the monitoring target person. The reinforcement information acquisition unit (135), which is at least one of the frequency change acquisition units (134) for acquiring the change amount of the rapid eye movement frequency,
In addition to the rapid eye movement frequency acquired by the frequency acquisition unit, a state estimation unit (140, 140b, 140c) that estimates the state of the monitored person by using the information acquired by the reinforcement information acquisition unit is provided. ,
The reinforcement information acquisition unit is at least the frequency change amount acquisition unit.
The state estimation unit estimates that the state of the monitored subject is normal when the rapid eye movement frequency acquired by the frequency acquisition unit is larger than the threshold value for the rapid eye movement frequency, while the frequency is estimated. Even when the rapid eye movement frequency acquired by the acquisition unit is equal to or less than the threshold value for the rapid eye movement frequency, the change amount of the rapid eye movement frequency of the monitored person acquired by the frequency change acquisition unit. There the case is an increase greater than a threshold for the amount of change, the state estimating device status of the monitored person estimated to be normal. The state estimation unit determines a threshold value for the amount of change in the rapid eye movement frequency from the data group of the actual amount of change in the rapid eye movement frequency in the past fixed period sequentially acquired by the frequency change acquisition unit. The state estimation device according to claim 9 or 10. The state estimation unit uses the information acquired by the reinforcement information acquisition unit for the latest fixed period in addition to the rapid eye movement frequency acquired by the frequency acquisition unit for the latest fixed period, and uses the information of the monitored person. The state estimation device according to any one of claims 1 to 11, wherein the state is sequentially estimated. Any one of claims 1 to 12, wherein the state estimation unit determines a threshold value for the rapid eye movement frequency from the data group of the actual rapid eye movement frequency in the past fixed period sequentially acquired by the frequency acquisition unit. The state estimation device according to item 1.

Images