JP2018183532A - State estimation apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To make it possible to more accurately estimate a state of a monitoring target person on the basis of eye movement of the monitoring target person.SOLUTION: A state estimation apparatus comprises: a rapid eye movement frequency calculation unit 132 for calculating a rapid eye movement frequency, an occurrence frequency of at least any of a saccade and micro saccade of a driver; a nictation frequency calculation unit 131 for calculating a nictation frequency of the driver; a tracking eye movement time calculation unit 133 for calculating an occurrence time of tracking eye movement of the driver; a variation calculation unit 134 for calculating a variation in the rapid eye movement frequency of the driver; and a state estimation unit 140 for estimating a state of the driver using the nictation frequency, tracking eye movement time, and rapid eye movement frequency variation in addition to the rapid eye movement frequency.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a state estimation device that estimates the state of a monitoring target person based on the eye movement of the monitoring target person.

  In Patent Document 1, an abnormality occurs when the total value of the normal saccade frequency indicating the occurrence frequency of the saccade of the monitoring target and the microsaccade frequency indicating the occurrence frequency of the microsaccade of the monitoring target is less than a threshold value. A technique for presuming that is is disclosed. That is, a technique is disclosed that estimates that the occurrence frequency of saccade and microsaccade is abnormal when it is less than a threshold value.

JP 2017-23519 A

  However, even if the occurrence frequency of saccades and microsaccades is low, the monitoring subject may not be abnormal. For example, the occurrence frequency of saccades and microsaccades is also reduced by blinking and / or following eye movements caused by the surrounding environment of the monitoring subject. Therefore, depending on the surrounding environment of the monitoring target person, even if the monitoring target person is normal, the occurrence frequency of saccades and microsaccades may decrease, and the estimation accuracy of the state of the monitoring target person may decrease.

  Even if the occurrence frequency of saccades and microsaccades is low, it is considered that the state of the monitoring subject has already recovered normally if there is a tendency to increase over time. Therefore, even in such a case, it is considered that the estimation accuracy of the state of the monitoring subject is lowered.

  One object of this disclosure is to provide a state estimation device that makes it possible to estimate the state of the monitoring subject with higher accuracy based on the eyeball movement of the monitoring subject.

  The above object is achieved by a combination of the features described in the independent claims, and the subclaims define further advantageous embodiments of the invention. Reference numerals in parentheses described in the claims indicate a correspondence relationship with specific means described in the embodiments described later as one aspect, and do not limit the technical scope of the present invention. .

  In order to achieve the above object, a state estimation device according to the present invention is a state estimation device that estimates a state of a monitoring subject based on eye movements of the monitoring subject, and includes a saccade and a microsaccade of the monitoring subject. A frequency acquisition unit (132) that acquires a rapid eye movement frequency that is at least one of the following, a blink acquisition unit (131) that acquires a blink occurrence frequency of the monitoring target person, and a tracking eye of the monitoring target person A reinforcing information acquisition unit (133) that is at least one of a follow-up eye movement acquisition unit (133) that acquires the occurrence time of movement and a frequency change amount acquisition unit (134) that acquires the amount of change in the rapid eye movement frequency of the monitoring subject. 135) and the state estimation unit (140, 140b, 140c) that estimates the state of the person to be monitored using information acquired by the reinforcement information acquisition unit in addition to the rapid eye movement frequency acquired by the frequency acquisition unit. Provided with a door.

  According to this, at least one of the occurrence frequency of the blink of the monitoring subject, the occurrence time of the tracking eye movement of the monitoring subject, and the amount of change in the occurrence frequency of the saccade and microsaccade of the monitoring subject. It becomes possible to estimate the state of the person being monitored. When the occurrence frequency of blinks of the monitoring target person is used, it is possible not to estimate the decrease in the rapid eye movement frequency due to the blinking as an abnormality in the state of the monitoring target person. When the generation time of the follow-up eye movement of the monitoring target person is used, it is possible not to estimate the decrease in the rapid eye movement frequency due to the follow-up eye movement as an abnormal state of the monitoring target person. When using the amount of change in the rapid eye movement frequency of the monitored person, do not estimate the decrease in the rapid eye movement frequency when the rapid eye movement frequency is increasing as an abnormal condition of the monitored person. Is possible. Therefore, it is possible to estimate the state of the monitoring subject with higher accuracy than when only the occurrence frequency of the saccade and microsaccade of the monitoring subject is used. As a result, it is possible to estimate the state of the monitoring subject with higher accuracy based on the eye movement of the monitoring subject.

1 is a diagram illustrating an example of a schematic configuration of a vehicle system 1. FIG. It is a figure which shows an example of a schematic structure of HCU10. It is a figure for demonstrating determination of the determination threshold value about rapid eye movement frequency. It is a flowchart which shows an example of the flow of the state estimation related process in HCU10. It is a flowchart which shows an example of the flow of the eye movement extraction process in HCU10. It is a flowchart which shows an example of the flow of the determination threshold value determination process in HCU10. It is a flowchart which shows an example of the flow of the state estimation process in HCU10. It is a figure which shows an example of a schematic structure of HCU10a. It is a figure which shows an example of a schematic structure of HCU10b. It is a figure which shows an example of a schematic structure of HCU10c.

  A plurality of embodiments for disclosure will be described with reference to the drawings. For convenience of explanation, among the embodiments, parts having the same functions as those shown in the drawings used in the explanation so far may be given the same reference numerals and explanation thereof may be omitted. is there. For the parts denoted by the same reference numerals, the description in other embodiments can be referred to.

(Embodiment 1)
<Schematic configuration of vehicle system 1>
A vehicle system 1 shown in FIG. 1 is used in a moving body such as a vehicle, and includes an HCU (Human Machine Interface Control Unit) 10, a vehicle sensor 20, a periphery monitoring sensor 30, a periphery monitoring ECU 40, a driver imaging unit 50, and a notification. A device 60 is included. It is assumed that the HCU 10, the vehicle sensor 20, and the periphery monitoring ECU 40 are connected to an in-vehicle LAN, for example. Below, the case where the vehicle system 1 is used by a motor vehicle is mentioned as an example, and description is continued.

  The vehicle sensor 20 is a sensor group for detecting the state of the host vehicle. Examples of the vehicle sensor 20 include a humidity sensor that detects the humidity in the cabin of the host vehicle, an external light sensor that detects the brightness in front of the host vehicle, and a steering angle sensor that detects the steering angle of the host vehicle. As an example, an illuminance sensor that detects illuminance may be used as the external light sensor. The vehicle sensor 20 outputs the sensing result to the in-vehicle LAN. Note that the sensing result of the vehicle sensor 20 may be output to the vehicle LAN via an ECU mounted on the host vehicle.

  The surrounding monitoring sensor 30 detects objects around the vehicle such as pedestrians, moving animals such as animals other than humans, vehicles, and still objects such as signs, guardrails, curbs, trees, and the like. In addition, road markings such as travel lane lines and stop lines around the vehicle are detected. The peripheral monitoring sensor 30 is, for example, a peripheral monitoring camera that images a predetermined range around the host vehicle, a millimeter wave radar that transmits an exploration wave to the predetermined range around the host vehicle, sonar, LIDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging). The peripheral monitoring camera sequentially outputs captured images to be sequentially captured to the peripheral monitoring ECU 40 as sensing information. A sensor that transmits an exploration wave such as sonar, millimeter wave radar, or LIDAR sequentially outputs a scanning result based on a reception signal obtained when a reflected wave reflected by an obstacle is received as sensing information to the periphery monitoring ECU 40.

  The peripheral monitoring ECU 40 includes a microcomputer including a processor, a volatile memory, a nonvolatile memory, and a bus that interconnects these, and executes various processes by executing a control program stored in the nonvolatile memory. To do. The surrounding monitoring ECU 40 recognizes the surrounding environment of the host vehicle from the sensing result acquired from the surrounding monitoring sensor 30.

  The surrounding monitoring ECU 40 detects an object around the vehicle as the surrounding environment of the vehicle from the sensing result of the surrounding monitoring camera. As an example, it is preferable to detect an object that is a gaze target of the driver by distinguishing it from an object that is not a gaze target by a known image recognition process such as template matching. As an example of an object to be watched by the driver (hereinafter referred to as a watched object), there are a pedestrian, another vehicle, a sign, and the like. In addition, the periphery monitoring ECU 40 detects a relative position and a relative speed of an object existing around the vehicle from the sensing result of the periphery monitoring sensor by a known method. In addition, the periphery monitoring ECU 40 determines the vehicle's own vehicle from the information on the position and speed of the target object such as other vehicles and pedestrians obtained by the inter-vehicle communication and / or the road-vehicle communication via the communication module used in the own vehicle. You may detect the relative position and relative speed with respect to the own vehicle about the object which exists in the periphery.

  The driver imaging unit 50 includes an imaging device for imaging the driver's head. As an example, a configuration including an illumination device and an imaging device having sensitivity in the same band as the illumination by the illumination device may be used. As a specific example, a configuration including a near-infrared camera and a near-infrared light source may be used. Below, the case where this near-infrared camera and a near-infrared light source are used is mentioned as an example, and is demonstrated. When extracting a microsaccade described later, a high-performance camera having a spatial resolution of 0.01 degrees and a temporal resolution of 220 Hz may be used, for example.

  The near-infrared camera is arranged, for example, on a steering column cover in a posture directed toward the driver's seat side of the own vehicle. Note that the near-infrared camera may be arranged at other positions as long as it can capture the face of the driver seated in the driver's seat of the host vehicle, such as the upper surface of the instrument panel. It may be a configuration. The driver imaging unit images a driver's head irradiated with near-infrared light from a near-infrared light source with a near-infrared camera. An image captured by the near-infrared camera (hereinafter referred to as a face image) is output to the HCU 10.

  The notification device 60 notifies the driver of the own vehicle. Examples of the notification device 60 include a display device, an audio output device, and a vibrator. The notification by the notification device 60 includes display by a display device, sound output by a sound output device, vibration by a vibrator, and the like.

  The HCU 10 is mainly composed of a processor, a volatile memory, a non-transitory tangible storage medium nonvolatile memory, an I / O, and a microcomputer having a bus connecting them, and a driver. It is connected to the imaging unit 50, the notification device 60, and the in-vehicle LAN. The HCU 10 executes the control program stored in the non-volatile memory, thereby estimating the state of the driver of the host vehicle or notifying the driver of the host vehicle. The HCU 10 corresponds to the state estimation device in the claims, and the driver corresponds to the person to be monitored in the claims.

<Schematic configuration of HCU10>
Here, the schematic configuration of the HCU 10 will be described with reference to FIG. As shown in FIG. 2, the HCU 10 includes, as functional blocks, an image recognition processing unit 100, an eye movement extraction unit 110, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140, and a notification processing unit 150. It has. Note that part or all of the functions executed by the HCU 10 may be configured by hardware using one or a plurality of ICs. Further, part or all of the functional blocks provided in the HCU 10 may be realized by a combination of execution of software by a processor and hardware members.

  The image recognition processing unit 100 performs extraction of the driver's face part, extraction of the facial feature of the face part, calculation of the line-of-sight angle, and the like from the face image sequentially output from the driver imaging unit 50 by known image recognition processing. Perform sequentially. The line-of-sight angle can be rephrased as the line-of-sight direction. Hereinafter, an example of image recognition processing performed by the image recognition processing unit 100 will be described.

  First, the image recognition processing unit 100 extracts eyes in the face image and extracts a black circle in the eyes as a pupil. Subsequently, a white point in the eye in the face image is extracted as a corneal reflection image. Then, the line-of-sight angle with respect to the reference position in the passenger compartment is detected from the positional relationship between the pupil and the corneal reflection. The reference position may be, for example, the installation position of the near infrared camera. The line-of-sight angle referred to here is the line-of-sight angle in both the horizontal direction and the vertical direction, which approximately coincide with the left-right direction of the vehicle. The calculated line-of-sight angle may be stored in a memory such as a volatile memory.

  The eye movement extraction unit 110 performs eye movement extraction processing for extracting the eye movement of the driver of the own vehicle based on the result of the image recognition processing in the image recognition processing unit 100. As shown in FIG. 2, the eye movement extraction unit 110 includes a blinking eye extraction unit 111, a rapid eye movement extraction unit 112, and a following eye movement extraction unit 113 as sub-function blocks.

  The blink extraction unit 111 extracts a driver's blink. A blink can be rephrased as a blink. The blink extraction unit 111 may be configured to extract, for example, a case where the pupil cannot be extracted in the image recognition processing sequentially performed by the image recognition processing unit 100. In addition, it may be configured to extract blinks based on the curvature radius of the driver's upper eyelid extracted by the image recognition processing unit 100. The blink extraction unit 111 stores, as the blink extraction result, the time when the blink is extracted in the eye movement information storage unit 120 until, for example, the valid time T elapses. The eye movement information storage unit 120 may be configured to use a volatile memory, for example. The effective time T is a time necessary for accurately calculating the occurrence frequency of rapid eye movements such as saccade and microsaccade, which will be described later, and may be, for example, about 1 to several minutes. The same applies to the following.

  The rapid eye movement extraction unit 112 extracts rapid eye movements such as a driver's saccade and microsaccade. Saccades are short-term, high-speed eye movements that occur when a person intentionally changes the line-of-sight direction. Microsaccade is a rapid eye movement that is very fine among the involuntary eye movements that occur involuntarily when a person is gazing. The rapid eye movement extraction unit 112 may be configured to extract only one of saccades and microsaccades, or may be configured to extract both.

  The rapid eye movement extraction unit 112 calculates a change amount of the line-of-sight angle from the line-of-sight angles for the past several frames (for example, 10 frames) calculated by the image recognition processing unit 100, and divides this change amount by time. The gaze angular velocity is calculated. The rapid eye movement extraction unit 112 calculates the gaze angular acceleration by dividing the change amount of the gaze angular velocity calculated as described above by time.

Then, the rapid eye movement extraction unit 112 extracts the saccade when the gaze angular acceleration is equal to or greater than the reference angular acceleration and the gaze movement distance is equal to or greater than the reference distance. On the other hand, the rapid eye movement extraction unit 112 extracts microsaccades when the gaze angular acceleration is equal to or greater than the reference angular acceleration and the gaze movement distance is less than the reference distance. The reference angular acceleration may be 1200 deg / s ² , for example, and the reference distance may be ² deg, for example. In addition, it is good also as a structure which replaces with a gaze angular acceleration and makes determination by whether a gaze angular velocity is more than a reference | standard angular velocity. The rapid eye movement extraction unit 112 may store the time when the saccade and microsaccade are extracted as the saccade and microsaccade extraction results in the eye movement information storage unit 120 until, for example, an effective time T elapses.

The following eye movement extracting unit 113 extracts the following eye movement of the driver. The following eye movement is an eye movement that makes the object follow the line of sight reflexively. The follow-up eye movement extraction unit 113 extracts the follow-up eye movement as the follow-up eye movement when the gaze angular velocity per fixed time is within a fixed range. As an example, when the visual axis angular velocity per ^second is 0.5 to 2.0 deg / s ² , the eye movement is extracted as the following eye movement. The following eye movement extraction unit 113 may be configured to make a determination based on whether or not the gaze angular acceleration is within a certain range instead of the gaze angular velocity. The follow-up eye movement extraction unit 113 may store, as the follow-up eye movement extraction result, whether or not the follow-up eye movement is extracted in the eye movement information storage unit 120 according to time until, for example, the effective time T elapses.

  The determination data calculation unit 130 acquires the determination data by calculating the determination data used to estimate the driver state. As shown in FIG. 2, the determination data calculation unit 130 includes, as sub function blocks, a blink frequency calculation unit 131, a rapid eye movement frequency calculation unit 132, a following eye movement time calculation unit 133, and a change amount calculation unit 134. Prepare. The blink frequency calculation unit 131, the following eye movement time calculation unit 133, and the change amount calculation unit 134 correspond to the reinforcement information acquisition unit 135.

  The blink frequency calculation unit 131 generates the blink frequency of the driver (hereinafter, referred to as “blink occurrence frequency”) based on the time when the blink is extracted as the extraction result of the blink extraction unit 111 accumulated in the eye movement information storage unit 120. The blink frequency is obtained by calculating the blink frequency). The blink frequency calculation unit 131 corresponds to the blink acquisition unit in the claims. As a specific example, the blink frequency calculation unit 131 may calculate the blink frequency per predetermined time t from the extraction result of the blink extraction unit 111 within the predetermined time t every predetermined time t. The predetermined time t is about 10 seconds, for example. The same applies to the following. The blink frequency calculation unit 131 accumulates the blink frequency to be sequentially calculated in the eye movement information storage unit 120 in time series.

  The rapid eye movement frequency calculating unit 132 extracts the saccade of the driver based on the time when the saccade and the micro saccade extracted from the rapid eye movement extraction unit 112 accumulated in the eye movement information storage unit 120 are extracted. , The rapid eye movement frequency is obtained by calculating the occurrence frequency of microsaccade (hereinafter, rapid eye movement frequency). The rapid eye movement frequency calculation unit 132 corresponds to a frequency acquisition unit in the claims.

  As a specific example, if the rapid eye movement frequency calculation unit 132 calculates the rapid eye movement frequency per predetermined time t from the extraction result of the rapid eye movement extraction unit 112 within the predetermined time t every predetermined time t. Good. The rapid eye movement frequency calculation unit 132 accumulates the rapid eye movement frequency that is sequentially calculated in the eye movement information storage unit 120 in time series. In addition, it is preferable to improve the calculation accuracy of the rapid eye movement frequency by excluding the data of the time when the blink has been extracted by the blink extraction unit 111 from the calculation process of the rapid eye movement frequency.

  The follow-up eye movement time calculation unit 133 determines the follow-up eye movement of the driver based on the presence / absence of the follow-up eye movement by time, which is the extraction result of the follow-up eye movement extraction unit 113 accumulated in the eye movement information storage unit 120. The follow-up eye movement time is acquired by calculating the generation time (hereinafter, follow-up eye movement time). The generation time of the following eye movement can be rephrased as the time required for the following eye movement. The following eye movement time calculation unit 133 corresponds to the following eye movement acquisition unit. As a specific example, the tracking eye movement time calculation unit 133 calculates the tracking eye movement time per predetermined time t from the extraction result of the tracking eye movement extraction unit 113 within the predetermined time t for each predetermined time t. Good. The follow-up eye movement time calculation unit 133 accumulates the follow-up eye movement time sequentially calculated in the eye movement information storage unit 120 in time series.

  The change amount calculation unit 134 changes the rapid eye movement frequency based on the time when the saccade and the microsaccade extracted from the rapid eye movement extraction unit 112 accumulated in the eye movement information storage unit 120 are extracted. The amount of rapid eye movement frequency change is acquired by calculating the amount (hereinafter, the amount of rapid eye movement frequency change). The change amount calculation unit 134 corresponds to a frequency change amount acquisition unit in claims. As a specific example, when the rapid eye movement frequency calculation unit 132 calculates the rapid eye movement frequency for each predetermined time t, the number of occurrences of saccades and microsaccades for s seconds immediately before the predetermined time t is calculated. Here, s seconds are shorter than the predetermined time t and may be, for example, 2 to 3 seconds. Then, the ratio of the number of occurrences of saccades and microsaccades in the immediately preceding s seconds to the total number of occurrences within a predetermined time t is calculated as a rapid eye movement frequency change amount. The change amount calculation unit 134 accumulates the rapid eye movement frequency change amount sequentially calculated in the eye movement information storage unit 120 in time series.

  The state estimation unit 140 performs a determination threshold value determination process for determining a determination threshold value used for estimating the driver state from the data group of determination data sequentially calculated by the determination data calculation unit 130. Then, state estimation processing for estimating the state of the driver is performed from the determination data sequentially calculated by the determination data calculation unit 130 and the determination threshold. As illustrated in FIG. 2, the state estimation unit 140 includes a determination threshold value determination unit 141 as a sub functional block.

  The determination threshold value determination unit 141 determines a determination threshold value for the blink frequency, a determination threshold value for the rapid eye movement frequency, a determination threshold value for the following eye movement time, and a determination threshold value for the rapid eye movement frequency change amount. I do.

  The determination threshold value for the blink frequency in the determination threshold value determination unit 141 is determined as follows as an example. The determination threshold value determination unit 141 stores an average value of blink frequency for each predetermined time t within the effective time T calculated by the blink frequency calculation unit 131 (hereinafter, average blink) accumulated in the eye movement information storage unit 120. Frequency). Then, a determination threshold for the blink frequency is determined from the calculated average blink frequency. In other words, the determination threshold for the blink frequency is determined from the data group of the actual blink occurrence frequency in a past fixed period calculated by the blink frequency calculation unit 131.

  For example, when the average blink frequency is less than 1 / second, the determination threshold is 0.5 times / second, and when the average blink frequency is 1 to 2 times, the determination threshold is 1 / second. When the blink frequency is 2 times or more, the determination threshold is 2 times / second. Further, it may be configured such that the variation in blink frequency for each predetermined time t is calculated, and the determination threshold is determined to be small as the degree of variation increases.

  The determination threshold for the rapid eye movement frequency in the determination threshold determination unit 141 is determined as follows as an example. Based on the rapid eye movement frequency for each predetermined time t within the effective time T calculated by the rapid eye movement frequency calculation unit 132 accumulated in the eye movement information storage unit 120, the determination threshold value determination unit 141 is a rapid eye movement. A determination threshold for the exercise frequency is determined. In other words, the determination threshold for the blink frequency is determined from the data group of the actual rapid eye movement frequency in the past fixed period calculated by the rapid eye movement frequency calculation unit 132.

  When only one of the saccade occurrence frequency and the microsaccade occurrence frequency is used as the rapid eye movement frequency, a value obtained by lowering the predetermined shift amount from the average value of the occurrence frequency is determined. What is necessary is just to set it as a threshold value. This shift amount may be determined in advance by experiments or the like so that the estimated accuracy of the driver state becomes a desired accuracy. Moreover, it is good also as a structure which determines a determination threshold value small according to the degree of the dispersion | variation in the occurrence frequency for every predetermined time t becoming large.

  On the other hand, when both the saccade frequency and the microsaccade frequency are used as the rapid eye movement frequency, a determination is made using a regression equation obtained from the saccade frequency and the microsaccade frequency. What is necessary is just to determine a threshold value. The regression equation may be obtained as follows. First, on the graph with the horizontal and vertical axes representing the occurrence frequency of saccade and the occurrence frequency of microsaccade, respectively, the occurrence frequency of saccade and the occurrence frequency of microsaccade for each predetermined time t within the effective time T. Is plotted (see FIG. 3). Subsequently, a line A such as a most probable straight line or curve indicating the tendency of these plots is obtained. For example, the most probable line A (see A in FIG. 3) may be obtained by the least square method. Then, the regression equation B (see B in FIG. 3) is obtained by shifting the line A toward the origin of the graph. The boundary indicated by the regression equation B is a determination threshold value.

  Note that the shift amount for shifting the line A in the direction of the origin may be determined in advance by experiments or the like so that the estimated accuracy of the driver state becomes a desired accuracy. Moreover, it is good also as a structure which enlarges the amount of shift according to the degree of the dispersion | variation in the generation | occurrence | production frequency of saccade and microsaccade for every predetermined time t becoming large.

  Determination of the determination threshold for the following eye movement time in the determination threshold determination unit 141 is performed as follows as an example. The determination threshold value determination unit 141 stores the average value of the following eye movement time for each predetermined time t within the effective time T calculated by the following eye movement time calculation unit 133 accumulated in the eye movement information storage unit 120 (hereinafter, average). Follow-up eye movement time) is calculated. Then, a determination threshold for the following eye movement time is determined from the calculated average following eye movement time. In other words, the determination threshold value for the tracking eye movement time is determined from the data group of the actual generation time of the tracking eye movement in the past fixed period calculated by the tracking eye movement time calculation unit 133. For example, the determination threshold value determination unit 141 may set a value that is lower than the average tracking eye movement time by a predetermined shift amount as the determination threshold value. This shift amount may be determined in advance by experiments or the like so that the estimated accuracy of the driver state becomes a desired accuracy. Moreover, it is good also as a structure which determines a determination threshold value small according to the degree of the dispersion | variation in the follow-up eyeball movement time for every predetermined time t becoming large.

  Determination of the determination threshold for the rapid eye movement frequency change amount by the determination threshold determination unit 141 is performed as follows as an example. The determination threshold value determination unit 141 stores the average value of the rapid eye movement frequency change amount for each predetermined time t within the effective time T calculated by the change amount calculation unit 134 and accumulated in the eye movement information storage unit 120 (hereinafter, average) Change amount). Then, a determination threshold for the rapid eye movement frequency change amount is determined from the calculated average change amount. In other words, the determination threshold for the rapid eye movement frequency change amount is determined from the data group of the actual rapid eye movement frequency change amount in the past fixed period calculated by the change amount calculation unit 134. For example, the determination threshold value determination unit 141 may set a value obtained by lowering the average change amount by a predetermined shift amount as the determination threshold value. This shift amount may be determined in advance by experiments or the like so that the estimated accuracy of the driver state becomes a desired accuracy. Moreover, it is good also as a structure which determines a determination threshold value small according to the degree of dispersion | variation in the average variation | change_quantity for every predetermined time t becoming large.

  In the state estimation process of the state estimation unit 140, when the rapid eye movement frequency sequentially calculated by the rapid eye movement frequency calculation unit 132 is equal to or higher than the determination threshold for the rapid eye movement frequency determined by the determination threshold determination unit 141, It is determined that the driver status is normal. Further, in the state estimation processing of the state estimation unit 140, even if the rapid eye movement frequency sequentially calculated by the rapid eye movement frequency calculation unit 132 is less than the determination threshold for the rapid eye movement frequency, the blink frequency calculation Each of the blink threshold frequency sequentially calculated by the unit 131, the following eye movement time sequentially calculated by the following eye movement time calculating unit 133, and the rapid eye movement frequency change amount sequentially calculated by the change amount calculating unit 134 is determined by each of the determination threshold values. If it is above, it is determined that the state of the driver is normal.

  On the other hand, the state estimation unit 140 is a case where the rapid eye movement frequency sequentially calculated by the rapid eye movement frequency calculation unit 132 is less than a determination threshold for the rapid eye movement frequency, and the blink frequency calculation unit 131 When the blinking frequency calculated sequentially, the tracking eye movement time sequentially calculated by the tracking eye movement time calculation unit 133, and the rapid eye movement frequency change amount sequentially calculated by the change amount calculation unit 134 are less than the respective determination thresholds. Then, it is determined that the state of the driver is abnormal. In the present embodiment, as an example, it is assumed that the concentration state where the driver is concentrated is normal, and the careless state where the driver's attention is reduced is abnormal.

  The notification processing unit 150 instructs the notification device 60 to perform notification when the state estimation unit 140 estimates that the driver's state is careless. The notification may be anything that allows the driver to recognize that it is an inattention state, for example, an alarm.

<State estimation related processing in HCU10>
Next, an example of the flow of processing related to driver state estimation (hereinafter referred to as state estimation related processing) in the HCU 10 will be described using the flowcharts of FIGS. 4 to 7. The flowchart in FIG. 4 may be configured to start when the power supply of the HCU 10 is turned on and to end when the power supply of the HCU 10 is turned off. When the power supply of the HCU 10 is switched according to the on / off of a switch for starting the internal combustion engine or motor generator of the own vehicle (hereinafter referred to as a power switch), it starts when the power switch is turned on and ends when the power switch is turned off. What is necessary is just to be the structure to do.

  First, in step S <b> 1, the image recognition processing unit 100 performs image recognition processing from the face image acquired from the driver imaging unit 50. The image recognition processing unit 100 extracts a face part of the driver, extracts a shape feature of the face part, calculates a gaze angle, and the like by this image recognition process.

  In step S2, the eye movement extraction unit 110 performs eye movement extraction processing, and proceeds to step S3. In the eye movement extraction process, when an eye movement to be extracted occurs, this eye movement is extracted. Here, an outline of the eye movement extraction process will be described with reference to the flowchart of FIG.

  First, in step S21, the blink extraction unit 111 extracts a driver's blink. In step S <b> 22, the rapid eye movement extraction unit 112 calculates the gaze angular velocity based on the gaze angles for the past several frames calculated by the image recognition processing unit 100. In step S23, the rapid eye movement extraction unit 112 calculates the gaze angular acceleration based on the gaze angular velocity described above.

  In step S24, the rapid eye movement extraction unit 112 extracts a saccade when the gaze angular acceleration calculated in S23 is greater than or equal to the reference angular acceleration and the gaze movement distance is greater than or equal to the reference distance. In step S <b> 25, the rapid eye movement extraction unit 112 extracts a microsaccade when the gaze angular acceleration is equal to or greater than the reference angular acceleration and the gaze movement distance is less than the reference distance. In step S26, the following eye movement extracting unit 113 extracts the driver's following eye movement, and the process proceeds to step S3. Note that the order of the processes in S24 to S26 may be changed.

  Returning to FIG. 4, in step S <b> 3, the eye movement data extracted by the eye movement extraction process of S <b> 2 is stored in the eye movement information storage unit 120 until the valid time T elapses. In step S4, when the valid time T has elapsed since the start of the state estimation related process (YES in S4), the process proceeds to step S5. If the valid time T has not elapsed since the start of the state estimation related process (NO in S4), the process returns to S1 and is repeated. Whether or not the valid time T has elapsed since the start of the state estimation-related processing may be determined by the state estimation unit 140 based on, for example, a count by a timer circuit.

  In step S5, when the determination threshold value determination unit 141 has already determined the determination threshold value, the process proceeds to step S7. On the other hand, when the determination threshold value determination unit 141 has not determined the determination threshold value, the process proceeds to step S6. In step S6, the determination threshold value determination unit 141 performs determination threshold value determination processing, and proceeds to step S7. Here, an outline of the determination threshold value determination process will be described using the flowchart of FIG.

  First, in step S61, the determination threshold value determination unit 141 determines a determination threshold value for the rapid eye movement frequency. In step S62, the determination threshold value determination unit 141 determines a determination threshold value for the blink frequency. In step S63, the determination threshold value determination unit 141 determines a determination threshold value for the following eye movement. In step S64, the determination threshold value determination unit 141 determines a determination threshold value for the rapid eye movement change amount, and proceeds to step S7. Note that the order of the processing of S61 to S64 may be changed.

  In the state estimation related process, the determination threshold determination process is performed only once, and after determining the determination threshold, the state estimation process may be performed using the determined determination threshold. In this embodiment, after the power of the HCU 10 is turned on, the determination threshold value determination process is performed once to determine the determination threshold value, and state estimation is performed using the determined determination threshold value until the power supply of the HCU 10 is turned off. Do. According to this, since the determination threshold value is newly determined for each trigger for starting the state estimation related processing such as turning on the power switch of the own vehicle and turning on the power of the HCU 10, a determination threshold value suitable for the driver is determined for each boarding. It becomes possible to redo. Therefore, even if the driver's physical condition varies from day to day, a determination threshold value that matches the physical condition of the driver of the day can be determined, and the estimation accuracy of the driver's state can be improved. Therefore, it is possible to improve the estimation accuracy of the driver's state while suppressing the processing load for re-determining the determination threshold value while the host vehicle is traveling.

  Returning to FIG. 4, in step S <b> 7, the state estimation unit 140 performs a state estimation process to estimate the state of the driver, and proceeds to step S <b> 8. Here, an outline of the state estimation process will be described using the flowchart of FIG. In the state estimation process, determination data for each predetermined time t is calculated, and the driver state is estimated by comparing the determination data with the determination threshold value determined in S6. In this embodiment, the driver's attention is estimated.

  First, in S <b> 71, the state estimation unit 140 acquires the saccade / microsaccade occurrence frequency (that is, rapid eye movement frequency) of the driver at the latest predetermined time t calculated by the rapid eye movement frequency calculation unit 132. The rapid eye movement frequency may be the occurrence frequency of either saccade or microsaccade, but here it is the sum of the occurrence frequency of saccade and the occurrence frequency of microsaccade. Subsequently, the state estimation unit 140 compares the rapid eye movement frequency of the latest predetermined time t with the determination threshold value for the rapid eye movement frequency determined in S6. If the rapid eye movement frequency is greater than the determination threshold (YES in S71), the process proceeds to step S75. On the other hand, when the rapid eye movement frequency is equal to or lower than the determination threshold value (NO in S71), the process proceeds to step S72.

  In step S72, the state estimation unit 140 acquires the blink frequency of the driver at the latest predetermined time t calculated by the blink frequency calculation unit 131, and a determination threshold for the blink frequency and the blink frequency determined in S6. And compare. If the blink frequency is greater than the determination threshold (YES in S72), the process proceeds to step S75. On the other hand, when the blink frequency is equal to or lower than the determination threshold value (NO in S72), the process proceeds to step S73.

  In step S73, the state estimation unit 140 acquires the driver's following eye movement time at the latest predetermined time t calculated by the following eye movement time calculation unit 133, and the following eye movement time determined in S6. Is compared with the determination threshold for. If the follow-up eye movement time is longer than the determination threshold (YES in S73), the process proceeds to step S75. On the other hand, when the follow-up eye movement time is equal to or less than the determination threshold value (NO in S73), the process proceeds to step S74.

  In step S74, the state estimation unit 140 acquires the rapid eye movement frequency change amount of the driver for the most recent predetermined time t calculated by the change amount calculation unit 134, and the rapid eye movement frequency change amount determined in S6. The determination threshold for the amount of change in eye movement frequency is compared. If the rapid eye movement frequency change amount is larger than this determination threshold (YES in S74), the process proceeds to step S75. On the other hand, when the rapid eye movement frequency change amount is equal to or smaller than the determination threshold value (NO in S74), the process proceeds to step S76.

  In step S75, the state estimation unit 140 estimates that the driver state is normal, that is, a concentrated state, and proceeds to step S8. That is, the state estimation unit 140 concentrates the driver state in a concentrated state when any of the rapid eye movement frequency, the blink frequency, the following eye movement time, and the rapid eye movement frequency change amount is larger than each determination threshold. Presume that there is.

  On the other hand, in step S76, the state estimation unit 140 estimates that the driver is in an abnormal state, that is, an inattention state, and proceeds to step S8. In other words, the state estimating unit 140 cares about the driver's state when the rapid eye movement frequency, the blink frequency, the following eye movement time, and the rapid eye movement frequency change amount are all equal to or less than the respective determination thresholds. Estimated. Note that the processing order of S71 to S74 may be changed.

  Returning to FIG. 4, in step S8, when the driver state is estimated to be careless in S7 (YES in S8), the process proceeds to step S9. On the other hand, when the driver state is estimated to be a concentrated state in S7 (NO in S8), the process proceeds to step S10. In step S9, the notification processing unit 150 instructs the notification device 60 to issue an alarm, and proceeds to step S10.

  In step S10, if it is the end timing of the state estimation related process (YES in S10), the state estimation related process is ended. On the other hand, when it is not the end timing of the state estimation related process (NO in S10), the process returns to S1 and the process is repeated. As the end timing of the state estimation related process, for example, when the power supply of the HCU 10 is turned off, the power switch of the own vehicle is turned off, or the like.

<Summary of Embodiment 1>
According to the configuration of the first embodiment, not only the rapid eye movement frequency that is the occurrence frequency of at least one of the driver's saccade and microsaccade, but also the driver's blink frequency, the following eye movement time, and the rapid eye movement frequency. By estimating the driver state using the amount of change, the driver state can be estimated more accurately.

  Specifically, by using the blink frequency, it is possible to prevent a decrease in the rapid eye movement frequency due to the blink from being estimated as an abnormal state of the driver. In addition, by using the following eye movement time, it is possible to prevent a decrease in the rapid eye movement frequency due to the following eye movement from being estimated as an abnormal state of the driver. Further, when the rapid eye movement frequency change amount is used, it is possible not to estimate a decrease in the rapid eye movement frequency when the rapid eye movement frequency tends to increase as an abnormal state of the driver. Therefore, the driver's state can be estimated with higher accuracy than when only the rapid eye movement frequency is used. In addition, the driver's state can be estimated more accurately than when using a part of the driver's blink frequency, follow-up eye movement time, and rapid eye movement frequency change amount.

(Embodiment 2)
In the first embodiment, the configuration in which the driver's following eyeball movement is extracted based on the gaze angular velocity or the gaze angular acceleration is shown, but the configuration is not necessarily limited thereto. For example, the configuration may be such that the driver's following eye movement is extracted by estimating the generation of the driver's following eye movement based on the position and / or speed of the gaze target with respect to the vehicle (hereinafter, Embodiment 2). Hereinafter, the configuration of the second embodiment will be described.

  The vehicle system 1 according to the second embodiment is the same as the vehicle system 1 according to the first embodiment except that the vehicle system 1 includes the HCU 10a instead of the HCU 10. Here, the schematic configuration of the HCU 10a will be described with reference to FIG.

  As shown in FIG. 8, the HCU 10a includes, as functional blocks, an image recognition processing unit 100, an eye movement extraction unit 110a, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140, a notification processing unit 150, And an object information acquisition unit 160. The HCU 10a is the same as the HCU 10 except that the HCU 10a includes an object information acquisition unit 160 and an eye movement extraction unit 110a instead of the eye movement extraction unit 110. The HCU 10a also corresponds to the state estimation device in the claims.

  The target object information acquisition unit 160 acquires target object information that is at least one of the position and speed of the target object around the host vehicle with respect to the host vehicle. As an example, the configuration may be such that the position and / or speed of the gaze target with respect to the host vehicle detected by the periphery monitoring ECU 40 is acquired as the target information. In the case where there are a plurality of gaze objects, the object information for the plurality of gaze objects may be acquired.

  The eye movement extraction unit 110a includes a blink extraction unit 111, a rapid eye movement extraction unit 112, and a tracking eye movement extraction unit 113a as sub-function blocks. The eye movement extraction unit 110a is the same as the eye movement extraction unit 110 except that it includes a tracking eye movement extraction unit 113a instead of the tracking eye movement extraction unit 113.

  The following eye movement extraction unit 113a extracts the following eye movement of the driver by estimating the occurrence of the following eye movement of the driver based on the object information acquired by the object information acquisition unit 160. Therefore, the following eye movement extraction unit 113a corresponds to the generation estimation unit in the claims. When the gaze object is located within the visual field range and the speed difference with respect to the own vehicle is larger than a certain level, the driver causes the following eye movement to the gaze object. Therefore, when the gaze target is located in front of the own vehicle or the speed difference of the gaze target with respect to the own vehicle is larger than a certain level, the probability that the following eye movement will occur is high. Therefore, the driver's following eye movement may be extracted by estimating the generation of the driver's following eye movement under such a condition.

  The configuration of the second embodiment is the same as that of the first embodiment except that the follow-up eye movement extraction mode is different, so that the same effect as the configuration of the first embodiment can be obtained.

(Embodiment 3)
In the first embodiment, the determination threshold for the following eye movement time is determined from the average value or the degree of variation of the following eye movement time for each predetermined time t within the effective time T. However, the present invention is not limited to this. . For example, it is good also as a structure (henceforth Embodiment 3) which determines the determination threshold value about tracking eyeball movement time based on the position and / or speed with respect to the own vehicle of a gaze target object. Hereinafter, the configuration of the third embodiment will be described.

  The vehicle system 1 of the third embodiment is the same as the vehicle system 1 of the first embodiment except that the vehicle system 1 includes the HCU 10b instead of the HCU 10. Here, the schematic configuration of the HCU 10b will be described with reference to FIG.

  As shown in FIG. 9, the HCU 10b includes, as functional blocks, an image recognition processing unit 100, an eye movement extraction unit 110, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140b, a notification processing unit 150, And an object information acquisition unit 160. The HCU 10 b is the same as the HCU 10 except that the HCU 10 b includes the object information acquisition unit 160 and a state estimation unit 140 b instead of the state estimation unit 140. The HCU 10b also corresponds to the state estimation device in the claims.

  The object information acquisition unit 160 is the same as the object information acquisition unit 160 of the second embodiment. The state estimation unit 140b includes a determination threshold value determination unit 141b as a sub functional block. The state estimation unit 140b is the same as the state estimation unit 140 except that it includes a determination threshold value determination unit 141b instead of the determination threshold value determination unit 141.

  The determination threshold value determination unit 141b determines a determination threshold value for the following eye movement time according to the object information acquired by the object information acquisition unit 160. Further, the determination threshold value determination unit 141b changes the determination threshold value for the follow-up eye movement time determined in the same manner as the determination threshold value determination unit 141 of the first embodiment according to the object information acquired by the object information acquisition unit 160. It is good also as a structure determined by doing.

  As an example, since the tracking eye movement time is likely to be shortened as the speed of the gaze target with respect to the vehicle increases, the determination of the tracking eye movement time according to the speed of the gaze object with respect to the vehicle increases. What is necessary is just to set it as the structure which determines a threshold value small. In addition, when the target object is far away from the host vehicle, the tracking eye movement time is likely to be long, so the degree of approach of the target object to the host vehicle from the position and speed of the target object relative to the host vehicle is high. It is good also as a structure which determines largely the determination threshold value about a follow-up eye movement time according to becoming low. In addition, since the tracking eye movement time for one gaze target is likely to be shorter as the number of gaze objects around the vehicle increases, the number of gaze objects around the vehicle is determined from the position of the gaze target with respect to the own vehicle. It is good also as a structure which determines the determination threshold value about a following eyeball movement time small according to increasing.

  Depending on the position and / or speed of the gaze target with respect to the own vehicle, the tendency of the length of time of occurrence of the following eye movement that occurs when the driver of the own vehicle gazes at the gaze target changes. According to the configuration of the third embodiment, by determining the determination threshold value for the tracking eye movement time according to the object information acquired by the object information acquisition unit 160, the tracking eye movement that changes according to the object information is determined. It is possible to determine a determination threshold for the following eye movement time in accordance with the tendency of the generation time. Therefore, it is possible to further improve the accuracy of the determination threshold for the tracking eye movement time and further improve the estimation accuracy of the driver state.

  Note that the configuration of the third embodiment is the same as that of the first embodiment except that the determination threshold value for the tracking eye movement time is changed. Therefore, the same effect as the configuration of the first embodiment can be obtained. . The configuration of the third embodiment may be combined with the first or second embodiment.

(Embodiment 4)
In the first embodiment, the determination threshold for the blink frequency is determined based on the average value or the degree of variation of the blink frequency for each predetermined time t within the effective time T. However, the present invention is not limited to this. For example, it is good also as a structure (henceforth Embodiment 4) which determines the determination threshold value about blink frequency based on the humidity and / or brightness of the interior of the own vehicle. Hereinafter, the configuration of the fourth embodiment will be described.

  The vehicle system 1 according to the fourth embodiment is the same as the vehicle system 1 according to the first embodiment except that the vehicle system 1 includes the HCU 10c instead of the HCU 10. Here, a schematic configuration of the HCU 10c will be described with reference to FIG.

  As shown in FIG. 10, the HCU 10b includes, as functional blocks, an image recognition processing unit 100, an eye movement extraction unit 110, an eye movement information storage unit 120, a determination data calculation unit 130, a state estimation unit 140c, a notification processing unit 150, And an indoor environment information acquisition unit 170. The HCU 10 c is the same as the HCU 10 except that the indoor environment information acquisition unit 170 is provided and a state estimation unit 140 c is provided instead of the state estimation unit 140. The HCU 10c also corresponds to the state estimation device in the claims.

  The indoor environment information acquisition unit 170 acquires indoor environment information that is the humidity and / or brightness of the vehicle interior. As an example, the illuminance detected by the external light sensor of the vehicle sensor 20 and / or the humidity detected by the humidity sensor of the vehicle sensor 20 may be acquired as room environment information.

  The state estimation unit 140c includes a determination threshold value determination unit 141c as a sub-function block. The state estimation unit 140c is the same as the state estimation unit 140 except that a determination threshold value determination unit 141c is provided instead of the determination threshold value determination unit 141.

  The determination threshold value determination unit 141c determines a determination threshold value for the blink frequency according to the indoor environment information acquired by the indoor environment information acquisition unit 170. In addition, the determination threshold value determination unit 141c changes the determination threshold value for the blink frequency determined in the same manner as the determination threshold value determination unit 141 of the first embodiment according to the indoor environment information acquired by the indoor environment information acquisition unit 170. It is good also as a structure determined by this. As an example, a configuration may be adopted in which the determination threshold for the blink frequency is largely determined as the illuminance detected by the external light sensor increases. Moreover, it is good also as a structure which changes the determination threshold value about blink frequency largely according to the humidity detected with a humidity sensor becoming low.

  The tendency of the blink frequency of the driver of the vehicle changes depending on the humidity and / or brightness of the vehicle interior. For example, the blink frequency increases because the blinking increases as the humidity decreases. In addition, the blink frequency increases because the blink increases as the brightness increases. According to the configuration of the fourth embodiment, by changing the determination threshold for the blink frequency according to the indoor environment information acquired by the indoor environment information acquisition unit 170, the tendency of the blink frequency that changes according to the indoor environment information. In accordance with this, it is possible to change the determination threshold for the blink frequency. Therefore, it is possible to further improve the accuracy of the determination threshold for the blink frequency and further improve the accuracy of estimating the driver state.

  The configuration of the fourth embodiment is the same as that of the first embodiment except that the determination threshold for the blink frequency is changed, and thus the same effect as the configuration of the first embodiment can be obtained. Further, the configuration of the fourth embodiment may be combined with the configurations of the first to third embodiments.

(Embodiment 5)
In the above-described embodiment, the configuration in which the rapid eye movement frequency, the blink frequency, the tracking eye movement time, and the rapid eye movement frequency change amount are obtained by calculating with the HCUs 10, 10a, 10b, and 10c is shown. Not limited to. For example, the HCU 10, 10a, 10b, 10c acquires the rapid eye movement frequency, blink frequency, follow-up eye movement time, and / or rapid eye movement frequency change amount calculated by a device other than the HCU 10, 10a, 10b, 10c. It is good also as a structure. The functions of the HCUs 10, 10a, 10b, and 10c may be configured to be performed by a plurality of devices.

(Embodiment 6)
In the above-described embodiment, the state estimation units 140, 140b, and 140c are configured to estimate the driver state using the blink frequency, the following eye movement time, and the rapid eye movement frequency change amount. Not exclusively. For example, only a part of the blink frequency, the following eye movement time, and the rapid eye movement frequency change amount may be used for estimating the driver state. Of the blink frequency, follow-up eye movement time, and rapid eye movement frequency change amount, the rapid eye movement frequency change amount can be calculated using the calculation result of the rapid eye movement frequency. Even when the configuration is used to estimate the state of the driver, there is an advantage that an increase in new arithmetic processing can be suppressed and the processing load can be suppressed.

(Embodiment 7)
In the above-described embodiment, the configuration in which the state estimation-related processing is executed every time the power of the HCUs 10, 10a, 10b, and 10c is turned on has been described. For example, it is good also as a structure which performs a state estimation related process, whenever a user performs operation input for performing a state estimation related process in the operation input part provided in the own vehicle. According to this, by receiving this operation input from the driver every time the driver of the own vehicle changes, a determination threshold value corresponding to each driver is determined, and the state of the driver is accurately estimated according to each driver. It becomes possible.

  In addition, by registering identification information such as an electronic key ID that is different for each driver in the memory of the vehicle system 1, based on the identification information for each driver obtained from the electronic key ID by short-range wireless communication, The state estimation related process may be executed every time the driver is changed. Even in this case, it is possible to determine a determination threshold corresponding to each driver and accurately estimate the driver state according to each driver.

(Embodiment 8)
In the above-described embodiment, the case where the present invention is applied to a vehicle has been described as an example, but the present invention is not necessarily limited thereto. For example, it is good also as a structure applied to moving bodies other than a vehicle and estimating the passenger | crew's state of moving bodies other than a vehicle. Moreover, it is good also as a structure applied other than a moving body. For example, it is good also as a structure which estimates the state of the worker in a factory line etc.

  Note that the present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. Embodiments are also included in the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Vehicle system 10, 10a, 10b, 10c HCU (state estimation apparatus), 20 Vehicle sensor, 30 Perimeter monitoring sensor, 40 Perimeter monitoring ECU, 50 Driver imaging unit, 100 Image recognition process part, 110 Eye movement extraction part, 111 Blink extraction unit, 112 Rapid eye movement extraction unit, 113 Tracking eye movement extraction unit, 113a Tracking eye movement extraction unit (generation estimation unit), 120 Eye movement information storage unit, 130 Determination data calculation unit, 131 Blink frequency calculation Part (blink acquisition part), 132 rapid eye movement frequency calculation part (frequency acquisition part), 133 following eye movement time calculation part (following eye movement acquisition part), 134 change amount calculation part (frequency change amount acquisition part), 135 Reinforcement information acquisition unit, 140, 140b, 140c state estimation unit, 141, 141b, 141c determination threshold value determination unit, 150 Notification processing unit, 160 object information acquisition unit, 170 indoor environment information acquisition unit

Claims (13)

A state estimation device that estimates the state of the monitoring subject based on the eye movement of the monitoring subject,
A frequency acquisition unit (132) for acquiring a rapid eye movement frequency, which is an occurrence frequency of at least one of saccade and microsaccade of the monitoring target;
A blink acquisition unit (131) that acquires the occurrence frequency of blinks of the monitoring target person, a tracking eye movement acquisition unit (133) that acquires the generation time of the tracking eye movement of the monitoring target person, and the monitoring target person A reinforcement information acquisition unit (135) that is at least one of a frequency change amount acquisition unit (134) that acquires a change amount of the rapid eye movement frequency;
In addition to the rapid eye movement frequency acquired by the frequency acquisition unit, a state estimation unit (140, 140b, 140c) that estimates the state of the monitoring target person using information acquired by the reinforcement information acquisition unit. State estimation device.   In addition to the rapid eye movement frequency in the latest fixed period acquired by the frequency acquisition unit, the state estimation unit uses the information of the latest fixed period acquired by the reinforcement information acquisition unit, and The state estimation apparatus according to claim 1, wherein the state is sequentially estimated.   The said state estimation part estimates that the said monitoring subject's state is normal, when the said rapid eye movement frequency acquired in the said frequency acquisition part is larger than the threshold value about the rapid eye movement frequency. 2. The state estimation device according to 2.   The state estimation unit according to claim 3, wherein the state estimation unit determines a threshold for the rapid eye movement frequency from a data group of actual rapid eye movement frequency in a past fixed period sequentially acquired by the frequency acquisition unit. apparatus. The reinforcement information acquisition unit is at least the blink acquisition unit,
The state estimation unit may generate the blinking frequency acquired by the blink acquisition unit even when the rapid eye movement frequency acquired by the frequency acquisition unit is equal to or less than a threshold value for the rapid eye movement frequency. The state estimation device according to claim 3 or 4 that estimates that the state of the person to be monitored is normal when is greater than a threshold value regarding the occurrence frequency.   The said state estimation part determines the threshold value about the occurrence frequency of the said blink from the data group of the actual occurrence frequency of the said blink in the past fixed period acquired sequentially by the said blink acquisition part. State estimation device. Used in a vehicle to estimate the state of the driver of the vehicle as the person to be monitored,
An indoor environment information acquisition unit (170) that acquires indoor environment information that is at least one of humidity and brightness in the vehicle interior;
The said state estimation part (140c) determines and updates the threshold value about the generation | occurrence | production frequency of the said blink sequentially according to the said indoor environment information acquired sequentially by the said indoor environment information acquisition part. State estimation device. The reinforcement information acquisition unit is at least the following eye movement acquisition unit,
Even if the rapid eye movement frequency acquired by the frequency acquisition unit is equal to or less than a threshold value for the rapid eye movement frequency, the state estimation unit is configured to acquire the tracking eye movement acquired by the tracking eye movement acquisition unit. The state estimation device according to any one of claims 3 to 7, wherein when the occurrence time is greater than a threshold value for the occurrence time, the state of the monitoring subject is estimated to be normal.   The said state estimation part determines the threshold value about the generation | occurrence | production time of the said tracking eye movement from the data group of the actual generation | occurrence | production time of the following eye movement in the past fixed period acquired sequentially by the said tracking eye movement acquisition part. The state estimation device according to claim 8. Used in a vehicle to estimate the state of the driver of the vehicle as the person to be monitored,
An object information acquisition unit (160) that acquires object information that is at least one of a position and a speed of a gaze object around the vehicle;
A generation estimation unit (113a) that estimates the generation of the following eye movement of the monitoring target person based on the target information acquired by the target information acquisition unit;
The state estimation device according to claim 8 or 9, wherein the tracking eye movement acquisition unit acquires a generation time of the tracking eye movement of the monitoring target person based on an estimation result in the generation estimation unit. It is used in a vehicle and estimates the state of the driver of the vehicle as the monitoring subject,
An object information acquisition unit (160) that acquires object information that is at least one of a position and a speed of a gaze object around the vehicle;
The said state estimation part (140b) determines the threshold value about the generation | occurrence | production time of the said following eye movement according to the said target object information acquired with the said target object information acquisition part. The state estimation apparatus described. The reinforcement information acquisition unit is at least the frequency change amount acquisition unit,
Even if the rapid eye movement frequency acquired by the frequency acquisition unit is equal to or less than a threshold value for the rapid eye movement frequency, the state estimation unit may acquire the monitoring target person acquired by the frequency change amount acquisition unit. 12. The method according to claim 3, wherein when the amount of change in the rapid eye movement frequency is greater than a threshold value for the amount of change, the state of the monitoring subject is estimated to be normal. State estimation device.   The state estimation unit determines a threshold for the amount of change in the rapid eye movement frequency from the data group of the actual amount of change in the rapid eye movement frequency in a past fixed period sequentially acquired by the frequency change amount acquisition unit. The state estimation apparatus according to claim 12.

Images