JP6910982B2 - Driver state estimation device and driver state estimation method - Google Patents

Description

The present invention relates to a driver state determination device for determining a driver state during vehicle driving (including simulated driving by a driving simulator) and a driver state determination method. The driver's state is a mental or sensory state (including at least one state of drowsiness, fatigue, and tension) of the driver while driving a vehicle, and refers to a state that a human feels in the brain.

In Patent Document 1, the direction of the driver's line of sight is detected based on an image captured by the camera of the driver's eye, and the movement time for the driver's line of sight to move to the position where the stimulus is generated within the range of the driver's vision is defined. A technique for determining a driver's arousal level based on the travel time by detecting the driver is disclosed. In Patent Document 2, the direction of the driver's line of sight is detected based on an image captured by the camera of the driver's eye, and the reaction time from when the judgment image is displayed on the display until the driver visually recognizes the judgment image. A technique for determining the driver state based on the reaction time by detecting the above is disclosed.

Patent Document 3 measures the saccade movement, which is the eye movement of the subject, based on the finding that the occurrence frequency of small-amplitude saccades is higher in the case of a human being in a loose state than in the normal state. A technique for determining that a determination target person is in a vague state based on conditions related to the amplitude and direction of a saccade motion is disclosed. Patent Document 4 discloses a technique of acquiring the occurrence frequency of saccades and the occurrence frequency of microsaccades in the eyeball of a monitored person and estimating the state of the monitored person based on these occurrence frequencies. ..

In Non-Patent Document 1, based on the fact that small saccade movements (so-called microsaccade movements) occur in the horizontal direction when the pupils are reduced due to decreased arousal, the amplitude and peak speed of the saccade movements (speed of the saccade movements). Shows a proportional relationship and the saccade interval (time interval of saccade movement of horizontal movement) tends to become shorter as the pupil area shrinks. The possibility of using it as an index for detection is being investigated.

Japanese Unexamined Patent Publication No. 2014-144906 (Abstract) JP-A-2017-204177 (Abstract) Japanese Unexamined Patent Publication No. 2011-115450 (Abstract, paragraph [0009]) JP-A-2017-23519 (Abstract, paragraph [0012])

"Detection of decreased arousal using saccade as an index" Hideki Wakui, Yutaka Hirata Biomedical Engineering 51 (6), 328-341, 2013

In the techniques disclosed in Patent Documents 1 and 2, the driver is given a visual stimulus (in Patent Document 1, a stimulus within the visual range of the driver, in Patent Document 2, a determination image displayed on a display), and the driver is provided. The movement time or reaction time of the line of sight is detected until the line of sight is moved to the visual stimulus and visually recognized. That is, the timing at which some visual stimulus is given to the driver is used as the start timing for measuring the movement time or reaction time of the line of sight, and the visual stimulus does not merely detect the direction of the driver's line of sight. It is necessary to detect the timing when it occurs.

On the other hand, Patent Documents 3 and 4 and Non-Patent Document 1 disclose techniques for measuring the eye movement of a subject to detect the saccade movement of the eyeball and determining the state of the subject from the detection result of the saccade movement. There is. However, the saccade movement is a very high-speed eye movement, which is also called a jumping eye movement, and is characterized in that the time required to move the viewpoint from one place to another is very short (20 ms to 80 ms). There is. For example, in Patent Document 4, in order to detect saccade movement, which is a high-speed eye movement, a high-speed and high-precision angle difference of 0.05 degree can be detected at a frame rate of about 200 fps (200 frames per second). It is stated that a camera is required. Further, for example, Non-Patent Document 1 describes that the pupil area and the binocular eyeball position of a subject are measured at 500 Hz by using a rapid eye movement analyzer in order to detect saccade movement.

In this way, a device that does not support saccade movement, which is a high-speed eye movement, for example, a low-performance camera (general in-vehicle camera, etc.) that captures images at a low frame rate of about 30 fps (30 frames per second). Then, it is difficult to detect the saccade movement. Therefore, when saccade movement is to be detected, it is necessary to provide a high-performance device corresponding to high-speed eye movement.

Furthermore, in each of the techniques described in the above-mentioned documents, the parameters of the movement time of the line of sight, the reaction time in which the direction of the line of sight changes, the interval and frequency of eye movements such as saccade movement, and the amplitude are set by the driver. It only makes the assumption that it correlates linearly with the tendency of state change.

In view of the above problems, the present invention observes a time-series coordinate space in which the line of sight stays instead of observing the saccade momentum. For example, the line of sight of a driver while driving a vehicle is observed by a low-performance camera. Even if it is detected, the driver's state is determined from the time-series data related to the line-of-sight motion generated based on the detection result, with the interval at which the line-of-sight stays, regardless of whether the line of sight is gaze or not. It is an object of the present invention to provide a driver state determination device (estimated) and a driver state determination method.

In order to achieve the above object, the driver state determination device of the present invention is a driver state determination device that determines the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle. hand,
A line-of-sight direction detection unit that detects the line-of-sight direction of the driver,
A line-of-sight movement distance detection unit that detects the movement distance of the driver's line of sight per predetermined time from a change over time in the direction of the driver's line of sight.
The time length of the time zone in which the moving distance of the driver's line of sight is relatively small is extracted as the line-of-sight retention interval, and the peak at which the change in the moving distance of the driver's line of sight with time is maximized is detected. A line-of-sight retention interval extraction unit that extracts the time length between adjacent peaks as the line-of-sight retention interval,
A driver state determination unit that identifies the tendency of the time-dependent change of the line-of-sight retention interval and determines the state of the driver from the specified tendency.
Have.
Further, in order to achieve the above object, the driver state determination device of the present invention is a driver state determination device that determines the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle. And
A line-of-sight direction detection unit that detects the line-of-sight direction of the driver,
A line-of-sight movement distance detection unit that detects the movement distance of the driver's line of sight per predetermined time from a change over time in the direction of the driver's line of sight.
A line-of-sight retention interval extraction unit that extracts the time length of a time zone in which the driver's line-of-sight movement distance is relatively small as a line-of-sight retention interval.
A driver state determination unit that identifies the tendency of the time-dependent change of the line-of-sight retention interval and determines the state of the driver from the specified tendency.
Have
The driver state determination unit calculates the Lyapunov exponent related to the line-of-sight residence interval by performing nonlinear analysis processing on the time-series data of the line-of-sight residence interval, and determines the state of the driver based on the Lyapunov exponent. It is configured.

Further, in order to achieve the above object, the driver state determination method of the present invention is a driver state determination method for determining the state of the driver including at least one state of drowsiness, fatigue and tension of the driver while driving a vehicle. And
The line-of-sight direction detection step for detecting the line-of-sight direction of the driver, and
A line-of-sight movement distance detection step that detects the movement distance of the driver's line-of-sight per predetermined time from a change over time in the direction of the driver's line-of-sight.
The time length of the time zone in which the moving distance of the driver's line of sight is relatively small is extracted as the line-of-sight retention interval, and the peak at which the change in the moving distance of the driver's line of sight with time is maximized is detected. A line-of-sight retention interval extraction step that extracts the time length between adjacent peaks as the line-of-sight retention interval, and
The driver state determination step of identifying the tendency of the time-dependent change of the line-of-sight retention interval and determining the state of the driver from the specified tendency.
Have.
Further, in order to achieve the above object, the driver state determination method of the present invention is used.
A driver state determination method for determining the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle.
The line-of-sight direction detection step for detecting the line-of-sight direction of the driver, and
A line-of-sight movement distance detection step that detects the movement distance of the driver's line-of-sight per predetermined time from a change over time in the direction of the driver's line-of-sight.
The line-of-sight retention interval extraction step for extracting the time length of the time zone in which the driver's line-of-sight movement distance is relatively small as the line-of-sight retention interval, and
The driver state determination step of identifying the tendency of the time-dependent change of the line-of-sight retention interval and determining the state of the driver from the specified tendency.
Have
In the driver state determination step, a non-linear analysis process is performed on the time-series data of the line-of-sight residence interval to calculate the Lyapunov exponent related to the line-of-sight residence interval, and the state of the driver is determined based on the Lyapunov exponent.

According to the present invention, for example, even when the line of sight of a driver while driving a vehicle is detected by a low-performance camera, the state of the driver can be determined from the time-series data related to the line of sight motion generated based on the detection result. It has the effect of being able to determine (estimate).

It is an image diagram for demonstrating the line-of-sight motion of a driver during driving of a vehicle used as a detection object in embodiment of this invention. It is a graph which shows an example of the time-series data of the movement distance of the driver's line of sight generated based on the detection result of the line-of-sight motion of the driver while driving a vehicle. It is a time-series data calculated based on the time-series data of the movement distance of a driver's line of sight, and is a graph which shows an example of the time-series data when the number of data points for calculating an average value is changed. It is a graph which shows an example of the time-series data of the average value (cumulative moving average) of the moving distance calculated from the time-series data of the moving distance of a driver's line of sight. It is a block diagram which shows an example of the structure of the driver state determination apparatus in embodiment of this invention. It is a flowchart which shows an example of the process executed by the driver state determination apparatus in embodiment of this invention. It is a figure for demonstrating the outline of the nonlinear analysis processing executed in the 3rd numerical analysis processing in embodiment of this invention.

Hereinafter, an outline and embodiments of the present invention will be described with reference to the drawings.

The driver's line-of-sight movement while driving a vehicle includes a state in which the line of sight stays in a specific space and a state in which the line of sight moves in the space. It can be said that the driver's line-of-sight movement while driving a vehicle is basically an intermittent glance movement, including the gaze act. The flickering movement is an act in which a driver while driving a vehicle directs his / her line of sight to various spaces for various purposes such as checking the running state of the vehicle, checking the safety of the surroundings, and relieving stress. For example, as shown in the image diagram of FIG. 1 (a view of the front of the vehicle from the vicinity of the driver's seat), the driver basically directs his / her line of sight to the space in front of the vehicle (A) and runs in parallel. Vehicle (B), left and right door mirrors (C) and (D), rearview mirror (E), speedometer (F), outside scenery (G) and (H) intermittently Drive the vehicle while looking at it. The driver while driving the vehicle basically looks at the space in front of the own vehicle (A), and the driver's line of sight draws a trajectory reciprocating between the space in front of the own vehicle (A) and other spaces. Often.

Although there are individual differences, differences in aging tendencies, and differences in vehicle speed while driving, the amount of time that the line of sight stays in such intermittent flickering movements is the physical condition of one individual (the fineness, variation, and distance traveled). It is presumed that it depends on how well you are. In other words, if there is fluctuation (richness of variation) in the time when the line of sight stays, it is presumed that it reflects the tendency of biological fluctuation (autonomic nerve activity), and the driver's physical condition and the tendency of autonomic nerve activity It can be said that there is a high possibility that the physiological state can be estimated.

Based on the above premise, the present invention focuses on the "line-of-sight retention interval" of the driver while driving the vehicle. The basic concept of the present invention is to detect the direction of the driver's line of sight while driving a vehicle, generate time-series data indicating changes in the line-of-sight movement over time from the detection result, and "retention of the line of sight" based on the time-series data. The tendency of "interval" is grasped, and the state of the driver is judged (estimated) from the tendency.

FIG. 2 is a graph showing an example of time-series data of the movement distance of the driver's line of sight, which is generated based on the detection result of the driver's line of sight movement while driving the vehicle. The vertical axis of the graph of FIG. 2 represents the moving distance of the driver's line of sight, and the horizontal axis represents the elapsed time. In the graph of FIG. 2, the moving distance of the driver's line of sight at each time is represented by a vertical bar in the elapsed time unit, and it is shown that the moving distance of the line of sight changes from moment to moment. Here, the state in which the length of the movement distance is alternately generated means that the movement in which the line of sight moves and stays in the space separated by a certain distance is repeatedly performed. It is considered that this is a state in which a flickering motion is performed in which the line of sight is moved by a certain distance. Taking the image diagram of FIG. 1 as an example, the driver is basically looking at the space in front of the vehicle (A), and is observing the surrounding disturbance by moving the line of sight to another space by glancing at the space. Is considered to be.

Further, FIG. 3 is time-series data calculated based on the time-series data of the movement distance of the driver's line of sight, and is an example of the time-series data when the number of data points for calculating the average value is changed. It is a graph which shows. In the graph of FIG. 3, the X-axis represents the elapsed time, the Y-axis represents the number of time window points, and the Z-axis represents the average value (cumulative moving average) with respect to the moving distance of the driver's line of sight. The number of time window points is the number of data sampled to calculate the average value. Here, the point interval of the window is set to 40 ms, and when N points are taken, the moving distance of N data included in the section of N × 40 ms is accumulated and divided by N (that is, the value). Average value) is expressed as the cumulative moving average of the Z axis. In FIG. 3, the graph when the number of time window points is one point (cross section when the number of time window points is one point) corresponds to the graph of FIG. Further, FIG. 4 shows an example of time-series data of the average value (cumulative moving average) of the moving distances calculated by taking a plurality of points (for example, 9 points).

As shown in FIGS. 3 and 4, when the average value (cumulative moving average) with respect to the moving distance of the driver's line of sight is taken, the motion peak of the line of sight transition is obtained. It can be considered that the motion peak of the line-of-sight transition obtained from this cumulative moving average corresponds to one unit of visual behavior (unit of patrol vision).

For example, when changing lanes, the driver performs a series of operations called patrol vision, such as visually observing the interruption space in the overtaking lane and the distance between the vehicle and the following vehicle, while paying attention to the front. implement. In patrol vision related to such visual behavior, the movement of the line of sight to a space distant to some extent increases, and the movement peak of the line-of-sight transition obtained when the cumulative moving average is calculated, and the work of patrol vision, which is a unit of visual behavior. Is associated with.

The time during which the driver's line of sight stays while driving the vehicle is, for example, the time during which the movement distance of the line of sight in the graph of FIG. 2 is low (low motion portion). This can be said to be a line-of-sight retention interval in a narrow sense, and can be read from the time-series data of the line-of-sight movement distance shown in the graph of FIG. In this case, various numerical values that can be read from the graph of FIG. 2 can be extracted as the line-of-sight retention interval. For example, the time from taking a moving distance exceeding a predetermined threshold value (which can be set as appropriate) to taking a moving distance exceeding the predetermined threshold value again may be defined as the line-of-sight retention interval. Further, the time during which the moving distance below a predetermined threshold value (which can be set as appropriate) continues may be defined as the line-of-sight retention interval.

On the other hand, as described with reference to FIGS. 3 and 4, the time between motion peaks obtained when the average value (cumulative moving average) regarding the movement distance of the driver's line of sight may be calculated as the line-of-sight retention interval. It is considered that this motion peak corresponds to patrol vision, which is a unit of visual behavior. Therefore, strictly speaking, the time between the peaks of the cumulative moving average is not the time during which the driver's line of sight stays while driving the vehicle, and the line of sight is more stable than when the patrol is performed. It can be said that it is time to do it. However, since the time between the peaks of the cumulative moving average is obtained when the average value is calculated for the moving distance, the time during which the driver's line of sight is staying while driving the vehicle (line-of-sight retention interval in a narrow sense). Similarly, it is inferred that there is a correlation between the variation and the tendency of biological fluctuation (autonomic nerve activity). Therefore, in the present specification, the time between the peaks of the cumulative moving average is also referred to as the (broadly defined) line-of-sight retention interval.

In the graphs of FIGS. 3 and 4, the interval between the points of the windows is 40 ms, and when viewed in a cumulative manner of 9 points (total 360 ms section set), the peak peaks (corresponding to patrol vision) vary. It turns out that it exists. In this way, when the cumulative moving average is calculated and the time series data is generated, there are clear peaks, and it is possible to easily extract the line-of-sight retention interval in a broad sense, which is the distance between the peaks.

Further, from the graph of FIG. 3, it can be seen that as the number of time window points gradually increases from one point, peak peaks corresponding to patrol vision appear. Specifically, the peaks of large peaks existing in the cumulative result of 9 points (total 360 ms section set) start to appear from 3 points (total 120 ms section set) or 4 points (total 160 ms section set). It is considered that the act of interest is carried out in 120 ms or 160 ms or more.

As described above, the "line-of-sight retention interval" in the present specification means that the direction of the line of sight is stable, and the movement distance or cumulative movement distance of the driver's line of sight (movement within a predetermined time range including a plurality of time windows). It is defined as meaning the length of a period in which the cumulative value of distance) is relatively small. Therefore, the "line-of-sight retention interval" in the present specification is at least the length of time in a state where the line-of-sight movement distance is low (low motion part) that can be read from the graph of FIG. 2, and the peak of the cumulative moving average that can be read from the graph of FIG. It embraces both implications of the length of time between. Then, the present invention observes a time-series coordinate space in which the line of sight stays, and from the time-series data related to the line-of-sight movement of the driver, the line-of-sight stays at intervals regardless of gaze or non-gaze. , Judge (estimate) the driver status.

Next, the configuration of the driver state determination device according to the embodiment of the present invention will be described with reference to FIG. FIG. 5 is a block diagram showing an example of the configuration of the driver state determination device according to the embodiment of the present invention. The driver state determination device 10 shown in FIG. 5 includes an imaging data acquisition unit 11, a line-of-sight motion detection unit 12, and a driver state determination unit 13.

The block diagram shown in FIG. 5 merely represents the functions related to the present invention, and may be realized by hardware, software, firmware, or any combination thereof in an actual implementation. Functions implemented in software are stored in any computer-readable medium as one or more instructions or codes, and these instructions or codes are stored in a CPU (Central Processing Unit) or GPU (Graphics Processing Unit). It can be executed by a hardware-based processing unit such as a graphics processing unit). Further, the functions related to the present invention may be realized by various devices including an IC (Integrated Circuit), an IC chipset, and the like.

The imaging data acquisition unit 11 has a function of acquiring imaging data including an image of the driver's face (particularly the driver's eye) captured by the camera 20. The imaging data obtained by imaging with the camera 20 is stored in a data storage device inside or outside the camera 20. The imaging data acquisition unit 11 acquires the imaging data by accessing the data storage device in which the imaging data is stored by wire or wirelessly.

For the camera 20, for example, an in-vehicle camera installed in the vehicle and capable of capturing the driver's face can be used. In the embodiment of the present invention, it is sufficient to obtain imaging data capable of detecting the direction of the driver's line of sight, and the camera 20 has a resolution (for example, HD image quality or less) and low enough to determine the direction of the driver's line of sight. It may be a low-performance camera that captures images at a frame rate (for example, about 30 fps). Many of the general in-vehicle cameras that are widely used at present perform imaging at a low frame rate of, for example, about 30 fps, and such a general in-vehicle camera may be used as the camera 20 according to the embodiment of the present invention. ..

The line-of-sight motion detection unit 12 has a function of performing image analysis processing on the imaged data acquired by the image pickup data acquisition unit 11 to detect the direction of the driver's line of sight (line-of-sight direction detection unit), and further, the time-lapse of the line-of-sight direction. A function that detects the movement distance of the driver's line of sight per unit time based on the target change and generates time-series data (hereinafter, also referred to as line-of-sight motion data) of the movement distance of the driver's line of sight (line-of-sight movement). It has a distance detection unit). The line-of-sight motion data detected by the line-of-sight motion detection unit 12 is supplied to the driver state determination unit 13.

The moving distance of the driver's line of sight is, for example, a distance on a screen (a reference plane having a substantially equidistant distance from the driver's head) set in front of the driver. For example, the distance traveled by the line of sight between time t1 and time t2 is the coordinates on the virtual screen where the direction of the line of sight at time t1 intersects the virtual screen and the virtual where the direction of the line of sight at time t2 intersects the virtual screen. It is represented by the distance between the coordinates on the screen (distance on the virtual screen).

The image analysis process for detecting the direction of the driver's line of sight from an image of the driver's face (particularly an image of the driver's eyes) is not particularly limited in the present invention, and existing techniques can be used. .. The driver's line-of-sight direction is, for example, the horizontal and vertical positions of the eyeball (horizontal and vertical directions of the eyeball) with reference to the eyeball position when visually recognizing a predetermined direction (initial value of the line-of-sight direction). It is possible to specify from the rotation angle of).

Further, a camera 20, an imaging data acquisition unit 11, and a line-of-sight motion detection unit 12 may be configured by using a commercially available line-of-sight measurement device. At present, for example, eyeglass-type and hat-type eye-gaze measuring devices that can be worn on the head are widely used, and a program that analyzes the direction of the line of sight from an eye image captured by a camera mounted on the gaze-measuring device. Is also widespread. The line-of-sight motion data may be generated by utilizing such an existing device or program.

Further, the driver state determination unit 13 has a function of performing a driver state determination process for determining the driver state based on the driver's line-of-sight motion data supplied from the line-of-sight detection unit 12 and outputting the determination result. For example, the driver state determination unit 13 extracts time-series data of the line-of-sight retention interval (the period during which the driver's line-of-sight movement distance or cumulative movement distance is relatively small) from the line-of-sight movement data, and changes over time in the line-of-sight retention interval. It is configured to perform a process of determining the state of the driver from the tendency (for example, the tendency of the fluctuation of the line-of-sight retention interval). The determination result by the driver state determination unit 13 includes the degree of the driver's state (including at least one state of drowsiness, fatigue, and tension), and alerts and recovers the state when the driver's state deteriorates. It can be used for various purposes such as driver's physical condition and business management. The details of the driver status determination process for determining the driver status will be described later.

Comparing the embodiment of the present invention with the conventional technique for detecting the saccade motion, the driver state determination device 10 in the embodiment of the present invention does not need to have a configuration for capturing a quick eye movement. The performance of the camera 20 has an advantage that it is sufficient to detect the direction of the line of sight. Further, when comparing the embodiment of the present invention with the conventional technique for detecting the movement time or reaction time of the driver's line of sight, the driver state determination device 10 in the embodiment of the present invention generates a visual stimulus. It is not necessary to have a configuration for detecting the timing, and there is an advantage that only the line-of-sight motion data obtained from the direction of the line-of-sight of the driver needs to be used.

Next, the operation of the driver state determination device according to the embodiment of the present invention will be described with reference to FIG. FIG. 6 is a flowchart showing an example of the operation of the driver state determination device according to the embodiment of the present invention. The flowchart shown in FIG. 6 shows an example of processing executed by the driver state determination device 10 shown in FIG.

The imaging data acquisition unit 11 acquires imaging data including an image of the driver's face (particularly, the driver's eyes) while driving the vehicle (step S110). The line-of-sight motion detection unit 12 performs image analysis processing on the imaged data to detect the direction of the driver's line of sight, and generates line-of-sight motion data including the moving distance of the driver's line of sight from the detection result (step S120). ..

If the direction of the driver's line of sight can be specified for each image included in the captured data, the direction of the driver's line of sight can be detected according to the frame rate of the camera 20. However, the imaging data acquisition unit 11 does not necessarily have to acquire all the images captured by the camera 20, and may be configured to acquire images at a predetermined sampling rate.

The driver state determination unit 13 performs a driver state determination process for determining the driver state based on the line-of-sight motion data (step S130). This driver state determination process may be executed substantially at the same time (almost in real time) as the driver drives the vehicle. In this case, the processes of steps S110 to S130 are executed while the vehicle is driving. On the other hand, when the driver drives the vehicle, only the imaging data and the line-of-sight motion data may be collected, and the driver state determination process may be executed after the vehicle is driven. In this case, the imaging data and the line-of-sight motion data are temporarily stored in the storage medium, and the stored data is collected after the vehicle is driven, and the process of step S130 is executed.

In the driver state determination process in step S130, for example, the driver state determination unit 13 performs a process of extracting time-series data of the line-of-sight retention interval from the line-of-sight motion data and numerically analyzes the time-series data of the line-of-sight retention interval to perform line-of-sight retention. It has a process of capturing the tendency of the fluctuation of the interval and a process of determining the state of the driver based on the tendency of the fluctuation of the line-of-sight retention interval.

The process of extracting the time-series data of the line-of-sight retention interval is as described above with reference to FIGS. It is possible to extract time-series data for a relatively small period (line-of-sight retention interval).

There are several possible methods for processing to capture the tendency of fluctuations in the line-of-sight retention interval by performing numerical analysis on the time-series data of the line-of-sight retention interval. For example, the following first to third numerical analysis methods are adopted. It is possible to do. However, the numerical analysis method is not limited to these.

For example, as a first numerical analysis method, a method of counting the number of line-of-sight retention intervals included in a predetermined time window while shifting the time window can be considered. Focusing on the peak that determines the line-of-sight retention interval (the boundary of the line-of-sight retention interval, which corresponds to a state in which the line-of-sight transition is large or a state in which patrol vision is performed), the first numerical analysis method is a predetermined time window. It can be said that it is a method of counting the number of peaks contained in the window while shifting the time window. When the peak that determines the line-of-sight retention interval is regarded as the heartbeat (heartbeat), the first numerical analysis method is similar to the method of counting the heartbeat within a certain time.

The range of the predetermined time window corresponds to the range for counting the line-of-sight retention interval or the number of peaks, and the size of the range can be appropriately set. Further, the width of shifting the predetermined time window corresponds to the frequency (interval) of counting, and the size of the width can be set as appropriate. The size of the width for shifting the time window may be the same as the size of the range of the time window, or may be smaller than the size of the range of the time window. As an example, when the range of the time window is 30 seconds and the width of shifting the time window is 10 seconds, the line-of-sight retention interval or the number of peaks included in the range of 30 seconds is counted every 10 seconds, and the line-of-sight retention interval or Time series data for the number of peaks is generated.

Further, with respect to the number of line-of-sight retention intervals or peaks counted at predetermined time intervals as described above, the distribution of the number of line-of-sight retention intervals or peaks included in the predetermined time range is obtained. Specifically, for example, when a predetermined time range is set to the latest 10 minutes and the number of line-of-sight retention intervals or peaks included in the range of 30 seconds is counted every 10 seconds, a large number (60 in this case). "The value of the line-of-sight retention interval or the number of peaks included in the range of 30 seconds" is extracted, and the distribution of these is obtained. By shifting this predetermined time range with time, it is possible to generate time-series data of the line-of-sight retention interval or the distribution of the number of peaks.

Assuming that the line-of-sight retention interval or the distribution of the number of peaks follows a normal distribution, the variance corresponding to the width of the bell-shaped distribution corresponds to the degree of variation in the line-of-sight retention interval or the number of peaks. Therefore, it can be said that the larger the variance is, the more the driver's movement is activated, while the smaller the variance is, the more monotonous the driver's movement is. In the process of determining the state of the driver, for example, it is possible to determine that the larger the dispersion, the higher the alertness of the driver, and the smaller the dispersion, the more drowsy or tired the driver is.

Further, for example, as a second numerical analysis method, a method of examining the variation in the line-of-sight retention interval can be considered. When the peak that determines the line-of-sight retention interval is regarded as the heartbeat (heartbeat), it can be said that the line-of-sight retention interval corresponds to the heartbeat interval (RRI). It is similar to the method of examining.

In the first example of the second numerical analysis method, for example, it is possible to obtain the distribution of the line-of-sight retention interval included in a predetermined time range with respect to the line-of-sight retention interval. It is desirable to set the predetermined time range to a time sufficiently longer than the line-of-sight retention interval. Specifically, for example, assuming that the predetermined time range is the latest 10 minutes, a large number of line-of-sight residence intervals (about several seconds) are included in this predetermined time range, and a large number of "values of the line-of-sight residence interval" are included. "Is extracted, and these distributions are obtained. By shifting this predetermined time range with time, it is possible to generate time-series data of the distribution of the line-of-sight retention interval.

Assuming that the distribution of the line-of-sight retention interval follows a normal distribution, the variance corresponding to the width of the bell-shaped distribution corresponds to the degree of variation in the line-of-sight retention interval. Therefore, it can be said that the larger the variance is, the more monotonous the line-of-sight retention interval is, while the smaller the variance is, the more monotonous the line-of-sight retention interval is. In the process of determining the state of the driver, for example, it is possible to determine that the larger the dispersion, the higher the alertness of the driver, and the smaller the dispersion, the more drowsy or tired the driver is.

Further, as a second example of the second numerical analysis method, a feature (frequency component) related to the periodic structure of the line-of-sight retention interval may be extracted. Here, in the periodic data reflecting the fluctuation of the living body, the high frequency component (HF component) reflects the activity of the parasympathetic nerve, and the low frequency component (LF component) / high frequency component (HF component) is the activity of the sympathetic nerve. It is based on the idea of reflecting the degree.

Here, we consider time-series data of the line-of-sight retention interval, which represents the relationship between the elapsed time and the value of the line-of-sight retention interval. When the line-of-sight retention interval is, for example, several seconds, the values are plotted every few seconds, so that the line-of-sight retention interval data is discrete data in the time axis direction. Therefore, time-series data of the line-of-sight retention interval continuous in the time axis direction may be generated by performing interpolation processing or the like as necessary.

By calculating the power spectral density for the time-series data of the line-of-sight retention interval, the periodic structure of the time-series data of the line-of-sight retention interval can be extracted. If the time-series data of the line-of-sight retention interval reflects the fluctuation of the living body, the peak of the power spectrum appears in each of the LF component and the HF component, as in the case of the time-series data of the heart rate variability. Then, the power spectrum integral value of the region including the LF component (integral value of the LF region) and the power spectrum integral value of the region including the HF component (integral value of the HF region) are calculated, for example, the integral value of the HF region. It is possible to determine that the higher the value, the more relaxed the state, and the higher the value of (integral value in the LF region) / (integrated value in the HF region), the more tense the state.

Further, for example, as a third numerical analysis method, a method of performing a non-linear analysis process on the line-of-sight retention interval can be considered. Specifically, an attractor is generated based on the values of a plurality of line-of-sight retention intervals extracted from the time-series data of the line-of-sight retention interval, and the Lyapunov exponent is calculated based on the generated attractor.

The outline of the third numerical analysis method is shown in FIG. The line-of-sight retention interval is treated as information arranged in chronological order, as represented by, for example, P [1], P [2], P [3], P [N], .... In the non-linear analysis process, N sets of data (P [1], P [2], ..., P [N]) adjacent to each other in a time series are plotted in an N-dimensional space, and the data sets are sequentially shifted. However, coordinates in N-dimensional space such as (P [2], P [3], ..., P [N + 1]), (P [3], P [4], ..., P [N + 2]), ... Plot sequentially on top. Note that FIG. 7 schematically shows an example in the case of N = 3. The line connecting the coordinates in the N-dimensional space on which the set of data of the line-of-sight retention interval is plotted draws a locus and forms an attractor. From the attractors formed in this way, it is possible to calculate the Lyapunov exponent by analyzing the stability tendency for the past attractor group. The calculation of the Lyapunov exponent can be performed at any period, and as a result, time series data of the Lyapunov exponent is generated. Based on the premise that the fluctuation of the line-of-sight retention interval correlates with the biological fluctuation, it can be said that the Lyapunov exponent obtained from the time-series data of the line-of-sight retention interval is a numerical value that reflects the state of physical function and brain function. The lower the Lyapunov exponent, the lower the physical function and brain function of the driver. Therefore, for example, the lower the Lyapunov exponent, the more drowsy or tired the driver can be determined.

The present invention can be applied to a technique for determining a driver's state during vehicle driving (including simulated driving by a driving simulator).

10 Driver status determination device 11 Imaging data acquisition unit 12 Line-of-sight motion detection unit 13 Driver status determination unit 20 Camera

Claims (12)

A driver state determination device that determines the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle.
A line-of-sight direction detection unit that detects the line-of-sight direction of the driver,
A line-of-sight movement distance detection unit that detects the movement distance of the driver's line-of-sight per predetermined time from a change over time in the direction of the driver's line-of-sight.
The time length of the time zone in which the moving distance of the driver's line of sight is relatively small is extracted as the line-of-sight retention interval, and the peak at which the change in the moving distance of the driver's line of sight with time is maximized is detected. A line-of-sight retention interval extraction unit that extracts the time length between adjacent peaks as the line-of-sight retention interval,
A driver state determination unit that identifies the tendency of the time-dependent change of the line-of-sight retention interval and determines the state of the driver from the specified tendency.
Driver status determination device to have. The driver state determination unit detects the number of time zones extracted as the line-of-sight retention interval within a predetermined time range, calculates the variance of the detected number distribution, and changes the dispersion over time. The driver state determination device according to claim 1, which is configured to determine the state of the driver from the tendency. The driver state determination unit is configured to calculate the dispersion of the distributions of the plurality of line-of-sight retention intervals included in a predetermined time range, and determine the state of the driver from the tendency of the change over time of the dispersion. The driver state determination device according to claim 1. The driver state determination unit calculates the Lyapunov exponent related to the line-of-sight residence interval by performing nonlinear analysis processing on the time-series data of the line-of-sight residence interval, and determines the state of the driver based on the Lyapunov exponent. The driver state determination device according to claim 1, which is configured. A driver state determination device that determines the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle.
A line-of-sight direction detection unit that detects the line-of-sight direction of the driver,
A line-of-sight movement distance detection unit that detects the movement distance of the driver's line of sight per predetermined time from a change over time in the direction of the driver's line of sight.
A line-of-sight retention interval extraction unit that extracts the time length of a time zone in which the driver's line-of-sight movement distance is relatively small as a line-of-sight retention interval.
A driver state determination unit that identifies the tendency of the time-dependent change of the line-of-sight retention interval and determines the state of the driver from the specified tendency.
Have
The driver state determination unit calculates the Lyapunov exponent related to the line-of-sight residence interval by performing nonlinear analysis processing on the time-series data of the line-of-sight residence interval, and determines the state of the driver based on the Lyapunov exponent. The configured driver status determination device. The driver state determination device according to any one of claims 1 to 5, wherein the line-of-sight direction detection unit is configured to detect the direction of the line-of-sight of the driver from an image of the driver's eyes captured by a camera. A driver state determination method for determining the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle.
The line-of-sight direction detection step for detecting the line-of-sight direction of the driver, and
A line-of-sight movement distance detection step that detects the movement distance of the driver's line-of-sight per predetermined time from a change over time in the direction of the driver's line-of-sight.
The time length of the time zone in which the moving distance of the driver's line of sight is relatively small is extracted as the line-of-sight retention interval, and the peak at which the change in the moving distance of the driver's line of sight with time is maximized is detected. A line-of-sight retention interval extraction step that extracts the time length between adjacent peaks as the line-of-sight retention interval, and
The driver state determination step of identifying the tendency of the time-dependent change of the line-of-sight retention interval and determining the state of the driver from the specified tendency.
Driver status determination method to have. In the driver state determination step, the number of time zones extracted as the line-of-sight retention interval is detected within a predetermined time range, the variance of the detected number distribution is calculated, and the change over time of the dispersion is calculated. The driver state determination method according to claim 7, wherein the state of the driver is determined from the tendency. The seventh aspect of claim 7, wherein in the driver state determination step, the variance of the distribution of the plurality of line-of-sight retention intervals included in a predetermined time range is calculated, and the state of the driver is determined from the tendency of the change over time of the dispersion. Driver status judgment method. In the driver state determination step, a non-linear analysis process is performed on the time-series data of the line-of-sight residence interval to calculate the Lyapunov exponent for the line-of-sight residence interval, and the driver status is determined based on the Lyapunov exponent. Item 7. The driver state determination method according to Item 7. A driver state determination method for determining the state of the driver including at least one state of drowsiness, fatigue, and tension of the driver while driving a vehicle.
The line-of-sight direction detection step for detecting the line-of-sight direction of the driver, and
A line-of-sight movement distance detection step that detects the movement distance of the driver's line-of-sight per predetermined time from a change over time in the direction of the driver's line-of-sight.
The line-of-sight retention interval extraction step for extracting the time length of the time zone in which the driver's line-of-sight movement distance is relatively small as the line-of-sight retention interval, and
The driver state determination step of identifying the tendency of the time-dependent change of the line-of-sight retention interval and determining the state of the driver from the specified tendency.
Have
In the driver state determination step, a driver that calculates the Lyapunov exponent for the line-of-sight residence interval by performing nonlinear analysis processing on the time-series data of the line-of-sight residence interval, and determines the state of the driver based on the Lyapunov exponent. State judgment method. The driver state determination method according to any one of claims 7 to 11, wherein in the line-of-sight direction detection step, the direction of the line-of-sight of the driver is detected from an image obtained by capturing the eyes of the driver with a camera.

Images