JP7342638B2 - Driver status detection device - Google Patents

Description

The technology disclosed herein relates to a driver condition detection device.

Patent Document 1 discloses a vehicle control device that detects abnormality of a driver of a vehicle. This vehicle control device includes a body temperature measurement section that measures the body temperature of a vehicle driver, a running state determination section that determines whether the running state of the vehicle is abnormal, and a measurement result of the body temperature measurement section and a running state determination section. and a vehicle control section that controls the vehicle based on the determination result of the section.

JP 2019-73105 Publication

The device disclosed in Patent Document 1 cannot detect abnormalities that do not involve fever, such as brain disease or epilepsy. Furthermore, in a driving scene where there are relatively few changes in vehicle behavior, it may not be possible to determine whether the driving state of the vehicle is abnormal. Therefore, it is difficult to improve the accuracy of detecting abnormalities of the driver.

The technology disclosed herein has been developed in view of this point, and its purpose is to improve the accuracy of detecting driver abnormalities.

The technology disclosed herein relates to a driver state detection device installed in a vehicle, and the driver state detection device includes a saccade detection unit that detects saccades of a driver of the vehicle, and a saccade detection unit that detects the degree of attention in the external environment of the vehicle. Detecting an abnormality of the driver based on an attentiveness detecting unit, the attentiveness detected by the attentiveness detecting unit, and the amplitude or frequency of the driver's saccades detected by the saccade detecting unit. and an abnormality detection section.

As a result of intensive research, the inventors of the present application have found that when the driver has an abnormality, the amplitude or frequency of the saccades of the vehicle driver decreases significantly depending on the degree of caution in the external environment of the vehicle. Therefore, by detecting the driver's abnormality according to the degree of attentiveness in the external environment of the vehicle and the amplitude or frequency of the saccades of the driver of the vehicle, it is possible to The detection accuracy of the driver's abnormality can be improved compared to the case where the driver's abnormality is detected based on the driver's abnormality.

Further, in the driver state detection device, the abnormality detection unit performs a first operation when the level of attentiveness detected by the attentiveness level detecting unit is a high level of attentiveness, and The second operation may be configured to be performed when the attention level is low. In the first operation, the abnormality detection unit may detect that the driver has an abnormality when the amplitude of the saccade of the driver is less than or equal to a preset amplitude threshold. In the second operation, the abnormality detection unit may detect that the driver has an abnormality when the frequency of saccades of the driver is less than or equal to a preset frequency threshold.

The inventors of the present invention have found that when the driver is abnormal, the amplitude of the vehicle driver's saccades is significantly reduced when the level of attentiveness in the external environment of the vehicle is relatively high. In addition, the inventors of the present invention have found that when the driver has an abnormality, the frequency of saccades of the vehicle driver decreases significantly when the degree of attentiveness in the external environment of the vehicle is relatively low. Therefore, the first operation (detection of driver abnormality based on the amplitude of the driver's saccades) is performed when the level of attentiveness in the external environment of the vehicle is high, and In this case, by performing the second operation (detecting the driver's abnormality based on the frequency of the driver's saccades), the accuracy of detecting the driver's abnormality can be improved.

Further, in the driver state detection device, the amplitude threshold may be set based on the amplitude of a saccade of a person with an attentional dysfunction when the level of attentiveness in the external environment of the vehicle is high. The frequency threshold may be set based on the frequency of saccades of the person with impaired attention function when the degree of attentiveness in the external environment of the vehicle is low.

The inventors of the present application have discovered that by observing the behavior of a person with an attentional dysfunction while driving a vehicle, it is possible to simulate the behavior of a vehicle driver when driving a vehicle. Therefore, the amplitude threshold can be appropriately set based on the amplitude of the saccade of the person with attentional dysfunction when the degree of attentiveness in the external environment of the vehicle is high. Further, the frequency threshold can be appropriately set based on the frequency of saccades of a person with an attentional dysfunction when the degree of attentiveness in the external environment of the vehicle is low.

According to the technology disclosed herein, it is possible to improve the accuracy of detecting a driver's abnormality.

FIG. 1 is a block diagram illustrating the configuration of a vehicle control system according to an embodiment. FIG. 2 is a block diagram illustrating the configuration of a driver state detection section. It is a graph illustrating the results of a medical test regarding attention function. FIG. 2 is a schematic diagram showing an example of line of sight movement by a healthy person. FIG. 2 is a schematic diagram showing an example of gaze movement by an attention dysfunction patient (severe). It is a graph showing an example of eye movement by a healthy person. It is a graph showing an example of gaze movement by an attention dysfunction patient (severe). It is a graph for explaining saccades. It is a graph for explaining extraction of saccades. 3 is a graph illustrating the amplitude of saccades of a healthy person and an attention dysfunction patient in each of a high-attention scene and a low-attention scene. 2 is a graph illustrating the frequency of saccades of healthy subjects and patients with attention dysfunction in high-attention scenes and low-attention scenes, respectively. 12 is a flowchart for explaining the operation (saccade detection) of the driver state detection section. FIG. 3 is a flowchart for explaining the operation (abnormality detection) of the driver state detection section. FIG.

Hereinafter, embodiments will be described in detail with reference to the drawings. It should be noted that the same or corresponding parts in the figures are given the same reference numerals, and the description thereof will not be repeated.

(vehicle control system)
FIG. 1 illustrates the configuration of a vehicle control system 10 according to an embodiment. The vehicle control system 10 is provided in a vehicle (specifically, a four-wheeled motor vehicle). The vehicle can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in response to the driver's operations (for example, accelerator operation). Assisted driving is driving in which the vehicle supports the driver's operations. Autonomous driving is driving without driver input. The vehicle control system 10 controls the vehicle in assisted driving and automatic driving. Specifically, the vehicle control system 10 controls the operation (especially driving) of the vehicle by controlling an actuator 11 provided in the vehicle.

The vehicle control system 10 includes an information acquisition section 20, a vehicle control device 30, and a notification section 40. In the following description, the vehicle provided with the vehicle control system 10 will be referred to as the "host vehicle", and other vehicles existing around the host vehicle will be referred to as "other vehicles".

[Actuator]
The actuator 11 includes a drive system actuator, a steering system actuator, a brake system actuator, and the like. Examples of drive system actuators include engines, motors, and transmissions. An example of a steering system actuator is a steering wheel. An example of a brake system actuator is a brake.

[Information acquisition department]
The information acquisition unit 20 acquires various information used for vehicle control (particularly driving control). In this example, the information acquisition section 20 includes a plurality of cameras 21, a plurality of radars 22, a position sensor 23, an external input section 24, a vehicle condition sensor 25, a driving operation sensor 26, and a driver condition sensor 27. including.

<camera>
The plurality of cameras 21 have mutually similar configurations. A plurality of cameras 21 are provided on the vehicle so as to surround the vehicle. Each of the plurality of cameras 21 acquires image data representing a part of the external environment of the vehicle by capturing an image of a part of the environment surrounding the vehicle (external environment of the vehicle). Image data obtained by each of the plurality of cameras 21 is transmitted to the vehicle control device 30.

In this example, camera 21 is a monocular camera with a wide-angle lens. For example, the camera 21 is configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor). Note that the camera 21 may be a monocular camera with a narrow-angle lens, or may be a stereo camera with a wide-angle lens or a narrow-angle lens.

<Radar>
The plurality of radars 22 have mutually similar configurations. The plurality of radars 22 are provided in the vehicle so as to surround the vehicle. Each of the plurality of radars 22 detects a part of the external environment of the vehicle. Specifically, the radar 22 detects a part of the external environment of the vehicle by transmitting radio waves toward a part of the external environment of the vehicle and receiving reflected waves from the part of the external environment of the vehicle. do. The detection results of the plurality of radars 22 are transmitted to the vehicle control device 30.

For example, the radar 22 may be a millimeter wave radar that transmits millimeter waves, a lidar (Light Detection and Ranging) that transmits laser light, or an infrared radar that transmits infrared light. Alternatively, it may be an ultrasonic sensor that transmits ultrasonic waves.

<Position sensor>
The position sensor 23 detects the position (for example, latitude and longitude) of the vehicle. For example, the position sensor 23 receives GPS information from the global positioning system and detects the position of the vehicle based on the GPS information. Information (vehicle position) obtained by the position sensor 23 is transmitted to the vehicle control device 30.

<External input section>
The external input unit 24 inputs information through an external network (for example, the Internet) provided outside the vehicle. For example, the external input unit 24 receives communication information from other vehicles (not shown) located around the vehicle, car navigation data from a navigation system (not shown), traffic information, high-precision map information, and the like. Information obtained by the external input section 24 is transmitted to the vehicle control device 30.

<Vehicle condition sensor>
The vehicle condition sensor 25 detects the condition of the vehicle (for example, speed, acceleration, yaw rate, etc.). For example, the vehicle condition sensor 25 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, and the like. Information (vehicle status) obtained by the vehicle status sensor 25 is transmitted to the vehicle control device 30.

<Driving operation sensor>
The driving operation sensor 26 detects a driving operation applied to the vehicle. For example, the driving operation sensor 26 includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the amount of operation of the accelerator of the vehicle. The steering angle sensor detects the steering angle of the steering wheel of the vehicle. The brake oil pressure sensor detects the amount of brake operation of the vehicle. Information (vehicle driving operation) obtained by the driving operation sensor 26 is transmitted to the vehicle control device 30.

<Driver status sensor>
The driver condition sensor 27 detects the condition of the driver who drives the vehicle (for example, the driver's health condition, emotions, physical behavior, etc.). The information (driver's condition) obtained by the driver condition sensor 27 is transmitted to the vehicle control device 30. In this example, driver condition sensor 27 includes an in-vehicle camera 28 and a biological information sensor 29.

《In-vehicle camera》
The in-vehicle camera 28 is provided inside the vehicle. The in-vehicle camera 28 acquires image data including the driver's eyeballs by capturing an image of a region including the driver's eyeballs. Image data obtained by the in-vehicle camera 28 is transmitted to the vehicle control device 30. For example, the in-vehicle camera 28 is placed in front of the driver, and the imaging range is set so that the driver's eyeballs are within the imaging range. Note that the in-vehicle camera 28 may be provided in goggles (not shown) worn by the driver.

《Biological information sensor》
The biological information sensor 29 detects biological information (for example, sweating) of the driver. Information obtained by the biometric information sensor 29 (driver's biometric information) is transmitted to the vehicle control device 30.

[Vehicle control device]
The vehicle control device 30 is electrically connected to the actuator 11 and each section of the vehicle control system 10 (in this example, the information acquisition section 20, the notification section 40, etc.). Then, the vehicle control device 30 controls the actuator 11 and each part of the vehicle control system 10 based on information obtained by each part of the vehicle control system 10. Note that the vehicle control device 30 is an example of a driver state detection device.

In assisted driving or automatic driving, the vehicle control device 30 determines a target route, which is the route the vehicle should travel, based on various information acquired by the information acquisition unit 20, and determines the target route that is the route the vehicle should travel, and determines the route required for traveling on the target route. Determine the target motion, which is the motion of the vehicle. The vehicle control device 30 then controls the operation of the actuator 11 so that the motion of the vehicle matches the target motion. For example, the vehicle control device 30 includes one or more electronic control units (ECUs). The electronic control unit may be composed of a single IC (Integrated Circuit) or may be composed of a plurality of ICs. Additionally, an IC may include a single core or die, or multiple cooperating cores or dies. The core or die may be configured of, for example, a CPU (processor) and a memory that stores information such as programs for operating the CPU and processing results by the CPU.

In this example, the vehicle control device 30 includes a vehicle behavior recognition section 31 , a driving operation recognition section 32 , an external environment recognition section 33 , a driver behavior recognition section 34 , and a vehicle control section 35 .

<Vehicle behavior recognition unit>
The vehicle behavior recognition unit 31 recognizes vehicle behavior (for example, speed, acceleration, yaw rate, etc.) based on the output of the vehicle state sensor 25. For example, the vehicle behavior recognition unit 31 generates data indicating the behavior of the vehicle from the output of the vehicle state sensor 25 using a learning model generated by deep learning.

<Driving operation recognition unit>
The driving operation recognition unit 32 recognizes the driving operation applied to the vehicle based on the output of the driving operation sensor 26. For example, the driving operation recognition unit 32 generates data indicating a driving operation applied to the vehicle from the output of the driving operation sensor 26 using a learning model generated by deep learning.

<External environment recognition department>
The external environment recognition unit 33 recognizes the vehicle based on the outputs of the plurality of cameras 21, the outputs of the plurality of radars 22, the outputs of the position sensor 23, the outputs of the external input unit 24, and the outputs of the vehicle behavior recognition unit 31. Recognize the external environment. For example, the external environment recognition unit 33 generates data (for example, three-dimensional map data) indicating the external environment of the vehicle from the above output using a learning model generated by deep learning. The external environment of the vehicle recognized by the external environment recognition unit 33 includes objects. Examples of objects include a moving object that displaces over time and a stationary object that does not displace over time. Examples of moving objects include four-wheeled motor vehicles, motorcycles, bicycles, and pedestrians. Examples of stationary objects include signs, street trees, median strips, center poles, and buildings.

<Driver behavior recognition unit>
The driver behavior recognition unit 34 recognizes the driver's behavior (for example, the driver's health condition, emotions, physical behavior, etc.) based on the output of the driver condition sensor 27. For example, the driver behavior recognition unit 34 generates data indicating the driver's behavior from the output of the driver condition sensor 27 using a learning model generated by deep learning. In this example, the driver behavior recognition section 34 includes a driver state detection section 300. The driver state detection section 300 will be explained in detail later.

<Vehicle control section>
The vehicle control unit 35 controls the actuator 11 based on the output of the vehicle behavior recognition unit 31, the output of the driving operation recognition unit 32, the output of the external environment recognition unit 33, and the output of the driver behavior recognition unit 34. . In this example, the vehicle control section 35 includes a candidate route generation section 36, a target route determination section 37, and a movement control section 38.

The candidate route generation unit 36 generates one or more candidate routes based on the output of the external environment recognition unit 33. A candidate route is a route on which a vehicle can travel, and is a candidate for a target route.

The target route determination unit 37 selects a target route from among the one or more candidate routes generated by the candidate route generation unit 36 based on the output of the driving operation recognition unit 32 and the output of the driver behavior recognition unit 34. Select a candidate route. For example, the target route determining unit 37 selects a candidate route that the driver feels is most comfortable from among a plurality of candidate routes.

The motion control unit 38 determines a target motion based on the candidate route selected as the target route by the target route determination unit 37, and controls the actuator 11 based on the determined target motion. For example, the motion control unit 38 derives a target driving force, a target braking force, and a target steering amount, which are a driving force, a braking force, and a steering amount for achieving the target movement, respectively. Then, the motion control unit 38 transmits the drive command value indicating the target driving force, the braking command value indicating the target braking force, and the steering command value indicating the target steering amount to the actuators of the drive system, the actuators of the braking system, and the steering system. and the actuator respectively.

[Notification Department]
The notification unit 40 is provided inside the vehicle. The notification unit 40 notifies the driver of the vehicle of various information. In this example, the notification section 40 includes a display section 41 and a speaker 42. The display unit 41 outputs various information in the form of images. The speaker 42 outputs various information in the form of audio.

[Driver status detection unit]
FIG. 2 illustrates the configuration of the driver state detection section 300. The driver condition detection unit 300 detects abnormality of the driver of the vehicle. Specifically, the driver state detection section 300 includes a saccade detection section 301 , an attention level detection section 302 , and an abnormality detection section 303 .

Note that the driver's abnormality refers to an abnormality that causes a decline in the driver's attention function. Examples of such driver abnormalities include brain diseases such as stroke, heart diseases such as myocardial infarction, epilepsy, hypoglycemia, and drowsiness.

<Saccade detection part>
A saccade detection unit 301 detects a saccade of a vehicle driver. A saccade is a jumping eye movement in which the driver intentionally moves his/her line of sight, and is an eye movement in which the driver moves his/her line of sight from a point of gaze where the line of sight remains stagnant for a predetermined period of time to the next point of gaze. In this example, the saccade detection unit 301 detects the driver's line of sight by performing line of sight detection processing on image data obtained by the in-vehicle camera 28. Note that this line of sight detection processing may be a process performed using a learning model (a learning model for detecting line of sight) generated by deep learning, or a process performed using a well-known line of sight detection technology. It may be. Further, the driver's line of sight may be the line of sight of the driver's right eye, the line of sight of the driver's left eye, or the line of sight of the driver's right eye and the line of sight of the left eye. The line of sight derived based on this may also be used. Then, the saccade detection unit 301 detects the driver's saccade based on the movement of the driver's line of sight. The driver's saccades detected by the saccade detection unit 301 are supplied to the abnormality detection unit 303. Note that saccade detection will be explained in detail later.

<Attention level detection unit>
The caution level detection unit 302 detects the degree of caution in the external environment of the vehicle. As the number of caution points (points of caution that the vehicle driver should check while driving) increases in the external environment of the vehicle, the degree of caution in the vehicle external environment increases. The detection result by the caution level detection unit 302 is supplied to the abnormality detection unit 303. Examples of areas to be careful of include areas where moving objects are predicted to jump out, areas where there are objects that may become obstacles to the host vehicle, etc.

For example, the caution level detection unit 302 may detect the caution level in the external environment of the vehicle as follows. First, the caution level detection unit 302 inputs the information (map information) obtained by the external input unit 24 and detects a high caution level area from the map information. A high degree of caution region is a region where there are relatively many caution points that the driver of the vehicle should check while driving (for example, a region where the number of caution points is equal to or greater than a predetermined threshold). Examples of high-attention areas include areas including road environments with poor visibility such as T-junctions, and road environments where the vehicle's travel may be obstructed by obstacles such as intersections (for example, other vehicles). Examples include areas. Further, the caution level detection unit 302 inputs information (vehicle position) obtained by the position sensor 23. Then, when the position of the vehicle is included in the high-attention area, the caution level detection unit 302 detects that the level of caution in the external environment of the vehicle is high, and the position of the vehicle is in the high-attention area. If the vehicle is not included in the area, it is detected that the degree of caution in the external environment of the vehicle is low.

Alternatively, the caution level detection unit 302 may detect the caution level in the external environment of the vehicle as follows. First, the caution level detection section 302 detects caution points from inside the vehicle's external environment recognized by the external environment recognition section 33 . Then, when the number of caution points included in the external environment of the vehicle is relatively large (for example, when the number of caution points is greater than or equal to a predetermined threshold), the caution level detection unit 302 determines the degree of caution in the external environment of the vehicle. is detected to have a high degree of caution, and when the number of caution points included in the vehicle's external environment is relatively small (for example, when the number of caution points is less than a predetermined threshold), the vehicle's external environment Detecting that the level of attention is low.

Note that a driving scene in which the level of caution in the external environment of the vehicle is high is a driving scene in which there are relatively many obvious caution points, and the driver is required to direct his/her line of sight to these caution points over a wide range. It can be said. In addition, in driving scenes where the degree of caution in the external environment of the vehicle is low, there are relatively few obvious caution points, and it is required to frequently move the line of sight to search for potential caution points. It can be said that this is a driving scene. In the following, a driving scene in which the vehicle's external environment has a high caution level is referred to as a "high caution scene," and a driving scene in which the vehicle's external environment has a low caution level is referred to as a "low caution scene." ”.

<Abnormality detection part>
The abnormality detection unit 303 detects an abnormality of the driver based on the attentiveness level detected by the attentiveness level detection unit 302 and the amplitude ds or frequency fs of the driver's saccades detected by the saccade detection unit 301.

Specifically, the abnormality detection unit 303 performs the first operation when the level of alertness detected by the alertness level detection unit 302 is high alertness, and the abnormality detection unit 303 performs the first operation when the alertness level detected by the alertness level detection unit 302 is low alertness. The device is configured to perform the second operation when the temperature is high. In the first operation, the abnormality detection unit 303 detects that the driver has an abnormality when the amplitude ds of the driver's saccade is less than or equal to a preset amplitude threshold dth. In the second operation, the abnormality detection unit 303 detects that the driver has an abnormality when the frequency fs of the driver's saccades is less than or equal to a preset frequency threshold fth.

In this example, the amplitude threshold dth is set based on the amplitude of a saccade of a person with an attentional dysfunction when the degree of attentiveness in the external environment of the vehicle is high. The frequency threshold value fth is set based on the frequency of saccades of a person with an attentional dysfunction when the degree of attentiveness in the external environment of the vehicle is low.

Note that the operation of the abnormality detection unit 303 will be explained in detail later.

[Findings obtained by the inventors]
As a result of intensive research, the inventors of the present application have simulated the behavior of vehicle drivers during abnormal situations (abnormal situations that cause a decline in attention function) by observing the behavior of people with attention dysfunction when driving a vehicle. I found out that it can be done.

The inventors of the present application analyzed the observed results of the behavior of people with attentional dysfunction when driving a vehicle, and the observed results of the behavior of healthy people without attentional dysfunction when driving a vehicle. As a result of this analysis, the inventors found that when a driver has an abnormality (an abnormality that causes a decline in attentional function), the amplitude or frequency of the vehicle driver's saccades increases significantly depending on the level of attentiveness in the vehicle's external environment. was found to decrease.

Specifically, the present inventors found that when the driver has an abnormality (an abnormality that causes a decline in attention function), the amplitude of the saccade of the vehicle driver decreases when the level of attentiveness in the external environment of the vehicle is relatively high. was found to be significantly reduced. In addition, the inventors of the present invention have found that when the driver has an abnormality (an abnormality that causes a decline in attention function), the frequency of saccades of the vehicle driver significantly increases when the level of attentiveness in the external environment of the vehicle is relatively low. found that it decreased.

The experiments conducted by the inventors of the present application will be described below.

<Selection of subjects>
In the experiments conducted by the inventors, patients with mild attentional dysfunction (hereinafter referred to as "mild patients") and patients with moderate attentional dysfunction (hereinafter referred to as "moderate patients") ), patients with severe attentional dysfunction (hereinafter referred to as "severe patients"), and healthy individuals without attentional dysfunction (hereinafter simply referred to as "healthy subjects") were selected as subjects. Note that the degree of attention dysfunction, ie, "mild,""moderate," and "severe," is classified based on the score of a medical test called TMT (Trail Making Test). TMT is a test in which consecutive symbols (e.g., numbers, alphabets, hiragana, katakana, etc.) written randomly on a sheet of paper are connected in order with lines, and the time required from the start to the end of the test is The time (seconds) is the score. Note that the attention dysfunction patient (patient with attention dysfunction) is an example of an attention dysfunction person.

FIG. 3 illustrates reference values for TMT scores of healthy individuals by age (for example, average values or cutoff values for multiple subjects) and reference values for TMT scores of patients by degree of attention dysfunction. . As shown in Figure 3, as age increases from 20s, 30s, 40s, 50s, and 60s, the standard value of the TMT score (hereinafter referred to as "score standard value") gradually increases. I will do it. This is thought to be due to a decline in attentional function due to aging.

Moreover, as shown in FIG. 3, the score standard value for mild patients is smaller than the score standard value for healthy people in their 60s. The standard score value for moderate patients is larger than the standard score value for mild patients, and is comparable to the standard score value for healthy people in their 60s. The score reference value for severe patients is larger than the score reference value for moderate patients, and is larger than the score reference value for healthy people in their 60s. In addition, in this experiment, we compared the driving behavior of severely ill patients whose score standard value is higher than that of a healthy person in their 60s when the driver has an abnormality (an abnormality that causes a decline in attention function). This was observed as an example of the driver's behavior when driving a vehicle.

<Experiment details>
The subjects selected as described above were given a simulated experience of driving a vehicle using a driving simulator. Specifically, by having a subject view a moving image of a vehicle in motion (a moving image showing the vehicle's external environment as seen from inside the vehicle) using a driving simulator, and observing the subject's behavior while viewing, the test subject's vehicle The behavior during driving was observed in a simulated manner. In this experiment, a camera was installed in front of a subject who was viewing a moving image of a moving vehicle, and the camera was set so that the subject's eyeballs were included within the imaging range. Then, the subject's line of sight was detected by performing line of sight detection processing on the image data obtained by the camera. Note that this line of sight detection processing may be a process performed using a learning model (a learning model for detecting line of sight) generated by deep learning, or a process performed using a well-known line of sight detection technology. It may be.

<Subject's line of sight movement>
FIG. 4 shows an example of the line of sight movement of a healthy person, and FIG. 5 shows an example of the line of sight movement of a severely ill patient. In addition, in the figure, the black circle indicates the point of gaze where the subject's line of sight stagnated for a predetermined time, and the straight line connecting the two points of gaze is the trajectory of the subject's line of sight movement from one point of gaze to the other point. It shows. Furthermore, the examples in FIGS. 4 and 5 show a driving scene in which the host vehicle overtakes another vehicle 51 from the right side. Furthermore, in the examples shown in FIGS. 4 and 5, another vehicle 52 is present in front of the host vehicle. In the example of FIGS. 4 and 5, the other vehicle 51, the other vehicle 52, and the left side wall 53 of the roadway correspond to the caution points. The driving scenes shown in FIGS. 4 and 5 are examples of high-attention scenes (driving scenes in which the external environment of the vehicle has a high degree of caution).

As shown in FIG. 4, when the subject was a healthy person, the subject's line of sight was evenly directed to multiple attention points. On the other hand, as shown in Figure 5, when the subject is a severely ill patient, the subject's line of sight moves within a relatively narrow range, and the distance traveled by the subject's line of sight is longer than when the subject is a healthy person. was getting shorter.

<Subject's line of sight movement after viewing multiple driving scenes>
FIG. 6 shows an example of the line-of-sight movement of a healthy person who viewed a plurality of driving scenes, and FIG. 7 shows an example of the line-of-sight movement of a severely ill patient who viewed a plurality of driving scenes. In the examples of FIGS. 6 and 7, the vertical axis indicates the travel distance (angle) of the subject's line of sight, and the horizontal axis indicates the elapsed time. Furthermore, in the examples of FIGS. 6 and 7, the portion surrounded by the first broken line indicates the movement of the subject's line of sight in a driving scene in which the subject is traveling on a motorway. The part surrounded by the second broken line shows the movement of the subject's line of sight in a driving scene where the subject is driving in a city area where other vehicles are present. The part surrounded by the third broken line shows the movement of the subject's line of sight in a driving scene in which the subject is driving in a city area where no other vehicles are present. The part surrounded by the fourth broken line shows the movement of the subject's line of sight in a driving scene in a crowded city area.

Comparing Figures 6 and 7, the difference in distance traveled by the subject's line of sight is relatively small in the area surrounded by the first broken line, but in the area surrounded by the second broken line. In this case, the difference in distance traveled by the subject's line of sight is relatively large. In other words, the distance the subject's line of sight moves in the area surrounded by the second broken line in FIG. 7 is significantly smaller than the distance moved in the area surrounded by the second broken line in FIG. ing. This is due to the impaired attention function of the subject, a severely ill patient, who is forced to focus his/her gaze on multiple points of attention in high-attention scenes (driving scenes that require directing the line of sight over a wide range of multiple revealed points of attention). This is thought to be due to the inability to aim over a wide area.

Furthermore, when comparing Figures 6 and 7, the difference in the number of times the subject's gaze moves is relatively small in the part surrounded by the fourth broken line, but The difference in the number of times the subject's line of sight moved was relatively large. In other words, the number of times the subject's line of sight moves in the area surrounded by the third broken line in FIG. 7 is significantly smaller than the number of times the subject moves in the area surrounded by the third broken line in FIG. . This is due to the severely impaired attention function of the test subjects, causing them to frequently shift their line of sight in low-attention scenes (driving scenes that require frequent movement of the line of sight to search for potential attention points). This is thought to be due to the inability to search for potential cautionary points.

<Derivation of saccades>
Next, in order to evaluate the movement of the subject's gaze, the subject's saccades were derived based on the following procedure. A saccade is a jumping eye movement in which the driver intentionally moves his/her line of sight, and is an eye movement in which the driver moves his/her line of sight from a point of gaze where the line of sight remains stagnant for a predetermined period of time to the next point of gaze. As shown in FIG. 8, a period sandwiched between two adjacent gaze periods is a saccade period. Note that the gaze period is a period in which the line of sight is considered to be stationary. The saccade amplitude ds is the distance the line of sight moves during the saccade period.

In this experiment, the moving speed of the line of sight is calculated based on the change in the moving distance of the line of sight, and the state in which the moving speed of the line of sight is less than a predetermined speed threshold (for example, 40 deg/s) is determined by a predetermined stagnation time ( For example, a period lasting 0.1 seconds) was extracted as a "gazing period". Then, the movement speed of the line of sight movement during the period sandwiched between two adjacent gaze periods is equal to or higher than the speed threshold (for example, 40 deg/s), and the movement distance is predetermined and the distance threshold (for example, 3 deg). The above gaze movement was extracted as a "saccade."

Note that the saccades extracted as described above include noise such as the subject's blinking. Therefore, in this experiment, in order to exclude such noise, saccades were extracted based on the following procedure.

First, saccade candidates were extracted based on changes in the distance traveled by the subject's line of sight. Next, as shown in FIG. 9, a regression curve L10 was derived based on the plurality of saccade candidates. Specifically, a regression curve L10 was derived from a plurality of saccade candidates using the least squares method. Next, the first reference curve L11 is derived by shifting the regression curve L10 in the direction in which the moving speed increases (increasing direction on the vertical axis in FIG. A second reference curve L12 is derived by shifting the curve in the decreasing direction on the vertical axis), and a saccade range R10 is defined between the first reference curve L11 and the second reference curve L12. Then, among the plurality of saccade candidates, saccade candidates included within the saccade range R10 are extracted as saccades.

Next, saccade amplitude ds and saccade frequency fs, which are indicators of saccades, were calculated. Specifically, at every predetermined period (for example, every 10 seconds), the average value of the amplitude ds of saccades included in that period is calculated as "saccade amplitude ds", and the saccade amplitude ds included in that period is calculated. The value obtained by dividing the number of times by the period time was calculated as the "saccade frequency fs".

<Saccade amplitude>
FIG. 10 illustrates the saccade amplitude ds of a healthy person, a moderate patient, and a severe patient in each of a high-attention scene and a low-attention scene. As shown in Figure 10, the difference in saccade amplitude ds between healthy subjects and severe patients in high-attention scenes is greater than the difference in saccade amplitude ds between healthy subjects and severe patients in low-attention scenes. is also getting bigger. This means that if the driver has an abnormality (an abnormality that causes a decline in attention function), the amplitude ds of the vehicle driver's saccades will significantly decrease in driving scenes where the level of caution in the external environment of the vehicle is relatively high. It shows.

<Saccade frequency>
FIG. 11 illustrates the saccade frequency fs of a healthy person, a moderate patient, and a severe patient in each of a high-attention scene and a low-attention scene. As shown in Figure 11, the difference in saccade frequency fs between healthy subjects and severe patients in low-attention scenes is greater than the difference in saccade frequency fs between healthy subjects and severe patients in high-attention scenes. is also getting bigger. This means that if the driver has an abnormality (an abnormality that causes a decline in attention function), the frequency of saccades fs of the vehicle driver will significantly decrease in driving scenes where the level of attention in the external environment of the vehicle is relatively low. It shows.

<Summary of the experiment>
As described above, if the driver has an abnormality (an abnormality that causes a decline in attention function), the amplitude ds or frequency fs of the vehicle driver's saccades will decrease significantly depending on the degree of attentiveness in the external environment of the vehicle. I understand.

Specifically, if the driver has an abnormality (an abnormality that causes a decline in attention function), the amplitude ds of the vehicle driver's saccades will significantly decrease when the level of attentiveness in the external environment of the vehicle is relatively high. I understand. In addition, it was found that when the driver has an abnormality (an abnormality that causes a decline in attention function), the frequency of saccades fs of the vehicle driver decreases significantly when the level of attention in the external environment of the vehicle is relatively low. Ta.

In addition, based on the amplitude of saccades of patients with attention dysfunction when the level of attentiveness in the external environment of the vehicle is high, the amplitude threshold dth, which is the standard for distinguishing whether or not the driver has an abnormality, is appropriately set. I found out that it can be configured. For example, in the case of the example shown in FIG. 10, the amplitude threshold dth may be set to "5 deg" based on the amplitude of the saccade of a severely ill patient whose score standard value is larger than the score standard value of a healthy person in his 60s. preferable. That is, the amplitude threshold dth is set to be larger than the amplitude of a saccade of a severe patient in a high-attention scene, and less than or equal to the amplitude of a saccade of a severe patient in a low-attention scene.

In addition, based on the frequency of saccades of patients with attention dysfunction when the degree of attention in the external environment of the vehicle is relatively low, a frequency threshold value fth, which serves as a standard for distinguishing whether or not the driver has an abnormality, is appropriately set. It turns out that it can be done. For example, in the case of the example shown in FIG. 11, the frequency threshold fth is set to "0.7 times/s" based on the frequency of saccades of a severe patient whose score standard value is higher than that of a healthy person in his 60s. It is desirable that this is set. That is, this frequency threshold fth is set higher than the frequency of saccades of a severe patient in a low-attention scene, and below the frequency of saccades of a severe patient in a high-attention scene.

[Saccade detection]
Next, with reference to FIG. 12, the operation (saccade detection) of the driver state detection section 300 will be described. This saccade detection is mainly performed by the saccade detection section 301.

<Step ST11>
First, the saccade detection unit 301 detects the driver's line of sight. In this example, the saccade detection unit 301 detects the driver's line of sight by performing line of sight detection processing on image data obtained by the in-vehicle camera 28. For example, the saccade detection unit 301 detects the driver's pupil from the image (image data) obtained by the in-vehicle camera 28, and detects the driver's line of sight based on the detected pupil. Next, the saccade detection unit 301 calculates the travel distance of the driver's line of sight. Then, the saccade detection unit 301 calculates the speed of the driver's line of sight based on the temporal change in the travel distance of the driver's line of sight. For example, the saccade detection unit 301 calculates the speed of the driver's line of sight by differentiating the moving distance of the line of sight that changes over time.

<Step ST12>
Next, the saccade detection unit 301 extracts saccade candidates that are saccade candidates based on the moving speed of the line of sight. For example, the saccade detection unit 301 defines a period in which the movement speed of the line of sight continues to be less than a predetermined speed threshold (for example, 40 deg/s) for a predetermined stagnation time (for example, 0.1 second) as a "gaze period". ” (see Figure 8). Then, the saccade detection unit 301 determines whether the movement speed is equal to or higher than a speed threshold (for example, 40 deg/s) and the movement distance is a predetermined distance among the gaze movements during the period sandwiched between two adjacent gaze periods. Eye movement that is equal to or greater than a distance threshold (for example, 3 degrees) is extracted as a "saccade candidate."

<Step ST13>
Next, the saccade detection unit 301 reads a saccade range R10 (see FIG. 9) based on the regression curve L10.

For example, the saccade range R10 is derived as follows. First, the saccade detection unit 301 derives a regression curve L10 based on a plurality of saccade candidates. Specifically, the saccade detection unit 301 derives the regression curve L10 from a plurality of saccade candidates using the least squares method. Next, the saccade detection unit 301 derives the first reference curve L11 by shifting the regression curve L10 in the direction in which the movement speed increases (increasing direction on the vertical axis in FIG. 9), and shifts the regression curve L10 in the direction in which the movement speed increases. A second reference curve L12 is derived by shifting in a decreasing direction (decreasing direction on the vertical axis in FIG. 9). Then, the saccade detection unit 301 sets a saccade range R10 between the first reference curve L11 and the second reference curve L12. Note that the saccade range R10 may be derived periodically.

The saccade detection unit 301 then compares each of the plurality of saccade candidates with the saccade range R10 to extract a saccade. Specifically, the saccade detection unit 301 extracts a saccade candidate included within the saccade range R10 from among the plurality of saccade candidates as a saccade. Note that the saccade detection unit 301 does not extract, as a saccade, a saccade candidate that is not included within the saccade range R10 from among the plurality of saccade candidates.

<Step S14>
Next, the saccade detection unit 301 calculates a saccade amplitude ds and a saccade frequency fs, which are indicators of a saccade. For example, the saccade detection unit 301 calculates the average value of the amplitudes ds of saccades included in each predetermined period (for example, every 10 seconds) as the "saccade amplitude ds", and The value obtained by dividing the number of included saccades by the period time is calculated as the "saccade frequency fs."

[Anomaly detection]
Next, the operation (abnormality detection) of the driver condition detection section 300 will be described with reference to FIG. 13. This abnormality detection is mainly performed by the abnormality detection section 303.

<Step ST21>
First, the abnormality detection section 303 determines whether the degree of caution detected by the degree of caution detection section (the degree of caution in the external environment of the vehicle) is a high degree of caution. If the degree of caution in the external environment of the vehicle is high, the process proceeds to step S22; otherwise, the process proceeds to step ST25.

<Step ST22>
When the degree of caution in the external environment of the vehicle is high, the abnormality detection unit 303 performs the first operation. In the first operation, the abnormality detection unit 303 determines whether the amplitude ds of the saccade (driver's saccade) detected by the saccade detection unit 301 is greater than or equal to a predetermined amplitude threshold dth. If the amplitude ds of the driver's saccade is equal to or greater than the amplitude threshold dth, the process proceeds to step ST23, and if not, the process proceeds to step ST24.

<Step ST23>
If the amplitude ds of the driver's saccade is equal to or greater than the amplitude threshold dth in step ST22, the abnormality detection unit 303 detects that the driver has an abnormality.

Note that the abnormality detection unit 303 may output a detection result indicating that the driver has an abnormality to the notification unit 40. In this case, the notification unit 40 may notify the driver of a detection result indicating that there is an abnormality. For example, the display unit 41 may output an image showing a detection result indicating that the driver is abnormal. The speaker 42 may output a sound indicating a detection result indicating that the driver is abnormal.

Further, when the abnormality detection unit 303 detects that the driver is abnormal, the vehicle control unit 35 switches the driving mode of the vehicle to automatic driving and directs the own vehicle toward a safe area (for example, the shoulder of the road). The actuator 11 may be controlled so that the vehicle travels and stops in a safe area. For example, the candidate route generated by the candidate route generation unit 36 may include a safe route heading to a safe area. The target route determining unit 37 selects a safe route as a target route from among the candidate routes generated by the candidate route generating unit 36 when the abnormality detecting unit 303 detects that the driver has an abnormality. good. The motion control unit 38 may determine the target motion based on the safe route selected as the target route by the target route determining unit 37, and may control the actuator 11 based on the target motion.

<Step ST24>
On the other hand, if the amplitude ds of the driver's saccade is less than the amplitude threshold dth in step ST22, the abnormality detection unit 303 detects that there is no abnormality in the driver.

Note that the abnormality detection unit 303 may output a detection result indicating that there is no abnormality with the driver to the notification unit 40. In this case, the notification unit 40 may notify the driver of a detection result indicating that there is no abnormality. For example, the display unit 41 may output an image showing a detection result indicating that there is no abnormality with the driver.

<Step ST25>
Further, in step S21, if the degree of caution in the external environment of the vehicle is not a high degree of caution (if the degree of caution in the external environment of the vehicle is low), the abnormality detection unit 303 detects the abnormality detected by the saccade detection unit 301. It is determined whether the frequency fs of saccades (driver's saccades) is greater than or equal to a predetermined frequency threshold fth. If the frequency fs of the driver's saccades is equal to or higher than the frequency threshold value fth, the process proceeds to step ST26; otherwise, the process proceeds to step ST27.

<Step ST26>
If the frequency fs of the driver's saccades is equal to or higher than the frequency threshold fth in step ST25, the abnormality detection unit 303 detects that the driver has an abnormality.

Note that, similarly to step ST23, the abnormality detection section 303 may output a detection result indicating that the driver has an abnormality to the notification section 40. In this case, the notification unit 40 may notify the driver of a detection result indicating that there is an abnormality. Further, when the abnormality detection unit 303 detects that the driver is abnormal, the vehicle control unit 35 switches the driving mode of the vehicle to automatic driving and directs the own vehicle toward a safe area (for example, the shoulder of the road). The actuator 11 may be controlled so that the vehicle travels and stops in a safe area.

<Step ST27>
On the other hand, if the frequency fs of the driver's saccades is less than the frequency threshold fth in step ST25, the abnormality detection unit 303 detects that there is no abnormality in the driver.

Note that, similarly to step ST24, the abnormality detection section 303 may output a detection result indicating that there is no abnormality with the driver to the notification section 40. In this case, the notification unit 40 may notify the driver of a detection result indicating that there is no abnormality.

[Effects of embodiment]
As described above, by detecting the driver's abnormality according to the degree of caution in the external environment of the vehicle and the amplitude ds or frequency fs of the vehicle driver's saccades, It is possible to improve the accuracy of detecting a driver's abnormality compared to the case where the driver's abnormality is detected based on the amplitude or frequency of saccades.

Specifically, when the level of attentiveness in the external environment of the vehicle is high, the first operation (detection of an abnormality of the driver based on the amplitude ds of the driver's saccade) is performed, and the level of attentiveness in the external environment of the vehicle is determined. By performing the second operation (detecting the driver's abnormality based on the driver's saccade frequency fs) when the driver's attention level is low, the accuracy of detecting the driver's abnormality can be improved.

Further, the amplitude threshold dth can be appropriately set based on the amplitude of a saccade of a person with an attentional dysfunction when the level of attentiveness in the external environment of the vehicle is high. Further, the frequency threshold fth can be appropriately set based on the frequency of saccades of a person with an attentional dysfunction when the degree of attentiveness in the external environment of the vehicle is low.

(Other embodiments)
Furthermore, the above embodiments may be combined as appropriate. The above embodiments are essentially preferred examples and are not intended to limit the scope of the technology disclosed herein, its applications, or its uses.

As explained above, the technology disclosed herein is useful as a driver condition detection device.

10 Vehicle control system 11 Actuator 20 Information acquisition section 21 Camera 22 Radar 23 Position sensor 24 External input section 25 Vehicle condition sensor 26 Driving operation sensor 27 Driver condition sensor 28 In-vehicle camera 29 Biological information sensor 30 Vehicle control device (driver condition detection Device)
31 Vehicle behavior recognition section 32 Driving operation recognition section 33 External environment recognition section 34 Driver behavior recognition section 35 Vehicle control section 40 Notification section 41 Display section 42 Speaker 300 Driver state detection section 301 Saccade detection section 302 Attention level detection section 303 Abnormality Detection unit

Claims (2)

A driver state detection device provided in a vehicle,
a saccade detection unit that detects a saccade of the driver of the vehicle;
an attentiveness detection unit that detects an attentiveness level in the external environment of the vehicle;
an abnormality detection unit that detects an abnormality of the driver based on the attentiveness detected by the attentiveness detection unit and the amplitude or frequency of the driver's saccades detected by the saccade detection unit,
The abnormality detection unit performs a first operation when the level of alertness detected by the alertness level detection unit is high level of alertness, and when the level of alertness detected by the level of alertness detection unit is low level of alertness. configured to perform a second operation;
In the first operation, the abnormality detection unit detects that the driver has an abnormality when the amplitude of the saccade of the driver is less than or equal to a preset amplitude threshold;
In the second operation, the abnormality detection unit detects that the driver has an abnormality when the frequency of saccades of the driver is less than or equal to a preset frequency threshold. Condition detection device. In claim 1 ,
The amplitude threshold is set based on the amplitude of a saccade of a person with an attentional dysfunction when the degree of attentiveness in the external environment of the vehicle is high,
The driver state detection device is characterized in that the frequency threshold is set based on the frequency of saccades of the person with impaired attention function when the degree of attentiveness in the external environment of the vehicle is low.

Images