JP7415460B2 - Vehicle control device and driver condition determination method - Google Patents

Description

The technology disclosed herein relates to, for example, a vehicle control device and a driver state determination method for determining the state of a driver who drives an automobile.

Recently, the development of autonomous driving systems has been promoted nationally. The applicant of this application currently believes that there are two main directions for automatic driving systems.

The first direction is a system in which the vehicle plays a central role in transporting passengers to their destination without requiring any operation by the driver, which is what is called fully automated vehicle driving. The second direction is an automated driving system that is based on the assumption that a human will be driving the car, such as ``providing an environment where driving can be enjoyed.''

In the second direction of automated driving systems, when a situation arises in which the driver becomes ill and is unable to drive normally, for example, the vehicle automatically replaces the passenger and performs automated driving. is assumed. Therefore, it is important to be able to detect abnormalities in the driver as early as possible and accurately, especially when the driver has developed a functional disorder or disease, to improve the survival rate of the driver and to ensure the safety of everyone, including those around them. This is extremely important from the perspective of ensuring that

As a method for determining abnormality of a driver, for example, Patent Document 1 describes a method in which a front image of a vehicle is mapped, a conspicuousness map (saliency map) is generated based on the pixel values of the mapped front image, and A technique has been disclosed that calculates the degree of conspicuousness of a driver's line of sight, and compares the degree to which the driver has turned his/her line of sight to a predetermined threshold with a predetermined threshold value to determine whether or not the driver is distracted.

In the technology of Patent Document 1, when the driver is distracted, the intentional attention function does not work, so for example, the driver's line of sight does not go to a dangerous place where a pedestrian may jump out of the blind spot. , the judgment is based on the gaze distribution characteristic that the gaze is directed to the conspicuous part of the visual stimulus. Further, Patent Document 1 discloses that a mindless state is determined by subtracting the degree of risk calculated by a degree of risk calculation means.

Patent No. 5966640

The inventors conducted line-of-sight detection using a driving simulator using a driving simulator for people with impairments and healthy people, and found that there were many cases where even people with impairments were not attracted to areas with high salience, and even healthy people were often attracted to areas with high salience. I found out that it occurs.

For example, if an area with high salience is also a caution point that should be looked at while driving, even a healthy person may direct their line of sight to the area with high salience, and as a result, it may be erroneously determined that a disease has occurred.

The technology disclosed herein has been developed in view of the above, and is designed to provide a driver condition determination device that determines the driver's condition based on the tendency to attract attention to salience, regardless of the driving environment of the vehicle. The purpose is to be able to handle abnormalities more appropriately.

In order to solve the above problem, the technology disclosed herein includes a gaze point detection unit that detects a gaze point of a driver of the vehicle, and an environment in front of the vehicle, for a vehicle control device installed in a vehicle. The vehicle exterior environment information is received from the vehicle exterior information acquisition means that acquires the information, and the driver's gaze point detected by the gaze point detection unit has a predetermined first tendency to view an area with high salience based on the vehicle exterior environment information. and a vehicle control unit that performs an operation corresponding to a driver's abnormality when the level of risk is higher than a predetermined second standard. and when the saliency of the gaze point is equal to or lower than a third criterion lower than the first criterion, and the amplitude of the saccade based on the gaze point is smaller than a predetermined fourth criterion, it also corresponds to an abnormality of the driver. It is configured to perform operations.

Here, saliency is a visual feature amount consisting of saliency that changes from moment to moment due to color, brightness, movement, etc. The high saliency area refers to an area where saliency is high (high conspicuousness) in the driver's visual field, or in other words, an area that stands out from the surroundings. More specifically, a high saliency region refers to, for example, a region that has a large color difference or brightness difference compared to surrounding regions, or that has a large movement relative to the surrounding regions.

Furthermore, a saccade is a jumping eye movement in which the driver intentionally moves his/her line of sight, and is an eye movement in which the driver moves his/her line of sight from a point of gaze where the line of sight remains stagnant for a predetermined period of time to the next point of gaze. The amplitude of a saccade refers to the amount of movement when the driver's line of sight moves from a gaze point where it remains stagnant for a predetermined period of time to the next gaze point.

According to the above aspect, by setting a criterion based on the amplitude of saccades in addition to the tendency to see areas with high salience, for example, even if the driver is in an abnormal state due to an increased risk , it is possible to avoid as much as possible a situation in which the driver is unable to respond to an abnormality.

In the vehicle control device of the above aspect, if the degree of risk estimated based on the vehicle exterior environment information is higher than the second standard, the first standard may be lowered in accordance with the increase in the degree of risk.

In this way, by lowering the first standard in accordance with the increase in the degree of danger, it is possible to identify dangerous spots where the salience is relatively low even though the driver is in an abnormal state due to the increase in the degree of danger. It is possible to reduce the possibility of an undetermined state in such a case.

The vehicle control device according to the above aspect includes an external environment recognition unit that recognizes an environment outside the vehicle in order to generate a route for automatic driving, and the vehicle control unit determines the degree of risk based on the recognition result of the external environment recognition unit. You can also say to estimate.

In this way, by using the degree of risk recognized by the external environment recognition unit, there is no need to add a new configuration for estimating the degree of risk.

In order to solve the above problem, the technology disclosed herein targets a driver state determination method for determining the state of a driver of a vehicle, and includes a method for determining the state of the driver of a vehicle, and a movement of the point of view of the driver of the vehicle. a detection step of detecting a saccade amplitude based on the saccade amplitude, an extraction step of extracting a region with high salience based on the vehicle exterior environment information received from the vehicle exterior information acquisition means that acquires the vehicle front side environment information, and the vehicle exterior information acquisition means an estimation step of estimating the degree of danger of a driving scene based on the external environment information received from the vehicle; and a tendency for the driver's gaze point detected in the detection step to look at the high salience region extracted in the extraction step. is higher than a predetermined first standard, a determining step of determining that the driver is abnormal; When the driver's gaze point is smaller than a predetermined third criterion, which is lower than the first criterion, and the driver's tendency to look at an area with high salience is smaller than a predetermined fourth criterion, the amplitude of a saccade based on the gaze point is smaller than a predetermined fourth criterion. It is also determined that there is an abnormality on the part of the driver.

In this way, by adding an abnormality judgment based on a criterion based on the amplitude of saccades to the driver's abnormality judgment based on the driver's tendency to look at areas with high salience, for example, the driver may Even in a case where a dangerous location with relatively low salience is observed despite being in an abnormal state, it is possible to reduce the possibility of an undetermined state.

As described above, according to the technology disclosed herein, the driver can more appropriately respond to abnormalities.

1 is a schematic diagram showing a front portion of a vehicle interior of a vehicle equipped with a driver state estimation device according to an exemplary embodiment; FIG. FIG. 2 is a front view of the vehicle seen from the front. FIG. 1 is a block diagram illustrating the configuration of a vehicle control system according to an exemplary embodiment. FIG. 2 is a block diagram illustrating the configuration of a driver state detection section. FIG. 3 is a diagram showing an example of an external environment image on the front side of the vehicle taken by a front camera. 6 is a diagram showing an example of a composite image based on the external environment image of FIG. 5. FIG. 7 is a diagram showing an example of a saliency map of the composite image of FIG. 6. FIG. FIG. 6 is a diagram showing an example of eye movement by a healthy person and an attention dysfunction patient in the example of FIG. 5; FIG. 7 is a diagram showing another example of a composite image based on an external environment image on the front side of the vehicle taken by a front camera. 10 is a diagram illustrating an example of gaze movement by a healthy person and an attention dysfunction patient in the example of FIG. 9. FIG. FIG. 7 is a diagram showing another example of a composite image based on an external environment image on the front side of the vehicle taken by a front camera. FIG. 12 is a diagram showing an example of eye movement by a healthy person and an attention dysfunction patient in the example of FIG. 11; It is a graph for explaining extraction of saccades. It is a graph for explaining saccade noise removal processing. FIG. 3 is a diagram showing an example of a change in saliency over time of a driver's gaze point. FIG. 16 is a diagram showing an example of changes in saliency over time when coordinates are specified randomly in the same environment as FIG. 15; It is a graph comparing the proportion exceeding the threshold value at random points and the proportion exceeding the threshold value at the driver's gaze point. FIG. 2 is a diagram for explaining a saliency index. 3 is a flowchart illustrating the operation of the vehicle control device. FIG. 2 is a diagram showing an example of the time distribution of the salience index of healthy individuals. FIG. 3 is a diagram illustrating an example of the temporal distribution of the salience index of an attention function patient.

<Findings obtained by the inventors>
As disclosed in Patent Document 1, a technique is known that detects an abnormality (an abnormality that causes a decline in attention function) of the driver by observing changes in the driver's line of sight movement with respect to saliency. In this technique, for example, an abnormality of the driver is detected when the line of sight movement toward an area with high salience (high saliency area) exceeds a predetermined threshold value.

As a result of intensive research, the inventors of the present application have simulated the behavior of vehicle drivers during abnormal situations (for example, abnormal situations that cause a decline in attention function) by observing the behavior of people with attention dysfunction when driving a vehicle. We found that it is possible to observe Based on the above observation results, the inventors of the present application verified a method of detecting a driver's abnormality when the line of sight movement toward a high saliency region exceeds a predetermined threshold value. Specifically, we examined the behavior of people with attention dysfunction when driving a vehicle and the behavior of healthy people without attention dysfunction when driving a vehicle, using the saliency index described later to determine the shift of the gaze toward a high salience region. We tested a method that detects trends and detects driver abnormalities when the salience index exceeds a predetermined threshold.

As a result of this verification, the inventors of the present invention found that there are many cases in which even people with attentional dysfunction are not attracted to areas with high salience, and even healthy people are often attracted to areas with high salience. For example, if an area with high salience is also a cautionary point to look at while driving, even healthy people may direct their eyes to the area with high salience, and as a result, it may be erroneously determined that a disease has occurred. I understand. Furthermore, even if the driver has an abnormality, as the degree of danger increases, the driver tends to look at the dangerous area regardless of the level of salience, and the salience of the gaze point decreases, making it impossible to determine the abnormal condition. Understood.

20 and 21 show the driver's line of sight detected using a driving simulator and plotted as a time change in the saliency index. FIG. 20 shows the measurement results of a healthy person, and FIG. 21 shows the measurement results of a patient with attention dysfunction. However, since the running speeds are different from each other, the time axis and the running location do not necessarily match.

Although the details will be described later, the saliency index is an index whose numerical value increases as the degree of attraction to a high saliency region increases. In the examples shown in Figures 20 and 21, the threshold is set to 0.6, which indicates that the degree of attraction to the high salience region is relatively high, and when the salience index exceeds 0.6, it is determined that there is a failure in attention function. It was assumed that the

As a result, it was found that even in healthy subjects, there were some cases where the saliency index exceeded 0.6 (see AR1 in FIG. 21). For example, when passing a vehicle with high salience such as a fire truck, even healthy people tend to be attracted to places with high salience. In addition, even in patients with functional impairment, there is a tendency to check even in high-risk places and scenes such as at intersections and when overtaking parked vehicles, even in places where the salience index is low, and areas with low salience index have been confirmed. (See AR2 in Figure 22).

Therefore, the inventors of the present application further conducted intensive studies and found that by using the risk potential indicating the degree of danger, abnormalities of the driver can be detected with higher accuracy. Risk potential is a field that is artificially set to appropriately reflect the degree of danger in the driving environment of the own vehicle (the sense of danger felt by the driver of the own vehicle). An appropriate function is set that has a shape that has a maximum value at the center position of the vehicle and expands around each surrounding vehicle. Furthermore, the risk potential may be used as an index for estimating the degree of danger that may be expected in blind spots, intersections, etc. Further, as a method for determining the degree of risk (risk potential), the distance between vehicles and surrounding vehicles, TTC (Time To Collision), road alignment, distance to a linear change point, etc. may be used.

More specifically, in driver abnormality determination, if the degree of danger is increasing, the saliency index determination threshold is relatively lowered, while if the degree of danger is decreasing, the saliency index determination threshold is relatively lowered. Increase the threshold relatively. That is, the threshold value of the saliency index is changed in the direction opposite to the trend of the risk level according to the trend of increase/decrease in the risk level. It has been found that this can reduce the number of undetermined abnormalities of the driver. The undetermined abnormality of the driver includes, for example, the undetermined abnormality when the driver tends to look at dangerous places with low salience even when the driver has an abnormality. Furthermore, the inventors of the present invention propose that when the degree of risk is higher than a predetermined standard value (equivalent to the second standard) and the salience index of the gaze point becomes less than the predetermined standard value (equivalent to the third standard), In addition, we found that it is easier to determine whether a driver is abnormal by using saccade amplitude to determine whether the driver is abnormal. As a result of intensive studies based on the above-mentioned experiments, the inventors of the present application have found that when a driver is abnormal, especially those with attentional dysfunction, when the degree of danger exceeds a predetermined standard, the driver's viewpoint tends to become localized. The purpose of driver abnormality determination using saccade amplitude is to detect a state in which the driver's viewpoint tends to be localized.

In the following, exemplary embodiments will be specifically described with reference to the drawings. In the following description, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. Also, when looking from the rear to the front, the left side is called the left side, and the opposite is called the right side.

FIG. 1 schematically shows the interior of an automobile as a vehicle. This vehicle is a right-hand drive vehicle, and a steering wheel 58 is disposed on the right side.

In the vehicle interior, a front window glass 51 is arranged on the front side of the vehicle when viewed from the driver's seat. The front window glass 51 is partitioned by a plurality of vehicle structural members when viewed from the inside of the vehicle. Specifically, the front window glass 51 is divided by left and right front pillar trims 52, a roof trim 53, and an instrument panel 54.

The left and right front pillar trims 52 constitute outer boundaries of the front window glass 51 in the vehicle width direction, respectively. Each front pillar trim 52 is arranged along each front pillar. The roof trim 53 constitutes the upper boundary of the front window glass 51. The roof trim 53 covers the interior side of the roof panel of the vehicle. A rearview mirror 55 is attached to a portion at the center of the front window glass 51 in the vehicle width direction and slightly below the roof trim 53. An in-vehicle camera 28 (see FIG. 3) for photographing the interior of the vehicle, particularly the driver's face, is provided in a portion of the roof trim 53 near the rearview mirror 55. The in-vehicle camera 28 will be explained in detail later. The instrument panel 54 constitutes the lower boundary of the front window glass 51. The instrument panel 54 is provided with a meter box and a display 57.

Further, the vehicle has side mirrors 56 on the outer sides of the left and right front pillars in the vehicle width direction. Each side mirror 56 is arranged so that the driver seated in the driver's seat can see through the window of the side door.

As shown in FIG. 2, the vehicle is provided with a camera (hereinafter referred to as front camera 21a) for photographing the external environment on the front side of the vehicle. The front camera 21a is located at the front end of the vehicle, slightly below the hood 59 of the vehicle. The front camera 21a is an example of outside-vehicle information acquisition means that acquires environmental information on the front side of the vehicle.

<Vehicle control system>
FIG. 3 illustrates the configuration of the vehicle control system 10 according to the embodiment. The vehicle control system 10 is provided in a vehicle (specifically, a four-wheeled motor vehicle). The vehicle can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in response to the driver's operations (for example, accelerator operation). Assisted driving is driving in which the vehicle supports the driver's operations. Autonomous driving is driving without driver input. The vehicle control system 10 controls the vehicle in assisted driving and automatic driving. Specifically, the vehicle control system 10 controls the operation (especially driving) of the vehicle by controlling an actuator 11 provided in the vehicle.

The vehicle control system 10 includes an information acquisition section 20, a control section 30, and a notification section 40. In the following description, the vehicle provided with the vehicle control system 10 will be referred to as the "host vehicle", and other vehicles existing around the host vehicle will be referred to as "other vehicles".

-Actuator-
The actuator 11 includes a drive system actuator, a steering system actuator, a brake system actuator, and the like. Examples of drive system actuators include engines, motors, and transmissions. An example of a steering system actuator is a steering wheel. An example of a brake system actuator is a brake.

-Information acquisition department-
The information acquisition unit 20 acquires various information used for vehicle control (particularly driving control). In this example, the information acquisition section 20 includes a plurality of cameras 21, a plurality of radars 22, a position sensor 23, an external input section 24, a vehicle condition sensor 25, a driving operation sensor 26, and a driver condition sensor 27. including.

〔camera〕
The plurality of cameras 21 have mutually similar configurations. A plurality of cameras 21 are provided on the vehicle so as to surround the vehicle. Each of the plurality of cameras 21 acquires image data representing a part of the external environment of the vehicle by capturing an image of a part of the environment surrounding the vehicle (external environment of the vehicle). Image data obtained by each of the plurality of cameras 21 is transmitted to the control unit 30. The plurality of cameras 21 include the above-mentioned front camera 21a.

In this example, camera 21 is a monocular camera with a wide-angle lens. For example, the camera 21 is configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor). Note that the camera 21 may be a monocular camera with a narrow-angle lens, or may be a stereo camera with a wide-angle lens or a narrow-angle lens.

[Radar]
The plurality of radars 22 have mutually similar configurations. The plurality of radars 22 are provided in the vehicle so as to surround the vehicle. Each of the plurality of radars 22 detects a part of the external environment of the vehicle. Specifically, the radar 22 detects a part of the external environment of the vehicle by transmitting radio waves toward a part of the external environment of the vehicle and receiving reflected waves from the part of the external environment of the vehicle. do. The detection results of the plurality of radars 22 are transmitted to the control unit 30.

For example, the radar 22 may be a millimeter wave radar that transmits millimeter waves, a lidar (Light Detection and Ranging) that transmits laser light, or an infrared radar that transmits infrared light. Alternatively, it may be an ultrasonic sensor that transmits ultrasonic waves.

[Position sensor]
The position sensor 23 detects the position (for example, latitude and longitude) of the vehicle. For example, the position sensor 23 receives GPS information from the global positioning system and detects the position of the vehicle based on the GPS information. Information (vehicle position) obtained by the position sensor 23 is transmitted to the control unit 30.

[External input section]
The external input unit 24 inputs information through an external network (for example, the Internet) provided outside the vehicle. For example, the external input unit 24 receives communication information from other vehicles (not shown) located around the vehicle, car navigation data from a navigation system (not shown), traffic information, high-precision map information, and the like. Information obtained by the external input section 24 is transmitted to the control section 30.

[Vehicle condition sensor]
The vehicle condition sensor 25 detects the condition of the vehicle (for example, speed, acceleration, yaw rate, etc.). For example, the vehicle condition sensor 25 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, and the like. Information (vehicle status) obtained by the vehicle status sensor 25 is transmitted to the control unit 30.

[Driving operation sensor]
The driving operation sensor 26 detects a driving operation applied to the vehicle. For example, the driving operation sensor 26 includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the amount of operation of the accelerator of the vehicle. The steering angle sensor detects the steering angle of the steering wheel of the vehicle. The brake oil pressure sensor detects the amount of brake operation of the vehicle. Information (vehicle driving operation) obtained by the driving operation sensor 26 is transmitted to the control unit 30.

[Driver status sensor]
The driver condition sensor 27 detects the condition of the driver who drives the vehicle (for example, the driver's health condition, emotions, physical behavior, etc.). Information obtained by the driver condition sensor 27 (driver condition) is transmitted to the control unit 30. In this example, driver condition sensor 27 includes an in-vehicle camera 28 and a biological information sensor 29.

《In-vehicle camera》
The in-vehicle camera 28 is provided inside the vehicle. The in-vehicle camera 28 acquires image data including the driver's eyes by capturing an image of a region including the driver's eyes. Image data obtained by the in-vehicle camera 28 is transmitted to the control unit 30. For example, the in-vehicle camera 28 is placed in front of the driver, and the imaging range is set so that the driver's eyeballs are within the imaging range. Note that the in-vehicle camera 28 may be provided in goggles (not shown) worn by the driver.

《Biological information sensor》
The biological information sensor 29 detects biological information (for example, sweating) of the driver. Information obtained by the biometric information sensor 29 (driver's biometric information) is transmitted to the control unit 30 .

-Control unit-
The control unit 30 determines a target route, which is the route the vehicle should travel, based on various information acquired by the information acquisition unit 20 during assisted driving or automatic driving, and is necessary for driving the target route. Determine the target motion, which is the motion of the vehicle. The control unit 30 then controls the operation of the actuator 11 so that the motion of the vehicle matches the target motion. For example, the control section 30 is configured by an electronic control unit (ECU) having one or more arithmetic chips. In other words, the control unit 30 is configured by an electronic control unit (ECU) having one or more processors, one or more memories that store programs and data for operating the one or more processors, and the like. Ru.

In this example, as shown in FIG. 3, the control unit 30 includes an image processing unit 31, an external environment recognition unit 32, a candidate route generation unit 33, a vehicle behavior recognition unit 34, and a driver behavior recognition unit 35. , a target route determining section 36, and a motion control section 37.

[Image processing section]
The image processing unit 31 receives images captured by the plurality of cameras 21 and performs image processing. The image processing performed by the image processing unit 31 includes first image processing for images used by the external environment recognition unit 32 to recognize the external environment such as objects, and second image processing for use in generating a saliency map. Image processing is included.

《First image processing》
The image processing unit 31 performs distortion correction processing to correct image distortion (in this example, distortion due to wide-angle camera 21) and white balance adjustment to adjust the white balance of the image, for images captured by each camera 21. Perform processing, etc. The image processing unit 31 also deletes pixels that are unnecessary for processing in the external environment recognition unit 32 in the subsequent stage from among the elements that make up the image, and thins out color-related data (such as representing all vehicles in the same color). to generate image data. The image data generated by the first image processing is sent to the external environment recognition section 32.

《Second image processing》
For example, the image processing unit 31 deletes, from the image taken by the front camera 21a, pixels that are unnecessary for processing by the map generation unit 301 in the subsequent stage (for example, generation of a saliency map) from among the elements constituting the image. Perform the processing to do. Further, the image processing unit 31 performs processing to generate a composite image by combining another image with the image showing the external environment on the front side of the vehicle taken by the front camera 21a.

FIG. 5 is an example of an image D11 showing the external environment on the front side of the vehicle, taken by the front camera 21a in a vehicle driving scene. The external environment shown in this image D11 includes a roadway 150 and a white line 151 on the roadway 150. In addition, the external environment shown in this image D11 includes a wall 162 formed on the left side of the roadway 150, a forest 163 formed on the left side of the wall 162, and a hill spreading on the right side of the roadway 150. 164 and a forest 165 formed on the hill 164. Furthermore, the external environment shown in this image D11 includes the sky 167 that extends above the road 150 and forests 163 and 165. In the following description, it is assumed that the sky 167 of the image D11 is a sunset sky, and a reddish sky is spreading. In other words, the image D11 is assumed to be an image in which the spread RW of the high saliency region is relatively large, as shown in FIG. Note that FIG. 7 will be explained in detail later.

The image processing unit 31 synthesizes, with the image D11, an image showing vehicle components that enter the driver's field of view when the vehicle is running. Specifically, as shown in FIG. 6, the image processing unit 31 synthesizes an image of vehicle components (hereinafter referred to as a vehicle image) when a driver seated in the driver's seat looks at the front side of the vehicle into an image D11. Then, a composite image D12 is generated. The vehicle components photographed in the vehicle image include, for example, the front pillar trim 52 on the right side (driver's seat side), the right part of the roof trim 53, the right part of the instrument panel 54, the rearview mirror 55, and the right side. side mirrors 56 and a steering wheel 58. The vehicle image can be, for example, taken in advance by photographing the interior structure of the vehicle that enters the driver's field of view from the driver's seat side, and storing the photographed image data as a layer in the memory (not shown) of the control unit 30. . When combining the vehicle image with the image D11, the image processing unit 31 may read the vehicle image from the memory into a separate layer from the image D11, align the image D11 and the vehicle image, and overlap them. Note that a part of the hood 59 may also be taken into consideration as a vehicle component. Further, in creating the composite image D12, information other than the front camera 21a, for example, the detection results of the radar 22 or input information from the external input unit 24 may be used. Image data of the composite image D12 (hereinafter referred to as composite image data) generated by the image processing unit 31 is transmitted to a map generation unit 301, which will be described later (see FIG. 4).

[External environment recognition department]
The external environment recognition unit 32 recognizes the external environment of the vehicle based on data output from the plurality of cameras 21 and radar 22. The external environment of the vehicle recognized by the external environment recognition unit 32 includes objects. Examples of objects include a moving object that displaces over time and a stationary object that does not displace over time. Examples of moving objects include four-wheeled motor vehicles, motorcycles, bicycles, and pedestrians. Examples of stationary objects include signs, street trees, median strips, center poles, and buildings.

[Candidate route generation unit]
The candidate route generation unit 33 generates one or more candidate routes based on the output of the external environment recognition unit 32. A candidate route is a route on which a vehicle can travel, and is a candidate for a target route. Further, the candidate route generation unit 33 calculates the above-mentioned risk potential when generating a candidate route. For example, as shown in FIG. 11, when there is another vehicle 161 (parked vehicle), a dangerous area that extends around the surrounding vehicle is set, and a degree of risk is scored and set according to the extent of the area. The degree of risk may be scored by calculation using a function, or by providing a reference table depending on the scene or the like.

5 to 12 show examples of changes in driving scenes. 5 to 8 show the same driving scene (hereinafter referred to as the first driving scene). FIGS. 9 and 10 show a driving scene (hereinafter referred to as a second driving scene) a little time after the first driving scene. Moreover, FIGS. 11 and 12 show a driving scene (hereinafter referred to as a third driving scene) after a little time has elapsed from the second driving scene. Specifically, in the first driving scene, there are no other vehicles around and the danger level is low, and in the second driving scene, another vehicle parked in the distance (hereinafter referred to as the stopped vehicle 161) can be seen. It is assumed that the risk level has increased slightly due to the transition. Further, in the third driving scene, it is assumed that the vehicle is close to the stopped vehicle 161 and the degree of danger exceeds a predetermined second standard. The candidate route generation unit 33 continuously estimates the degree of danger of the driving scene and the predicted danger locations while the vehicle is running, and generates a plurality of candidate routes according to the degree of danger and the predicted danger locations. It is configured as follows. The abnormality detection method according to the driving scene will be specifically explained later.

The candidate route generated by the candidate route generation unit 33 is transmitted to the target route determination unit 36. Furthermore, the degree of risk calculated by the candidate route generation unit 33 is transmitted to an abnormality detection unit 303, which will be described later (see FIG. 4). Note that the degree of risk may be calculated by a component other than the candidate route generation unit 33. For example, the calculation may be performed using the external environment recognition unit 32 or a component dedicated to risk calculation.

[Vehicle behavior recognition unit]
The vehicle behavior recognition unit 34 recognizes vehicle behavior (eg, speed, acceleration, yaw rate, etc.) based on the output of the vehicle state sensor 25. For example, the vehicle behavior recognition unit 34 recognizes the behavior of the vehicle from the output of the vehicle condition sensor 25 using a learning model generated by deep learning.

[Driver behavior recognition unit]
The driver behavior recognition unit 35 recognizes the driver's behavior (for example, the driver's health condition, emotions, physical behavior, etc.) based on the output of the driver condition sensor 27. For example, the driver behavior recognition unit 35 recognizes the driver's behavior from the output of the driver condition sensor 27 using a learning model generated by deep learning. In this example, the driver behavior recognition section 35 includes a driver state detection section 300. The driver state detection section 300 will be explained in detail later.

[Target route determination unit]
The target route determination unit 36 selects a target route from among the one or more candidate routes generated by the candidate route generation unit 33 based on the output of the vehicle behavior recognition unit 34 and the output of the driver behavior recognition unit 35. Select a candidate route. For example, the target route determining unit 36 selects the candidate route that the driver feels is most comfortable from among the plurality of candidate routes.

[Motor control unit]
The motion control unit 37 determines a target motion based on the candidate route selected as the target route by the target route determination unit 36, and controls the actuator 11 based on the determined target motion. For example, the motion control unit 37 derives a target driving force, a target braking force, and a target steering amount, which are a driving force, a braking force, and a steering amount for achieving the target movement, respectively. Then, the motion control unit 37 transmits the drive command value indicating the target driving force, the braking command value indicating the target braking force, and the steering command value indicating the target steering amount to the actuators of the drive system, the actuators of the braking system, and the steering system. and the actuator respectively.

《Notification Department》
The notification unit 40 notifies the driver of the vehicle of various information. In this example, the notification section 40 includes a display section 41 and a speaker 42. The display unit 41 outputs various information in the form of images. The speaker 42 outputs various information in the form of audio.

《Driver status detection section》
The driver condition detection unit 300 detects abnormality of the driver of the vehicle. Specifically, as shown in FIG. 4, the driver state detection section 300 includes a map generation section 301, a gaze point detection section 302, and an abnormality detection section 303.

Note that the driver's abnormality refers to an abnormality that causes a decline in the driver's attention function. Examples of such driver abnormalities include brain diseases such as stroke, heart diseases such as myocardial infarction, epilepsy, hypoglycemia, and drowsiness. Furthermore, in the following explanation, a driver who is in an abnormal state will be referred to as a person with impaired attention function.

[Map generator]
The map generation unit 301 receives composite image data from the image processing unit 31, and generates a saliency map D13 for the composite image D12 based on the composite image data. Specifically, the map generation unit 301 calculates the saliency of a portion of the composite image data representing the external environment, that is, a portion other than the vehicle image synthesized by the image processing unit 31. At this time, the map generation unit 301 does not calculate the saliency of the vehicle images used in the synthesis by the image processing unit 31, but uses it to calculate the saliency of the portion of the composite image D12 that represents the external environment. That is, the map generation unit 301 generates the saliency map D13 by taking into account the vehicle components that enter the driver's field of view when the vehicle is running.

As mentioned above, saliency changes depending on the color, brightness, movement, etc. of the target object. Therefore, in the present embodiment, the map generation unit 301 calculates saliency for each feature, such as saliency based on color, saliency based on brightness, saliency based on motion, etc., and generates a saliency map for each feature. By adding these together, a saliency map D13 based on the final composite image D12 is generated.

For example, with respect to color-based saliency, when the color difference between the nearby region and the vehicle component is large in a region near the vehicle component in the composite image data, the map generation unit 301 generates a color-based saliency, compared to when the color difference is small. Increase the salience of the neighborhood. Note that the color difference is calculated by the following formula when the RGB of a certain pixel color is (R1, G1, B1) and the RGB of another pixel is (R2, G2, B2).
(Color difference) = {(R2-R1) ² + (G2-G1) ² + (B2-B1) ² } ^1/2
The map generation unit 301 may calculate so that the saliency increases continuously as the color difference increases, or it may calculate such that the saliency increases by a certain value each time the threshold value is exceeded by providing a plurality of threshold values. .

Further, for example, regarding the saliency based on brightness, the map generation unit 301 can determine whether the difference in brightness between the nearby region and the vehicle component is large in the vicinity of the vehicle component in the composite image data, and when the brightness difference is small. In comparison, the salience of the neighboring region is increased. For example, when the synthesized vehicle component is black, the map generation unit 301 increases the saliency of the part closer to white in the nearby region because the brightness difference becomes larger as the part becomes closer to white. The map generation unit 301 may calculate so that the saliency increases continuously as the luminance difference becomes larger, or it may calculate such that the saliency increases by a certain value each time the threshold value is exceeded by providing a plurality of threshold values. good.

FIG. 7 shows an example of a saliency map generated by the map generation unit 301. In FIG. 7, hatching is changed depending on the height of saliency. In FIG. 7, denser hatched lines indicate regions with higher salience, and regions with no hatching indicate regions with the lowest salience. In FIGS. 8, 10, and 12, only the contour lines of the saliency maps are shown to make the drawings easier to read.

Note that the calculation of saliency itself can be performed using a known computer program published on the Internet or the like. Furthermore, a known computer program can be used to calculate a saliency map for each feature and to integrate each saliency map.

[Gaze point detection unit]
The gaze point detection unit 302 calculates the driver's line of sight direction from the driver's eyeball image captured by the in-vehicle camera 28. For example, the gaze point detection unit 302 calculates the direction of the driver's line of sight by detecting changes in the driver's pupils from a state in which the driver looks through the lens of the in-vehicle camera 28 as a reference. The line-of-sight direction may be calculated from either the driver's left eye or right eye, or may be the average value of the line-of-sight directions (line-of-sight vector) determined from each of the driver's eyes. Furthermore, when it is difficult to calculate the driver's line-of-sight direction from changes in the driver's pupils, the line-of-sight direction may be calculated by further considering the direction of the driver's face. Further, a learning model (a learning model for detecting the line of sight) generated by deep learning may be used to calculate the direction of the driver's line of sight.

《Detection of fixation points and saccades》
The gaze point detection unit 302 detects the driver's saccades based on the movement of the driver's line of sight. A saccade is a jumping eye movement in which the driver intentionally moves his/her line of sight, and is an eye movement in which the driver moves his/her line of sight from a point of gaze where the line of sight remains stagnant for a predetermined period of time to the next point of gaze. As shown in FIG. 13, a period sandwiched between two adjacent gaze periods is a saccade period. Note that the gaze period is a period in which the line of sight is considered to be stationary. The saccade amplitude ds is the distance the line of sight moves during the saccade period.

The gaze point detection unit 302 calculates the moving speed of the line of sight based on the change in the moving distance of the line of sight, and determines a state in which the moving speed of the line of sight is less than a predetermined speed threshold (for example, 2 deg/s). A period of stagnation time (for example, 0.1 second) is extracted as a "gazing period", and a point ahead of the driver's line of sight during the gaze period is extracted as a "gazing point". In addition, the gaze point detection unit 302 detects that the movement speed of the gaze movement during the period sandwiched between two adjacent gaze periods is equal to or higher than a speed threshold (for example, 40 deg/s), and the movement distance is a predetermined distance. A line-of-sight movement that is equal to or greater than a distance threshold (for example, 3 degrees) is extracted as a "saccade."

Note that the gaze point detection unit 302 may perform noise removal processing on the extracted saccade. Specifically, the gaze point detection unit 302 derives a regression curve L10 based on a plurality of saccade candidates, as shown in FIG. 14. The gaze point detection unit 302 derives the regression curve L10 from a plurality of saccade candidates using, for example, the least squares method. Next, the gaze point detection unit 302 derives the first reference curve L11 by shifting the regression curve L10 in the direction in which the moving speed increases (increasing direction on the vertical axis in FIG. 14), and shifts the regression curve L10 to the moving speed A second reference curve L12 is derived by shifting in a direction in which the curve decreases (decreasing direction on the vertical axis in FIG. 14), and a saccade range R10 is defined between the first reference curve L11 and the second reference curve L12. Then, the gaze point detection unit 302 extracts a saccade candidate included within the saccade range R10 from among the plurality of saccade candidates as a saccade.

Next, the gaze point detection unit 302 calculates a saccade amplitude ds and a saccade frequency fs, which are indicators of a saccade. Specifically, the gaze point detection unit 302 calculates the average value of the saccade amplitudes ds included in each predetermined period (for example, every 10 seconds) as the "saccade amplitude ds", The value obtained by dividing the number of saccades included in the cycle by the time of the cycle is calculated as the "saccade frequency fs".

FIG. 8, FIG. 10, and FIG. 12 are diagrams in which the positions of gaze points on the saliency map and temporally adjacent gaze points are connected by lines. In FIGS. 8, 10, and 12, the length of the line between the fixation points indicates the amplitude ds of the saccade. Furthermore, in FIGS. 8, 10, and 12, circles are examples of gaze points of a healthy person, and triangle marks are examples of gaze points of a person with an attention dysfunction. Furthermore, in FIGS. 8, 10, and 12, the solid line is an example of the movement of the line of sight of a healthy person, and the broken line is an example of the movement of the gaze point of a person with an attentional dysfunction.

[Abnormality detection part]
The abnormality detection unit 303 detects an abnormality of the driver based on the saliency map D3 generated by the map generation unit 301 and the driver's gaze point and saccade amplitude ds detected by the gaze point detection unit 302. do. The operation of the abnormality detection unit 303 will be specifically explained in "Operation of Vehicle Control System" below.

-Vehicle control system operation-
The operation of the vehicle control system will be described below with reference to FIG.

(Step ST11)
First, in step ST11, the vehicle control system executes a gaze point detection step of detecting the gaze point of the vehicle driver. Specifically, in step ST11, the gaze point detection unit 302 detects the gaze point based on the driver's line of sight detected by the in-vehicle camera 28, for example, as explained in the section “Detection of gaze point and saccade” above. The viewpoint and the saccade amplitude ds are detected.

(Step ST12)
In step ST12, the vehicle control system executes a map generation step of generating a saliency map based on the image data received from the front camera 21a. Specifically, in step ST12, the map generation unit 301 generates a salier of the composite image D12 based on the composite image data output from the image processing unit 31, for example, as described in the above “Map generation unit” section. A sea map D13 is generated.

(Step ST13)
In step ST13, the vehicle control system estimates the degree of danger in the vehicle driving scene based on the image data received from the front camera 21a. Specifically, in step ST13, for example, as explained in the section of the above-mentioned "candidate route generation section", the candidate route generation section 33 estimates the degree of danger of the driving scene during the period when the vehicle is traveling. The candidate route generation unit 33 updates the degree of risk, for example, at every predetermined time (for example, every 10 seconds). Note that the order of execution of step ST11, step ST12, and step ST13 is not particularly limited. For example, step S12 may be executed before step S11, step S13 may be executed before step S11 or step ST12, or steps ST11 to ST13 may be executed simultaneously.

After step ST13, the process from step ST21 and the process from step ST31 are executed in parallel.

(Step ST21)
In step ST21, the abnormality detection unit 303 determines whether the tendency of the driver to view a region with high salience is higher than a predetermined first criterion. Specifically, for example, a tendency of eye movement toward a high saliency area is detected using a saliency index, and it is determined whether the saliency index exceeds a predetermined threshold value. Specifically, the abnormality detection unit 303 determines whether the driver is in an area with high salience based on the saliency map generated by the map generation unit 301 and the driver's gaze point detected by the gaze point detection unit 302. It is determined whether the tendency to view is higher than a predetermined first criterion. In this embodiment, the abnormality detection unit 303 determines whether the saliency index is higher than 0.6 as a first criterion. Note that the saliency index used as the first criterion may be a numerical value other than 0.6, or an index other than the saliency index may be used in the determination in step ST21.

《Calculation of salience index》
A method for calculating the saliency index will be explained using FIGS. 15 to 18. Calculation of this saliency index is mainly performed by the abnormality detection unit 303.

FIG. 15 is a plot of the change in saliency of the driver's gaze point over time. For example, as shown in FIG. 8, this graph is a graph in which changes in the gaze point are applied to a saliency map, and the height of the saliency of the gaze point is plotted every time a saccade occurs. On the other hand, Fig. 16 is generated by randomly specifying coordinates (hereinafter referred to as random coordinates) from the same saliency map as Fig. 15 (for example, Fig. 8) and calculating the saliency of the random coordinates every time a saccade occurs. be done.

After creating the graphs shown in FIGS. 15 and 16, ROC (Receiver Operating Characteristic) curves of the proportion exceeding the threshold value at random points and the proportion exceeding the threshold value at the driver's gaze point are determined. Specifically, first, as a first step, the threshold value is set to a value larger than the maximum value in the graphs of FIGS. 15 and 16. Next, as a second step, while lowering the threshold value, the percentage of points exceeding the threshold value is determined for each threshold value. The process of this second step is then repeated until the threshold value becomes equal to or less than the minimum value in the graphs of FIGS. 15 and 16.

After the first and second steps, as a third step, the horizontal axis is the proportion exceeding the threshold at random points (referred to as the first probability), and the ratio exceeding the threshold at the driver's gaze point (referred to as the second probability) is plotted on the horizontal axis. A graph like the one shown in FIG. 17 is created for the vertical axis. The graph in FIG. 17 represents the gaze probability versus random probability at the same threshold. In the graph of FIG. 17, both the horizontal and vertical axes are probabilities, so the minimum value of the curve is 0 and the maximum value is 1.

In FIG. 17, the dashed line represents how the driver's line of sight moves with respect to saliency. When the curve is upwardly convex, like the curve C1 in FIG. 17, it indicates that the rate at which the driver's gaze point exceeds the threshold is higher than at a random point. When a curve with a shape like curve C1 in FIG. 17 is calculated, it means that the driver's line of sight tends to look at a high salience area, that is, the high salience area is highly attractive. There is. On the other hand, when the curve is convex downward like curve C2 in FIG. 17, it indicates that the rate at which the driver's gaze point exceeds the threshold is lower than that at a random point. When a curve shaped like curve C2 in FIG. 17 is calculated, it means that the driver's line of sight is not affected by the high salience region.

In this embodiment, as the fourth step, AUC (Area Under Curve) is determined. AUC is the integral value of the lower right portion of this curve. That is, AUC is the area of the region surrounded by the curve C1 (or curve C2) and the two-dot chain line in FIG. 17, as indicated by diagonal lines in FIG. In this embodiment, this integral value is called a saliency index. By using the salience index, dependence on driving scenes can be reduced. Specifically, for example, simply calculating the percentage of high-saliency areas that the driver views will show that when the entire driving scene includes high-saliency areas, the driver views the high-saliency areas more frequently. There may be consequences. On the other hand, by using a saliency index that is compared with random points as in this embodiment, it is possible to reduce the dependence on the driving scene, such as the number and spread of high saliency areas, and to Be able to make highly accurate judgments.

In step ST21, if the saliency index is higher than the first standard, the process proceeds to the next step ST22. On the other hand, if the saliency index is equal to or less than the first standard in step ST21, the process returns to step ST11. Step ST21 is, for example, an example of processing that constitutes a part of the determination step. Incidentally, steps ST11 to ST12 are continuously executed while the vehicle is running, regardless of the processing in steps ST21 and ST22.

(Step ST22)
In step S22, the control unit 30 performs control corresponding to the abnormality of the driver. For example, when the degree of abnormality of the driver is high (for example, when recognizing that the driver has developed a disease), the control unit 30 controls the actuator 11 via the motion control unit 37 etc. to switch to automatic driving. Perform processing. The control unit 30 also performs actuations directed at the driver, such as asking the driver, “Are you okay?” and “Would you like to take a break?” via the notification unit 40. You may also check the reaction. For example, the control unit 30 may perform a process of controlling the actuator 11 to switch to automatic driving when the driver's reaction is weak as a result of the above actuation.

In this way, by performing actuations such as asking the driver questions and calling attention to them, it is possible to improve the accuracy of the driver's abnormality determination and encourage the driver to take safe actions.

(Step ST31)
In step ST31, the abnormality detection unit 303 determines whether the degree of risk estimated in step ST13 is higher than a predetermined second standard. The specific content of the predetermined second criterion is not particularly limited as long as the predicted degree of risk can be quantitatively evaluated, but for example, the above-mentioned risk potential can be used. In the following explanation, the above-mentioned first driving scene (see FIG. 6) and second driving scene (see FIG. 9) have a risk level below a predetermined second standard, and the third driving scene (see FIG. 11) , the degree of risk is higher than a predetermined second standard. That is, when the vehicle moves from the first driving scene to the third driving scene and the other vehicle 161 enters within a predetermined range of the own vehicle, it is assumed that the degree of danger has become higher than the second predetermined standard. In step ST31, if it is determined that the degree of risk is higher than a predetermined second standard, that is, if the determination is YES, the flow proceeds to step ST32.

(Step ST32)
In step ST32, the abnormality detection unit 303 determines whether the tendency of the driver to view the high salience region is lower than a predetermined third criterion. Specifically, for example, similarly to step ST21, the tendency of the line of sight to move to a high saliency area is detected using the saliency index, and it is determined whether the saliency index is lower than a predetermined third standard. Here, the third criterion is a value lower than the first criterion. Further, the third criterion is set to a value lower than the probability (0.5) of exceeding the threshold at a random point, for example, to 0.3 seconds. Similar to step ST21, the determination may be made using an index other than the saliency index. In the example of FIG. 12, the other vehicle 161 is stopped in an area where the salience is relatively low. As mentioned above, in attention dysfunction, when the degree of danger exceeds a predetermined standard, the driver's viewpoint tends to become localized. For example, as shown in FIG. 12, an able-bodied person (marked with a circle) may check the road conditions and use mirrors to check for other parked vehicles 161, as well as check the rear or side of the vehicle in order to pass. I'm checking the direction. On the other hand, in the case of patients with attention dysfunction, although their attention is directed toward the other vehicle 161, their gaze is more focused on the other vehicle than necessary, and their surroundings are not sufficiently checked. . In a situation like the triangle mark in FIG. 12, the saliency index has a relatively low value, and as a result, the saliency index may have a value lower than the predetermined third standard. Here, it is assumed that in the situation indicated by the triangle mark in FIG. 12, the saliency index becomes lower than a predetermined third standard. If it is determined in step ST32 that the tendency of the driver to view the high salience region is lower than a predetermined third standard (for example, see AR2 in FIG. 21), that is, if the determination is YES, the flow proceeds to step ST33.

(Step ST33)
In step ST33, the abnormality detection unit 303 determines whether the amplitude ds of the driver's saccade is smaller than a predetermined fourth reference. By looking at the amplitude ds of the driver's saccades, it can be confirmed whether the driver's viewpoint is localized (eg, a place where danger is assumed). The predetermined fourth criterion can be set arbitrarily and is not particularly limited. For example, as the predetermined fourth criterion, a demonstration experiment using a driving simulator or the like may be conducted as the inventors of the present invention, and the boundary between a healthy person and an attention dysfunction patient may be set as the fourth criterion. If it is determined in step ST33 that the amplitude ds of the driver's saccade is smaller than the fourth predetermined standard, that is, if the determination is YES, the flow advances to step ST34.

(Step ST34)
In step ST34, the control unit 30 performs control corresponding to the abnormality of the driver, similarly to step S22. Note that the same control may be performed in step ST34 and step ST22, or different control may be performed in step ST34 and step ST22. Further, for example, the control unit 30 may perform control based on the results of both step ST21 and step ST33. For example, if either step ST21 or step ST33 is determined to be YES, actuation such as asking the driver or alerting the driver is performed, and if both step ST21 and step ST33 are determined to be YES, the actuation is automatically performed immediately. An operation such as switching to driving may be set.

As described above, according to the present embodiment, the abnormality of the driver can be detected as accurately as possible, and the vehicle can perform an operation corresponding to the abnormality of the driver. Specifically, for example, even if the driver is in an abnormal state due to an increased degree of danger and is looking at a dangerous location with relatively low salience, the state will be in an undetermined state. This can be reduced.

Note that in the above embodiment, the abnormality of the driver may be determined using the correlation between the degree of risk and the saliency of the gaze point. Specifically, when a driver is in an abnormal situation, their awareness of dangerous situations and places tends to be lower than when the driver is in a normal state, so the correlation between the degree of danger and the saliency of the gaze point tends to decrease. Therefore, by using the degree of correlation between the degree of risk and the saliency of the gaze point as a standard, it is possible to improve the accuracy of the driver's abnormality determination.

In the above embodiment, instead of or in addition to the processes from step ST31 to step ST33, in step ST21, if the degree of risk is lower than the predetermined second standard, the first standard is The standard may be lowered. The degree of risk in this case can also be scored using, for example, risk potential.

In the above embodiment, instead of or in addition to the processes from step ST31 to step ST33, in step ST21, when the degree of risk (risk potential) is increasing, the determination threshold of the saliency index is set to the second standard. The determination threshold value of the saliency index may be raised when the degree of risk (risk potential) is on a decreasing trend. As a result, even if, for example, the driver is in an abnormal state due to an increase in the degree of danger, he or she is looking at a dangerous location with relatively low salience, the situation will not be determined. can be reduced.

Furthermore, in the above embodiment, the first to fourth criteria (hereinafter simply referred to as predetermined criteria) are used to determine whether the criteria are higher or lower than these, but various methods can be used for this determination. can be applied. For example, a period exceeding the standard may be added to the determination factor, or an average value or a weighted average value for a predetermined period may be compared with a predetermined standard. Furthermore, regarding a predetermined standard, the concept of a period in which the vehicle exceeds or falls short of the standard may be added to the standard for driver abnormality determination. Furthermore, regarding a predetermined standard, the concept of the area of a region formed by the period and size of exceeding or falling below the standard may be added to the standard for abnormality determination by the driver.

The technology disclosed herein is useful as a driver condition determination device installed in a vehicle.

21a Front camera (external information acquisition means)
30 Control unit (vehicle control unit)
302 Gaze point detection unit

Claims (4)

A vehicle control device installed in a vehicle,
a gaze point detection unit that detects a gaze point of the driver of the vehicle;
A saliation map in which external environment information is received from an external information acquisition unit that acquires environmental information on the front side of the vehicle, and a driver's gaze point detected by the gaze point detection unit is generated based on the vehicle exterior environment information. A vehicle control unit that performs an operation in response to a driver's abnormality when an eye attraction degree, which is a degree of eye attraction that is a degree of eye attraction to a region with high salience that is relatively salient to the surroundings, is higher than a predetermined first standard. and
When the degree of risk estimated based on the vehicle external environment information is higher than a predetermined second standard and the saliency of the gaze point is equal to or lower than a third standard, which is lower than the first standard, the vehicle control unit A vehicle control device that performs an operation corresponding to an abnormality of the driver even when the amplitude of a saccade based on a gaze point is smaller than a fourth predetermined reference. The vehicle control device according to claim 1,
A vehicle control device characterized in that, when the degree of risk estimated based on the environment information outside the vehicle is higher than the second standard, the first standard is lowered in accordance with the increase in the degree of risk. The vehicle control device according to claim 1,
Equipped with an external environment recognition unit that recognizes the environment outside the vehicle to generate routes for autonomous driving.
The vehicle control device is characterized in that the vehicle control unit estimates the degree of risk based on the recognition result of the external environment recognition unit. A driver condition determination method for determining a driver condition executed by a vehicle control device, the method comprising:
a detection step of detecting a gaze point of a vehicle driver and a saccade amplitude based on the movement of the gaze point;
an extraction step of extracting a region with high salience that is relatively salient with respect to the surroundings in a salience map generated based on vehicle exterior environment information received from the vehicle exterior information acquisition means that acquires environment information on the front side of the vehicle; ,
an estimation step of estimating the degree of danger of the driving scene based on the vehicle exterior environment information received from the vehicle exterior information acquisition means;
When the driver's gaze point detected in the detection step has a degree of attraction, which is the degree to which the driver's gaze point is attracted to the region of high salience extracted in the extraction step, which is higher than a predetermined first standard, the driver is detected to be abnormal. a determination step for determining that there is a
In the determining step, the degree of risk estimated in the estimating step is higher than a predetermined second standard, and the degree of attraction, which is the degree to which the driver's gaze point is attracted to an area with high salience, is higher than the first standard. A method for determining a driver's condition, characterized in that the driver is determined to be abnormal even when the amplitude of a saccade based on the gaze point is smaller than a fourth predetermined criterion when the amplitude of the saccade based on the gaze point is smaller than a predetermined fourth criterion.

Images