JP7415459B2 - Vehicle control device and driver condition determination method - Google Patents

Description

The technology disclosed herein relates to, for example, a vehicle control device and a driver state determination method for determining the state of a driver who drives an automobile.

Recently, the development of autonomous driving systems has been promoted nationally. The applicant of this application currently believes that there are two main directions for automatic driving systems.

The first direction is a system in which the vehicle plays a central role in transporting passengers to their destination without requiring any operation by the driver, which is what is called fully automated vehicle driving. The second direction is an automated driving system that is based on the assumption that a human will be driving the car, such as ``providing an environment where driving can be enjoyed.''

In the second direction of automated driving systems, when a situation arises in which the driver becomes ill and is unable to drive normally, for example, the vehicle automatically replaces the passenger and performs automated driving. is assumed. Therefore, it is important to be able to detect abnormalities in the driver as early as possible and accurately, especially when the driver has developed a functional disorder or disease, to improve the survival rate of the driver and to ensure the safety of everyone, including those around them. This is extremely important from the perspective of ensuring that

As a method for determining abnormality of a driver, for example, Patent Document 1 describes a method in which a front image of a vehicle is mapped, a conspicuousness map (saliency map) is generated based on the pixel values of the mapped front image, and A technique has been disclosed that calculates the degree of conspicuousness of a driver's line of sight, and compares the degree to which the driver has turned his/her line of sight to a predetermined threshold with a predetermined threshold value to determine whether or not the driver is distracted.

In the technology of Patent Document 1, when the driver is distracted, the intentional attention function does not work, so for example, the driver's line of sight does not go to a dangerous place where a pedestrian may jump out of the blind spot. , the judgment is based on the gaze distribution characteristic that the gaze is directed to the prominent part of the visual stimulus.

Patent No. 5966640

The inventors conducted line-of-sight detection using a driving simulator using a driving simulator for people with impairments and healthy people, and found that there were many cases where even people with impairments were not attracted to areas with high salience, and even healthy people were often attracted to areas with high salience. I found out that it occurs.

For example, if an area with high salience is also a caution point that should be looked at while driving, even a healthy person may direct their line of sight to the area with high salience, and as a result, it may be erroneously determined that a disease has occurred.

The technology disclosed herein has been developed in view of these points, and is capable of determining the driver's condition as accurately as possible regardless of the driving environment of the vehicle in a driver condition determination device that determines the driver's condition based on the tendency to attract attention to saliency. The purpose is to be able to detect abnormalities of the driver.

In order to solve the above problem, the technology disclosed herein includes a gaze point detection unit that detects a gaze point of a driver of the vehicle, and an environment in front of the vehicle, for a vehicle control device installed in a vehicle. Receiving vehicle exterior environment information from vehicle exterior information acquisition means for acquiring information, and correlation between the spread of a high salience region based on the vehicle exterior environment information and the amplitude of the driver's saccade detected by the gaze point detection unit. If the driver's gaze point detected by the gaze point detection unit has a higher tendency to look at an area with high salience than a predetermined second criterion, The vehicle control unit is configured to include a vehicle control unit that performs operations in response to abnormalities.

Here, saliency is a visual feature amount consisting of saliency that changes from moment to moment due to color, brightness, movement, etc. The high saliency area refers to an area where saliency is high (high conspicuousness) in the driver's visual field, or in other words, an area that stands out from the surroundings. More specifically, a high saliency region refers to, for example, a region that has a large color difference or brightness difference compared to surrounding regions, or that has a large movement relative to the surrounding regions.

Furthermore, a saccade is a jumping eye movement in which the driver intentionally moves his/her line of sight, and is an eye movement in which the driver moves his/her line of sight from a point of gaze where the line of sight remains stagnant for a predetermined period of time to the next point of gaze. The amplitude of a saccade refers to the amount of movement when the driver's line of sight moves from a gaze point where it remains stagnant for a predetermined period of time to the next gaze point.

According to this configuration, by looking at the degree of correlation between the spread of the high saliency area and the amplitude of the driver's saccades, it is possible to extract the tendency for the driver's viewpoint to be drawn to the high saliency area. By combining this tendency for the driver's viewpoint to be drawn to the high saliency region and the tendency for the driver's gaze point to look at the high saliency region, the driver can more appropriately respond to abnormalities.

In one embodiment of the vehicle control device, the second reference may be changed depending on the extent of the high saliency region.

When the saliency region spreads widely, the number of high saliency regions increases, so it is assumed that even a healthy person will have a high probability of seeing a high saliency region. Therefore, by changing the second standard according to the spread of the high saliency region, it is possible to reduce erroneous abnormality determinations by the driver.

In order to solve the above problem, the technology disclosed herein targets a driver state determination method for determining the state of a driver of a vehicle, and includes a method for determining the state of the driver of a vehicle, and a movement of the point of view of the driver of the vehicle. a gaze point detection step of detecting a saccade amplitude based on the saccade amplitude; a map generation step of generating a saliency map based on the vehicle exterior environment information received from the vehicle exterior information acquisition means acquiring the vehicle front side environment information; and the map generation step. In the case where there is a correlation between the spread of the high saliency region of the saliency map generated in step and the magnitude of the saccade amplitude of the driver detected in the gaze point detection step, the saccade amplitude detected by the gaze point detection unit is and a determination step of determining that the driver has an abnormality when the tendency of the driver's gaze point to look at the high saliency region is higher than a predetermined reference value.

According to this method, by looking at the degree of correlation between the spread of the high saliency area and the amplitude of the driver's saccades, it is possible to extract the tendency for the driver's viewpoint to be drawn to the high saliency area. By combining this tendency for the driver's viewpoint to be attracted to the high saliency region and the tendency for the driver's gaze point to look at the high saliency region, it is possible to improve the accuracy of the driver's abnormality determination.

As described above, according to the technology disclosed herein, the driver can more appropriately respond to abnormalities.

1 is a schematic diagram showing a front portion of a vehicle interior of a vehicle equipped with a driver state estimation device according to an exemplary embodiment; FIG. FIG. 2 is a front view of the vehicle seen from the front. FIG. 1 is a block diagram illustrating the configuration of a vehicle control system according to an exemplary embodiment. FIG. 2 is a block diagram illustrating the configuration of a driver state detection section. FIG. 3 is a diagram showing an example of an external environment image on the front side of the vehicle taken by a front camera. 6 is a diagram showing an example of a composite image based on the external environment image of FIG. 5. FIG. 7 is a diagram showing an example of a saliency map of the composite image of FIG. 6. FIG. 6 is a graph showing an example of eye movement by a healthy person and an attention dysfunction patient in the example of FIG. 5. FIG. FIG. 7 is a diagram showing another example of a composite image based on an external environment image on the front side of the vehicle taken by a front camera. 10 is a diagram showing an example of a saliency map of the composite image of FIG. 9. FIG. 11 is a graph showing an example of gaze movement by a healthy person and an attention dysfunction patient in the example of FIG. 10. It is a graph for explaining extraction of saccades. It is a graph for explaining saccade noise removal processing. FIG. 2 is a conceptual diagram schematically showing the relationship between the spread of the driver's salience and the saccade amplitude. FIG. 3 is a diagram showing an example of a change in saliency over time of a driver's gaze point. FIG. 16 is a diagram showing an example of changes in saliency over time when coordinates are specified randomly in the same environment as FIG. 15; It is a graph comparing the proportion exceeding the threshold value at random points and the proportion exceeding the threshold value at the driver's gaze point. FIG. 2 is a diagram for explaining a saliency index. 3 is a flowchart illustrating the operation of the vehicle control device. FIG. 2 is a diagram showing an example of the time distribution of the salience index of healthy individuals. FIG. 3 is a diagram illustrating an example of the temporal distribution of the salience index of an attention function patient.

<Findings obtained by the inventors>
As disclosed in Patent Document 1, there is a known technique for detecting an abnormality in a driver (an abnormality that causes a decline in attention function) by observing changes in the driver's line of sight movement with respect to saliency. In this technique, for example, an abnormality of the driver is detected when the line of sight movement toward an area with high salience (high saliency area) exceeds a predetermined threshold value.

As a result of intensive research, the inventors of the present application have simulated the behavior of vehicle drivers during abnormal situations (abnormal situations that cause a decline in attention function) by observing the behavior of people with attention dysfunction when driving a vehicle. I found out that it can be done. Based on the above observation results, the inventors of the present application verified a method of detecting a driver's abnormality when the line of sight movement toward a high saliency region exceeds a predetermined threshold value. Specifically, we examined the behavior of people with attention dysfunction when driving a vehicle and the behavior of healthy people without attention dysfunction when driving a vehicle, using the salience index described later to move the line of sight to a high salience region. We verified a method that detects the tendency of the driver and detects driver abnormality when the salience index exceeds a predetermined threshold.

As a result of this verification, the inventors of the present application found that there are many cases in which even people with functional disabilities are not attracted to areas with high salience, and cases in which even healthy people are attracted to areas with high salience. For example, if an area with high salience is also a cautionary point to look at while driving, even healthy people may direct their eyes to the area with high salience, and as a result, it may be erroneously determined that a disease has occurred. I understand.

20 and 21 show the driver's line of sight detected using a driving simulator and plotted as a time change in the saliency index. FIG. 20 shows the measurement results of a healthy person, and FIG. 21 shows the measurement results of a functionally impaired patient. However, since the running speeds are different from each other, the time axis and the running location do not necessarily match.

Although the details will be described later, the saliency index is an index whose numerical value increases as the degree of attraction to a high saliency region increases. In the examples shown in Figures 20 and 21, the threshold is set to 0.6, which indicates that the degree of attraction to the high salience region is relatively high, and when the salience index exceeds 0.6, it is determined that there is a failure in attention function. It was assumed that the

As a result, it was found that even in healthy subjects, there were some cases where the salience index exceeded 0.6 (see AR1 in FIG. 20). For example, when passing a vehicle with high salience such as a fire truck, even healthy people tend to be attracted to places with high salience. In addition, even in patients with functional disorders, the saliency index may be confirmed in high-risk locations such as intersections, and areas with a low saliency index have been confirmed (see AR2 in FIG. 21).

Therefore, the inventors of the present application further conducted extensive studies and found that it is possible to detect abnormalities in the driver by looking at the spatial spread of saliency in addition to the saliency index.

Specifically, the inventors used a salience index to detect whether the driver's gaze point tends to move toward a high salience region, and also to detect the spatial spread of salience and the spread of the driver's line of sight distribution. Detect the degree of correlation between Then, when the driver's gaze point tends to move toward a high salience region, and when the degree of correlation between the spatial spread of saliency and the spread of the gaze distribution is high, it can be determined that there is an abnormality in the driver. We found that the accuracy of driver abnormality detection can be improved.

In the following, exemplary embodiments will be specifically described with reference to the drawings. In the following description, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. Also, when looking from the rear to the front, the left side is called the left side, and the opposite is called the right side.

FIG. 1 schematically shows the interior of an automobile as a vehicle. This vehicle is a right-hand drive vehicle, and a steering wheel 58 is disposed on the right side.

In the vehicle interior, a front window glass 51 is arranged on the front side of the vehicle when viewed from the driver's seat. The front window glass 51 is partitioned by a plurality of vehicle structural members when viewed from the inside of the vehicle. Specifically, the front window glass 51 is divided by left and right front pillar trims 52, a roof trim 53, and an instrument panel 54.

The left and right front pillar trims 52 constitute outer boundaries of the front window glass 51 in the vehicle width direction, respectively. Each front pillar trim 52 is arranged along each front pillar. The roof trim 53 constitutes the upper boundary of the front window glass 51. The roof trim 53 covers the interior side of the roof panel of the vehicle. A rearview mirror 55 is attached to a portion at the center of the front window glass 51 in the vehicle width direction and slightly below the roof trim 53. An in-vehicle camera 28 (see FIG. 3) for photographing the interior of the vehicle, particularly the driver's face, is provided in a portion of the roof trim 53 near the rearview mirror 55. The in-vehicle camera 28 will be explained in detail later. The instrument panel 54 constitutes the lower boundary of the front window glass 51. The instrument panel 54 is provided with a meter box and a display 57.

Further, the vehicle has side mirrors 56 on the outer sides of the left and right front pillars in the vehicle width direction. Each side mirror 56 is arranged so that the driver seated in the driver's seat can see through the window of the side door.

As shown in FIG. 2, the vehicle is provided with a camera (hereinafter referred to as front camera 21a) for photographing the external environment on the front side of the vehicle. The front camera 21a is located at the front end of the vehicle, slightly below the hood 59 of the vehicle. The front camera 21a is an example of outside-vehicle information acquisition means that acquires environmental information on the front side of the vehicle.

<Vehicle control system>
FIG. 3 illustrates the configuration of the vehicle control system 10 according to the embodiment. The vehicle control system 10 is provided in a vehicle (specifically, a four-wheeled motor vehicle). The vehicle can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in response to the driver's operations (for example, accelerator operation). Assisted driving is driving in which the vehicle supports the driver's operations. Autonomous driving is driving without driver input. The vehicle control system 10 controls the vehicle in assisted driving and automatic driving. Specifically, the vehicle control system 10 controls the operation (especially driving) of the vehicle by controlling an actuator 11 provided in the vehicle.

The vehicle control system 10 includes an information acquisition section 20, a control section 30, and a notification section 40. In the following description, the vehicle provided with the vehicle control system 10 will be referred to as the "host vehicle", and other vehicles existing around the host vehicle will be referred to as "other vehicles".

-Actuator-
The actuator 11 includes a drive system actuator, a steering system actuator, a brake system actuator, and the like. Examples of drive system actuators include engines, motors, and transmissions. An example of a steering system actuator is a steering wheel. An example of a brake system actuator is a brake.

-Information acquisition department-
The information acquisition unit 20 acquires various information used for vehicle control (particularly driving control). In this example, the information acquisition section 20 includes a plurality of cameras 21, a plurality of radars 22, a position sensor 23, an external input section 24, a vehicle condition sensor 25, a driving operation sensor 26, and a driver condition sensor 27. including.

〔camera〕
The plurality of cameras 21 have mutually similar configurations. A plurality of cameras 21 are provided on the vehicle so as to surround the vehicle. Each of the plurality of cameras 21 acquires image data representing a part of the external environment of the vehicle by capturing an image of a part of the environment surrounding the vehicle (external environment of the vehicle). Image data obtained by each of the plurality of cameras 21 is transmitted to the control unit 30. The plurality of cameras 21 include the above-mentioned front camera 21a.

In this example, camera 21 is a monocular camera with a wide-angle lens. For example, the camera 21 is configured using a solid-state imaging device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor). Note that the camera 21 may be a monocular camera with a narrow-angle lens, or may be a stereo camera with a wide-angle lens or a narrow-angle lens.

[Radar]
The plurality of radars 22 have mutually similar configurations. The plurality of radars 22 are provided in the vehicle so as to surround the vehicle. Each of the plurality of radars 22 detects a part of the external environment of the vehicle. Specifically, the radar 22 detects a part of the external environment of the vehicle by transmitting radio waves toward a part of the external environment of the vehicle and receiving reflected waves from the part of the external environment of the vehicle. do. The detection results of the plurality of radars 22 are transmitted to the control unit 30.

For example, the radar 22 may be a millimeter wave radar that transmits millimeter waves, a lidar (Light Detection and Ranging) that transmits laser light, or an infrared radar that transmits infrared light. Alternatively, it may be an ultrasonic sensor that transmits ultrasonic waves.

[Position sensor]
The position sensor 23 detects the position (for example, latitude and longitude) of the vehicle. For example, the position sensor 23 receives GPS information from the global positioning system and detects the position of the vehicle based on the GPS information. Information (vehicle position) obtained by the position sensor 23 is transmitted to the control unit 30.

[External input section]
The external input unit 24 inputs information through an external network (for example, the Internet) provided outside the vehicle. For example, the external input unit 24 receives communication information from other vehicles (not shown) located around the vehicle, car navigation data from a navigation system (not shown), traffic information, high-precision map information, and the like. Information obtained by the external input section 24 is transmitted to the control section 30.

[Vehicle condition sensor]
The vehicle condition sensor 25 detects the condition of the vehicle (for example, speed, acceleration, yaw rate, etc.). For example, the vehicle condition sensor 25 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, and the like. Information (vehicle status) obtained by the vehicle status sensor 25 is transmitted to the control unit 30.

[Driving operation sensor]
The driving operation sensor 26 detects a driving operation applied to the vehicle. For example, the driving operation sensor 26 includes an accelerator opening sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the amount of operation of the accelerator of the vehicle. The steering angle sensor detects the steering angle of the steering wheel of the vehicle. The brake oil pressure sensor detects the amount of brake operation of the vehicle. Information (vehicle driving operation) obtained by the driving operation sensor 26 is transmitted to the control unit 30.

[Driver status sensor]
The driver condition sensor 27 detects the condition of the driver who drives the vehicle (for example, the driver's health condition, emotions, physical behavior, etc.). The information (driver's condition) obtained by the driver condition sensor 27 is transmitted to the control unit 30. In this example, driver condition sensor 27 includes an in-vehicle camera 28 and a biological information sensor 29.

《In-vehicle camera》
The in-vehicle camera 28 is provided inside the vehicle. The in-vehicle camera 28 acquires image data including the driver's eyes by capturing an image of a region including the driver's eyes. Image data obtained by the in-vehicle camera 28 is transmitted to the control unit 30. For example, the in-vehicle camera 28 is placed in front of the driver, and the imaging range is set so that the driver's eyeballs are within the imaging range. Note that the in-vehicle camera 28 may be provided in goggles (not shown) worn by the driver.

《Biological information sensor》
The biological information sensor 29 detects biological information (for example, sweating) of the driver. Information obtained by the biometric information sensor 29 (driver's biometric information) is transmitted to the control unit 30 .

-Control unit-
The control unit 30 determines a target route, which is the route the vehicle should travel, based on various information acquired by the information acquisition unit 20 during assisted driving or automatic driving, and is necessary for driving the target route. Determine the target motion, which is the motion of the vehicle. The control unit 30 then controls the operation of the actuator 11 so that the motion of the vehicle matches the target motion. For example, the control section 30 is configured by an electronic control unit (ECU) having one or more arithmetic chips. In other words, the control section 30 is configured by an electronic control unit (ECU) having one or more processors, one or more memories that store programs and data for operating the one or more processors, and the like. Ru.

In this example, as shown in FIG. 3, the control unit 30 includes an image processing unit 31, an external environment recognition unit 32, a candidate route generation unit 33, a vehicle behavior recognition unit 34, and a driver behavior recognition unit 35. , a target route determining section 36, and a motion control section 37.

[Image processing section]
The image processing unit 31 receives images captured by the plurality of cameras 21 and performs image processing. The image processing performed by the image processing unit 31 includes first image processing for images used by the external environment recognition unit 32 to recognize the external environment such as objects, and second image processing for use in generating a saliency map. Image processing is included. Note that the first image processing is image processing that has been conventionally used for automatic driving and the like, and a detailed explanation thereof will be omitted here.

《About the second image processing》
For example, the image processing unit 31 deletes, from the image captured by the front camera 21a, pixels that are unnecessary for processing by the map generation unit 301 (for example, generation of a saliency map), which will be described later, from among the elements constituting the image. Perform processing. Further, the image processing unit 31 performs processing to generate a composite image by combining another image with the image showing the external environment on the front side of the vehicle taken by the front camera 21a.

FIG. 5 is an example of an image D11 showing the external environment on the front side of the vehicle, taken by the front camera 21a in a vehicle driving scene. The external environment shown in this image D11 includes a roadway 150 and a white line 151 on the roadway 150. In addition, the external environment shown in this image D11 includes a wall 162 formed on the left side of the roadway 150, a forest 163 formed on the left side of the wall 162, and a hill spreading on the right side of the roadway 150. 164 and a forest 165 formed on the hill 164. Furthermore, the external environment shown in this image D11 includes the sky 167 that extends above the road 150 and forests 163 and 165. In the following description, it is assumed that the sky 167 of the image D11 is a sunset sky, and a reddish sky is spreading. In other words, the image D11 is assumed to be an image in which the spread RW of the high saliency region is relatively large, as shown in FIG. Note that FIG. 7 will be explained in detail later.

The image processing unit 31 synthesizes, with the image D11, an image showing vehicle components that enter the driver's field of view when the vehicle is running. Specifically, as shown in FIG. 6, the image processing unit 31 synthesizes an image of vehicle components (hereinafter referred to as a vehicle image) when a driver seated in the driver's seat looks at the front side of the vehicle into an image D11. Then, a composite image D12 is generated. The vehicle components photographed in the vehicle image include, for example, the front pillar trim 52 on the right side (driver's seat side), the right part of the roof trim 53, the right part of the instrument panel 54, the rearview mirror 55, and the right side. side mirrors 56 and a steering wheel 58. The vehicle image can be, for example, taken in advance by photographing the interior structure of the vehicle that enters the driver's field of view from the driver's seat side, and storing the photographed image data as a layer in the memory (not shown) of the control unit 30. . When combining the vehicle image with the image D11, the image processing unit 31 may read the vehicle image from the memory into a separate layer from the image D11, align the image D11 and the vehicle image, and overlap them. Note that a part of the hood 59 may also be considered as a vehicle component. Further, in creating the composite image D12, information other than the front camera 21a, for example, the detection results of the radar 22 or input information from the external input unit 24 may be used.

FIG. 9 is an example of a composite image D22 obtained by combining a vehicle image with an image of the external environment in front of the vehicle taken by the front camera 21a in a driving scene different from that shown in FIGS. 5 to 8. The composite image D22 is generated by combining the captured image of the front camera 21a and the vehicle image, as in the case of the composite image D12. The external environment shown in the composite image D22 includes a roadway 150 and a white line 151 on the roadway 150. Further, the external environment shown in the composite image D22 includes other vehicles 161 and walls 162 formed on both sides of the roadway 150. A tree 168 is planted in an area to the left of the left wall 162, and a tree 169 is planted on top of a hill 164 that extends to the left of the left wall 162. Similarly, a tree 168 is planted on top of a hill 164 that extends to the left side of the right wall 162. Further, the external environment shown in the image D22 includes the road 150, the hill 164, and the sky 170 extending above the trees 168, 169. Note that in the following description, it is assumed that the sky 170 of the composite image D22 is a blue sky. In other words, it is assumed that the image D22 is an image in which the spread RW of the high saliency region is relatively small, as shown in FIG. Note that FIG. 10 will be explained in detail later.

As shown in FIGS. 6 and 9, the composite images D12 and D22 generated by the image processing unit 31 are in a state in which a portion of the images is blocked by a vehicle component. Image data of composite images D12 and D22 (hereinafter referred to as composite image data) generated by the image processing unit 31 is sent to a map generation unit 301, which will be described later.

[External environment recognition department]
The external environment recognition unit 32 recognizes the external environment of the vehicle based on data output from the plurality of cameras 21 and radar 22. The external environment of the vehicle recognized by the external environment recognition unit 32 includes objects. Examples of objects include a moving object that displaces over time and a stationary object that does not displace over time. Examples of moving objects include four-wheeled motor vehicles, motorcycles, bicycles, and pedestrians. Examples of stationary objects include signs, street trees, median strips, center poles, and buildings.

[Candidate route generation unit]
The candidate route generation unit 33 generates one or more candidate routes based on the output of the external environment recognition unit 32. A candidate route is a route on which a vehicle can travel, and is a candidate for a target route.

[Vehicle behavior recognition unit]
The vehicle behavior recognition unit 34 recognizes vehicle behavior (eg, speed, acceleration, yaw rate, etc.) based on the output of the vehicle state sensor 25. For example, the vehicle behavior recognition unit 34 recognizes the behavior of the vehicle from the output of the vehicle condition sensor 25 using a learning model generated by deep learning.

[Driver behavior recognition unit]
The driver behavior recognition unit 35 recognizes the driver's behavior (for example, the driver's health condition, emotions, physical behavior, etc.) based on the output of the driver condition sensor 27. For example, the driver behavior recognition unit 35 recognizes the driver's behavior from the output of the driver condition sensor 27 using a learning model generated by deep learning. In this example, the driver behavior recognition section 35 includes a driver state detection section 300. The driver state detection section 300 will be explained in detail later.

[Target route determination unit]
The target route determination unit 36 selects a target route from among the one or more candidate routes generated by the candidate route generation unit 33 based on the output of the vehicle behavior recognition unit 34 and the output of the driver behavior recognition unit 35. Select a candidate route. For example, the target route determining unit 36 selects the candidate route that the driver feels is most comfortable from among the plurality of candidate routes.

[Motor control unit]
The motion control unit 37 determines a target motion based on the candidate route selected as the target route by the target route determination unit 36, and controls the actuator 11 based on the determined target motion. For example, the motion control unit 37 derives a target driving force, a target braking force, and a target steering amount, which are a driving force, a braking force, and a steering amount for achieving the target movement, respectively. Then, the motion control unit 37 transmits the drive command value indicating the target driving force, the braking command value indicating the target braking force, and the steering command value indicating the target steering amount to the actuators of the drive system, the actuators of the braking system, and the steering system. and the actuator respectively. A drive system actuator, a brake system actuator, and a steering system actuator are examples of the actuator 11.

《Notification Department》
The notification unit 40 notifies the driver of the vehicle of various information. In this example, the notification section 40 includes a display section 41 and a speaker 42. The display unit 41 outputs various information in the form of images. The speaker 42 outputs various information in the form of audio.

《Driver status detection section》
The driver condition detection unit 300 detects abnormality of the driver of the vehicle. Specifically, as shown in FIG. 4, the driver state detection section 300 includes a map generation section 301, a gaze point detection section 302, and an abnormality detection section 303.

Note that the driver's abnormality refers to an abnormality that causes a decline in the driver's attention function. Examples of such driver abnormalities include brain diseases such as stroke, heart diseases such as myocardial infarction, epilepsy, hypoglycemia, and drowsiness.

[Map generator]
The map generation unit 301 receives composite images D12 and D22 (hereinafter collectively referred to as composite image D2) from the image processing unit 31, and generates a saliency map D3 for the composite images D12 and D22. Specifically, the map generation unit 301 calculates the saliency of a portion of the composite image data representing the external environment, that is, a portion other than the vehicle image synthesized by the image processing unit 31. At this time, the map generation unit 301 does not calculate the saliency of the vehicle images used in the synthesis by the image processing unit 31, but uses it to calculate the saliency of the portion of the composite image D2 that represents the external environment. That is, the map generation unit 301 generates the saliency map D3 in consideration of the vehicle components that enter the driver's field of view when the vehicle is running.

As mentioned above, saliency changes depending on the color, brightness, movement, etc. of the target object. Therefore, in the present embodiment, the map generation unit 301 calculates saliency for each feature, such as saliency based on color, saliency based on brightness, saliency based on motion, etc., and generates a saliency map for each feature. By adding them together, a final saliency map D13 based on the composite image D12 (see FIG. 7) and a saliency map D23 based on the composite image D22 (see FIG. 10) are generated. The saliency map D13 and the saliency map D23 are examples of the saliency map D3.

For example, with respect to color-based saliency, when the color difference between the nearby region and the vehicle component is large in a region near the vehicle component in the composite image data, the map generation unit 301 generates a color-based saliency, compared to when the color difference is small. Increase the salience of the neighborhood. Note that the color difference is calculated by the following formula when the RGB of a certain pixel color is (R1, G1, B1) and the RGB of another pixel is (R2, G2, B2).
(Color difference) = {(R2-R1) ² + (G2-G1) ² + (B2-B1) ² } ^1/2
The map generation unit 301 may calculate so that the saliency increases continuously as the color difference increases, or it may calculate such that the saliency increases by a certain value each time the threshold value is exceeded by providing a plurality of threshold values. .

Further, for example, regarding the saliency based on brightness, the map generation unit 301 can determine whether the difference in brightness between the nearby region and the vehicle component is large in the vicinity of the vehicle component in the composite image data, and when the brightness difference is small. In comparison, the salience of the neighboring region is increased. For example, when the synthesized vehicle component is black, the map generation unit 301 increases the saliency of the part closer to white in the nearby region because the brightness difference becomes larger as the part becomes closer to white. The map generation unit 301 may calculate so that the saliency increases continuously as the luminance difference becomes larger, or it may calculate such that the saliency increases by a certain value each time the threshold value is exceeded by providing a plurality of threshold values. good.

Note that the calculation of saliency itself can be performed using a known computer program published on the Internet or the like. Furthermore, a known computer program can be used to calculate a saliency map for each feature and to integrate each saliency map.

[Gaze point detection unit]
The gaze point detection unit 302 calculates the driver's line of sight direction from the driver's eyeball image captured by the in-vehicle camera 28. For example, the gaze point detection unit 302 calculates the direction of the driver's line of sight by detecting changes in the driver's pupils from a state in which the driver looks through the lens of the in-vehicle camera 28 as a reference. The line-of-sight direction may be calculated from either the driver's left eye or right eye, or may be the average value of the line-of-sight directions (line-of-sight vector) determined from each of the driver's eyes. Furthermore, when it is difficult to calculate the driver's line-of-sight direction from changes in the driver's pupils, the line-of-sight direction may be calculated by further considering the direction of the driver's face. Further, a learning model (a learning model for detecting the line of sight) generated by deep learning may be used to calculate the direction of the driver's line of sight.

《Detection of fixation points and saccades》
The gaze point detection unit 302 detects the driver's saccades based on the movement of the driver's line of sight. A saccade is a jumping eye movement in which the driver intentionally moves his/her line of sight, and is an eye movement in which the driver moves his/her line of sight from a point of gaze where the line of sight remains stagnant for a predetermined period of time to the next point of gaze. As shown in FIG. 12, a period sandwiched between two adjacent gaze periods is a saccade period. Note that the gaze period is a period in which the line of sight is considered to be stationary. The saccade amplitude ds is the distance the line of sight moves during the saccade period.

The gaze point detection unit 302 calculates the moving speed of the line of sight based on the change in the moving distance of the line of sight, and determines a state in which the moving speed of the line of sight is less than a predetermined speed threshold (for example, 2 deg/s). A period of stagnation time (for example, 0.1 second) is extracted as a "gazing period", and a point ahead of the driver's line of sight during the gaze period is extracted as a "gazing point". In addition, the gaze point detection unit 302 detects that the movement speed of the gaze movement during the period sandwiched between two adjacent gaze periods is equal to or higher than a speed threshold (for example, 2 deg/s), and the movement distance is a predetermined distance. A line-of-sight movement that is equal to or greater than a distance threshold (for example, 0.1 deg) is extracted as a "saccade."

Note that the gaze point detection unit 302 may perform noise removal processing on the extracted saccade. Specifically, the gaze point detection unit 302 derives a regression curve L10 based on a plurality of saccade candidates, as shown in FIG. 13. The gaze point detection unit 302 derives the regression curve L10 from a plurality of saccade candidates using, for example, the least squares method. Next, the gaze point detection unit 302 derives the first reference curve L11 by shifting the regression curve L10 in the direction in which the moving speed increases (increasing direction on the vertical axis in FIG. 13), and shifts the regression curve L10 to the moving speed is shifted in the decreasing direction (decreasing direction on the vertical axis in FIG. 13) to derive the second reference curve L12, and the saccade range R10 is defined between the first reference curve L11 and the second reference curve L12. Then, the gaze point detection unit 302 extracts a saccade candidate included within the saccade range R10 from among the plurality of saccade candidates as a saccade.

Next, the gaze point detection unit 302 calculates a saccade amplitude ds and a saccade frequency fs, which are indicators of a saccade. Specifically, the gaze point detection unit 302 calculates the average value of the saccade amplitudes ds included in each predetermined period (for example, every 10 seconds) as the "saccade amplitude ds", The value obtained by dividing the number of saccades included in the cycle by the time of the cycle is calculated as the "saccade frequency fs".

[Abnormality detection part]
The abnormality detection unit 303 detects an abnormality of the driver based on the saliency map D3 generated by the map generation unit 301 and the driver's gaze point and saccade amplitude ds detected by the gaze point detection unit 302. do. The operation of the abnormality detection unit 303 will be specifically explained in "Operation of Vehicle Control System" below.

-Vehicle control system operation-
The operation of the vehicle control system will be described below with reference to FIG. 19.

(Step ST11)
First, in step ST11, the vehicle control system executes a gaze point detection step of detecting the gaze point of the vehicle driver. Specifically, in step ST11, the gaze point detection unit 302 detects the gaze point based on the driver's line of sight detected by the in-vehicle camera 28, for example, as explained in the section “Detection of gaze point and saccade” above. The viewpoint and the saccade amplitude ds are detected.

(Step ST12)
In the next step ST12, the vehicle control system executes a map generation step of generating a saliency map based on the image data received from the front camera 21a. Specifically, in step ST12, the map generation unit 301 generates the saliency map D3 based on the synthesized image D2 synthesized by the image processing unit 31, for example, as explained in the above-mentioned “Map generation unit” section. generate. Note that the order of execution of step ST11 and step ST12 is not particularly limited. For example, step S12 may be executed before step S11, or step ST11 and step ST12 may be executed simultaneously.

(Step ST13)
In step ST13, it is determined whether there is a correlation between the spread of the high saliency region of the saliency map generated in step ST12 and the amplitude ds of the driver's saccade detected in step ST11.

Specifically, for example, the abnormality detection unit 303 determines the extent RW of the high saliency region of the saliency map generated by the map generation unit 301 and the amplitude ds of the driver's saccade detected by the gaze point detection unit. Determine whether there is a correlation between More specifically, the abnormality detection unit 303 prepares, for example, a plurality of saliency maps D3 in which the spread RW of the high saliency region is different from each other. The abnormality detection unit 303 then detects the spread of the high saliency region of each saliency map D3 and the amplitude of saccades in a predetermined time range (for example, 5 seconds) in which the image on which the saliency map D3 is based is captured. The presence or absence of a mutual correlation is determined by comparing with ds.

FIG. 14 is a conceptual diagram showing the relationship between "extension RW of high salience region" and "saccade amplitude ds" for healthy subjects and attentionally impaired subjects. In FIG. 14, the thick solid line schematically shows the characteristics of a healthy person, and the thin solid line schematically shows the characteristics of a person with attentional dysfunction.

Here, if the driver is in a normal state (for example, a healthy person), the driver drives while looking at visible objects to be watched, such as parked vehicles, pedestrians, and side streets. In other words, a healthy person drives while looking at the visual object to be focused on, regardless of the extent of salience. Therefore, as shown by the thick solid line in FIG. 14, the degree of correlation between the extent of the high saliency region RW and the saccade amplitude ds tends to be low. On the other hand, when the driver becomes unable to drive actively due to an abnormality in his or her physical condition, the driver naturally turns his/her eyes toward an area where salience is high. In other words, when the driver frequently looks at areas with high eye-attractiveness, there is a high possibility that something is wrong with the driver. Then, when the driver has an abnormality and the spread RW of the high saliency region is large, the amount of movement of the driver's line of sight increases, and the amplitude ds of the saccade tends to increase. On the other hand, when the driver has an abnormality and the spread RW of the high salience region is small, the amount of movement of the driver's line of sight decreases, and the amplitude ds of the saccade tends to decrease. That is, when the driver has an abnormality, as shown by the thin solid line in FIG. 14, there is a tendency for the degree of correlation between the size of the spread RW of the high salience region and the size of the saccade amplitude ds to become high. . Therefore, in step ST11, the degree of correlation between the spread RW of the high saliency region of the saliency map generated by the map generation unit 301 and the amplitude ds of the driver's saccade detected by the gaze point detection unit is determined to be a predetermined number. If it is higher than the 1 standard, proceed to the next step ST14. On the other hand, in step ST13, if the degree of correlation between the spread RW of the high saliency region and the amplitude ds of the saccade is less than or equal to the first criterion, the process returns to step ST11. Here, the predetermined first criterion can be arbitrarily set as long as it is a criterion that can determine the degree of correlation. For example, as shown by the broken line in Figure 14, a reference line is drawn that indicates a high degree of correlation, and the correlation is calculated based on the relationship with the reference line (for example, the degree of approximation of the slope or the degree of coincidence). The degree may also be determined. Step ST13 is, for example, an example of processing that constitutes a part of the determination step.

(Step ST14)
In step ST14, it is determined whether the tendency of the driver's gaze point detected in step ST11 to look at the high saliency region is higher than a predetermined reference value. Specifically, for example, a tendency of eye movement toward a high saliency area is detected using a saliency index, and it is determined whether the saliency index exceeds a predetermined threshold value.

Specifically, the abnormality detection unit 303 determines whether the driver is in an area with high salience based on the saliency map generated by the map generation unit 301 and the driver's gaze point detected by the gaze point detection unit 302. It is determined whether the tendency to view is higher than a predetermined second criterion. In this embodiment, it is determined whether the saliency index is higher than a predetermined second standard (for example, 0.6).

Note that in the determination in step ST14, a value other than 0.6 may be used as the second standard of the saliency index, or an index other than the saliency index may be used.

《Calculation of salience index》
A method for calculating the saliency index will be explained using FIGS. 15 to 18. Calculation of this saliency index is mainly performed by the abnormality detection unit 303.

FIG. 15 is a plot of the change in saliency of the driver's gaze point over time. For example, as shown in FIGS. 8 and 11, this graph is a graph in which changes in the gaze point are applied to a saliency map, and the height of the saliency of the gaze point is plotted every time a saccade occurs. On the other hand, in FIG. 16, coordinates (hereinafter referred to as random coordinates) are randomly specified from the same saliency map as in FIG. 15 (e.g., FIG. 8 and FIG. 11), and the saliency of the random coordinates is calculated every time a saccade occurs. It is generated by

After creating the graphs shown in FIGS. 15 and 16, ROC (Receiver Operating Characteristic) curves of the proportion exceeding the threshold value at random points and the proportion exceeding the threshold value at the driver's gaze point are determined. Specifically, first, as a first step, the threshold value is set to a value larger than the maximum value in the graphs of FIGS. 15 and 16. Next, as a second step, while lowering the threshold value, the percentage of points exceeding the threshold value is determined for each threshold value. The process of this second step is then repeated until the threshold value becomes equal to or less than the minimum value in the graphs of FIGS. 15 and 16.

After the first and second steps, as a third step, the horizontal axis is the proportion exceeding the threshold at random points (referred to as the first probability), and the ratio exceeding the threshold at the driver's gaze point (referred to as the second probability) is plotted on the horizontal axis. A graph like the one shown in FIG. 17 is created for the vertical axis. The graph in FIG. 17 represents the gaze probability versus random probability at the same threshold. In the graph of FIG. 17, both the horizontal and vertical axes are probabilities, so the minimum value of the curve is 0 and the maximum value is 1.

In FIG. 17, the dashed line represents how the driver's line of sight moves with respect to saliency. When the curve is upwardly convex, like the curve C1 in FIG. 17, it indicates that the rate at which the driver's gaze point exceeds the threshold is higher than at a random point. When a curve with a shape like curve C1 in FIG. 17 is calculated, it means that the driver's line of sight tends to look at a high salience area, that is, the high salience area is highly attractive. There is. On the other hand, when the curve is convex downward like curve C2 in FIG. 17, it indicates that the rate at which the driver's gaze point exceeds the threshold is lower than that at a random point. When a curve shaped like curve C2 in FIG. 17 is calculated, it means that the driver's line of sight is not affected by the high salience region.

In this embodiment, the fourth step is to obtain AUC (Area Under Curve). AUC is the integral value of the lower right portion of this curve. That is, AUC is the area of the region surrounded by the curve C1 (or curve C2) and the two-dot chain line in FIG. 17, as indicated by diagonal lines in FIG. In this embodiment, this integral value is called a saliency index. By using the saliency index, dependence on driving scenes can be reduced. Specifically, for example, simply calculating the percentage of high-saliency areas that the driver views will show that when the entire driving scene includes high-saliency areas, the driver views the high-saliency areas more frequently. There may be consequences. On the other hand, by using a saliency index that is compared with random points as in this embodiment, it is possible to reduce the dependence on the driving scene, such as the number and spread of high saliency areas, and to Be able to make highly accurate judgments.

In step ST14, if the saliency index is higher than the second standard, the process proceeds to the next step ST15. On the other hand, if the saliency index is equal to or less than the second standard in step ST14, the process returns to step ST11. Step ST14 is, for example, an example of processing that constitutes a part of the determination step. Note that steps ST11 and ST12 are continuously executed while the vehicle is running, regardless of the processing of steps ST13 and ST14.

(Step ST15)
In step S15, the control unit 30 performs control corresponding to the abnormality of the driver. For example, when the degree of abnormality of the driver is high (for example, when it is recognized that the driver has developed a disease), the control unit 30 controls the actuator 11 via the driving motion control unit 37 etc. to enable automatic driving. Perform the switching process. The control unit 30 also performs actuations directed at the driver, such as asking the driver, “Are you okay?” and “Would you like to take a break?” via the notification unit 40. You may also check the reaction. For example, the control unit 30 may perform a process of controlling the actuator 11 to switch to automatic driving when the driver's reaction is weak as a result of the above actuation.

In this way, by performing actuations such as asking the driver questions and calling attention to them, it is possible to improve the accuracy of the driver's abnormality determination and encourage the driver to take safe actions.

As described above, according to this embodiment, abnormality of the driver can be detected as accurately as possible. Specifically, for example, as shown in FIG. 19, when looking only at the saliency index, even in a situation where a healthy person is incorrectly judged as abnormal, as shown by the thick solid line in FIG. If the correlation between the spread RW of the high saliency region and the saccade amplitude ds is low, it is not determined that the driver is abnormal. As a result, it is possible to more accurately extract the tendency for the line of sight to be induced by salience, and as a result, it is possible to accurately determine the driver's abnormality.

Note that in the above embodiment, the second criterion may be changed in step ST14 of FIG. 19 according to the spread RW of the high saliency region. When the saliency region spreads widely, the number of high saliency regions also increases, so it is assumed that even a healthy person will have a high probability of seeing a high saliency region. Therefore, by changing the second standard according to the spread of the high saliency region, it is possible to reduce erroneous abnormality determinations by the driver.

Further, in the above embodiment, in step ST13 of FIG. 19, it is determined whether there is a correlation between the spread of the high saliency region of the saliency map and the magnitude of the amplitude ds of the driver's saccade. However, it is not limited to this. For example, the saccade frequency fs may be used as a parameter in addition to or in place of the saccade amplitude ds.

The technology disclosed herein is useful as a driver condition determination device installed in a vehicle.

21a Front camera (external information acquisition means)
30 Control unit (vehicle control unit)
302 Gaze point detection unit

Claims (3)

A vehicle control device installed in a vehicle,
a gaze point detection unit that detects a gaze point of the driver of the vehicle;
Receive external environmental information from the external information acquisition means that acquires environmental information on the front side of the vehicle, generate a salience map based on the external environmental information, and determine the salience relative to the surroundings in the salience map. When the degree of correlation between the degree of spread of the high saliency region in the saliency map and the amplitude of the saccade of the driver detected by the gaze point detection unit is higher than a predetermined first criterion, Vehicle control that performs an operation corresponding to an abnormality of the driver when an eye attraction degree, which is a degree to which the driver's gaze point detected by the viewpoint detection unit is attracted to the high salience area, is higher than a predetermined second standard. A vehicle control device comprising: The vehicle control device according to claim 1,
A vehicle control device characterized in that the second reference is changed depending on the degree of spread of the high saliency region in the saliency map. A driver condition determination method for determining a driver condition executed by a vehicle control device, the method comprising:
a gaze point detection step of detecting a gaze point of a vehicle driver and a saccade amplitude based on the movement of the gaze point;
a map generation step of generating a saliency map based on the vehicle exterior environment information received from the vehicle exterior information acquisition means that acquires the vehicle front side environment information;
In the saliency map generated in the map generation step, the degree of spread of a high saliency region that is relatively salient with respect to the surroundings in the saliency map and the driver's saccade detected in the gaze point detection step When there is a correlation between the magnitude of the amplitude and the degree of attraction, which is the degree to which the driver's gaze point detected in the gaze point detection step is attracted to the high salience area, is higher than a predetermined reference value. and a determining step of determining that the driver has an abnormality.

Images