JP2021077140A - Vehicle control device and driver state determination method - Google Patents

Abstract

To provide a vehicle control device and a driver state determination method that more appropriately respond to an abnormality of a driver.SOLUTION: In a vehicle control system, a driver state detection unit 300 of a vehicle control unit includes: a gazing point detection unit 302 that detects a gazing point of a driver of a vehicle; and a control unit that receives image data from a front camera 21a for acquiring environmental information on a vehicle front side, and takes an action responding to an abnormality of the driver when the tendency in which the gazing point of the driver detected by the gazing point detection unit 302 is directed to a high saliency region is higher than a given second reference, in the case where the degree of correlation between extent of the high saliency region based on the image data and amplitude of a saccade of the driver detected by the gazing point detection unit 302 is higher than a predetermined first reference.SELECTED DRAWING: Figure 4

Description

The technique disclosed herein relates to, for example, a vehicle control device and a driver state determination method for determining the state of a driver who drives a vehicle.

Recently, the development of autonomous driving systems is being promoted nationally. At present, the applicant of the present application considers that the autonomous driving system has two main directions.

The first direction is a system in which the automobile takes the lead in transporting the occupants to the destination without requiring the driver's operation, which is the so-called fully automatic driving of the automobile. The second direction is an automatic driving system that is premised on human driving, such as "providing an environment where people can enjoy driving a car."

In the second directional automatic driving system, for example, when a driver has a disease or the like and a situation occurs in which normal driving is difficult, the automobile automatically changes to the occupant and automatically drives. Is assumed. For this reason, how quickly and accurately it is possible to detect the occurrence of an abnormality in the driver, especially the occurrence of a dysfunction or disease in the driver, is the improvement of the lifesaving rate of the driver and the safety including the surroundings. It is extremely important from the viewpoint of ensuring.

As a method of determining a driver's abnormality, for example, in Patent Document 1, a front image of a vehicle is mapped, a conspicuity map (saliency map) is generated based on the pixel value of the mapped front image, and a conspicuity map (saliency map) is generated. A technique has been disclosed in which the degree of conspicuity of the driver's line of sight is calculated, and the degree to which the driver directs the line of sight to the conspicuous part is compared with a predetermined threshold value to determine whether or not the driver is in a vague state.

In the technique of Patent Document 1, when the driver is in a loose state, the intentional attention function does not work, so that the line of sight does not go to a dangerous place where, for example, a pedestrian may jump out from the blind spot area. , The judgment is based on the line-of-sight distribution characteristic that the line of sight is directed to the part where the visual stimulus is conspicuous.

Japanese Patent No. 5966640

As a result of line-of-sight detection by a dysfunctional person and a healthy person with a driving simulator, the inventors often do not attract the dysfunctional person to the area with high salience, or even the healthy person is attracted to the area with high salience. It turned out to occur.

For example, when a region with high salience is a point of caution that should be seen while driving at the same time, even a healthy person may look toward the region with high salience, and as a result, it may be erroneously determined that a disease has occurred.

The technology disclosed here has been made in view of these points, and is as accurate as possible in a driver condition determination device that determines a driver's condition based on a tendency to attract salience, regardless of the driving environment of the vehicle. The purpose is to be able to detect anomalies in the driver.

In order to solve the above-mentioned problems, in the technique disclosed here, a gaze point detecting unit for detecting a gaze point of the driver of the vehicle and an environment on the front side of the vehicle are targeted for a vehicle control device mounted on the vehicle. Correlation between the spread of the high salience region based on the outside environment information and the amplitude of the sack of the driver detected by the gaze detection unit after receiving the outside environment information from the outside information acquisition means for acquiring the information. When the degree is higher than the predetermined first criterion, the driver's gaze point detected by the gaze point detection unit is more likely to see a region with high salience than the predetermined second criterion. The configuration is provided with a vehicle control unit that operates in response to an abnormality.

Here, salience is a visual feature amount consisting of remarkableness that changes from moment to moment depending on color, brightness, movement, and the like. The high salience region refers to a region having high salience (high prominence) in the driver's field of vision, in other words, a region that stands out with respect to the surroundings. More specifically, the high salience region refers to, for example, a region in which a color difference or a luminance difference is large with respect to a surrounding region or a large movement is made with respect to the surrounding region.

Further, the saccade is a jumping eye movement in which the driver intentionally moves the line of sight, and is an eye movement in which the line of sight is moved from a gaze point where the line of sight is stagnant for a predetermined time to the next gaze point. The amplitude of the saccade refers to the amount of movement when the driver's line of sight moves from the gaze point where the driver's line of sight is stagnant for a predetermined time to the next gaze point.

According to this configuration, by looking at the degree of correlation between the extent of the high salience region and the amplitude of the driver's saccade, it is possible to extract the tendency that the driver's viewpoint is attracted to the high salience region. By combining the tendency of the driver's viewpoint to be attracted to the high-salience area and the tendency of the driver's gaze to see the high-salience area, it is possible to more appropriately deal with the driver's abnormality.

As one embodiment of the vehicle control device, the second criterion may be changed according to the expansion of the high salience region.

When the extent of the salience region is large, the high salience region also increases, so it is expected that even a healthy person will have a high probability of seeing the high salience region. Therefore, by changing the second criterion according to the expansion of the high salience region, it is possible to reduce the erroneous determination of the driver's abnormality determination.

In order to solve the above-mentioned problems, in the technique disclosed here, the gaze point of the driver of the vehicle and the movement of the gaze point are targeted at the driver state determination method for determining the state of the driver of the vehicle. A gaze point detection step for detecting the sackde amplitude based on the above, a map generation step for generating a salency map based on the outside environment information received from the outside information acquisition means for acquiring the environment information on the front side of the vehicle, and the map generation. When there is a correlation between the extent of the high salency region of the salency map generated in the step and the magnitude of the sacked amplitude of the driver detected in the gazing point detection step, it is detected by the gazing point detection unit. It is provided with a determination step of determining that the driver has an abnormality when the driver's gaze point is higher than a predetermined reference value in the tendency to see the high amplitude region.

According to this method, by observing the degree of correlation between the extent of the high salience region and the amplitude of the driver's saccade, it is possible to extract the tendency that the driver's viewpoint is attracted to the high salience region. By combining the tendency of the driver's viewpoint to be attracted to the high salience region and the tendency of the driver's gaze to see the region of high salience, the accuracy of the driver's abnormality determination can be improved.

As described above, according to the technology disclosed here, it is possible to more appropriately deal with abnormalities of the driver.

It is the schematic which shows the front side part in the vehicle interior of the vehicle equipped with the driver state estimation device which concerns on an exemplary embodiment. It is a front view which looked at the vehicle from the front side. It is a block diagram which illustrates the structure of the vehicle control system which concerns on an exemplary embodiment. It is a block diagram which illustrates the structure of the driver state detection part. It is a figure which shows an example of the external environment image of the front side of a vehicle taken by a front camera. It is a figure which shows an example of the composite image based on the external environment image of FIG. It is a figure which shows an example of the saliency map of the composite image of FIG. It is a graph which shows an example of the line-of-sight movement by a healthy person and a patient with attention dysfunction in the example of FIG. It is a figure which shows another example of the composite image based on the external environment image of the front side of a vehicle taken by the front camera. It is a figure which shows an example of the saliency map of the composite image of FIG. It is a graph which shows an example of the line-of-sight movement by a healthy person and a patient with attention dysfunction in the example of FIG. It is a graph for demonstrating the extraction of a saccade. It is a graph for demonstrating the noise removal processing of a saccade. It is a conceptual diagram which schematically shows the relationship between the spread of the driver's salience and the saccade amplitude. It is a figure which showed an example of the time change of the salience of the driver's gaze point. It is a figure which showed an example of the time change of salience when the coordinates are randomly specified in the same environment as FIG. It is a graph comparing the ratio exceeding the threshold value at a random point and the ratio exceeding the threshold value at the driver's gaze point. It is a figure for demonstrating the salience index. It is a flowchart which illustrates the operation of a vehicle control device. It is a figure which showed an example of the time distribution of the salience index of a healthy person. It is a figure which showed an example of the time distribution of the salience index of the attention function patient.

<Knowledge obtained by the inventors of the present application>
As shown in Patent Document 1, there is known a technique for detecting an abnormality (an abnormality causing a decrease in attention function) of a driver by observing a change in the movement of the line of sight with respect to the salience of the driver. In this technique, for example, when the movement of the line of sight to a region with high salience (high salience region) exceeds a predetermined threshold value, an abnormality of the driver is detected.

As a result of diligent research, the inventors of the present application observe the behavior of a person with attention dysfunction when driving a vehicle, thereby simulating the behavior of the driver of the vehicle at an abnormal time (an abnormal time causing a decrease in attention function). I found that I could do it. Then, based on the above observation results, the inventors of the present application have verified a method of detecting an abnormality of the driver when the movement of the line of sight to the high salience region exceeds a predetermined threshold value. More specifically, the behavior of a person with attention dysfunction when driving a vehicle and the behavior of a healthy person without attention dysfunction when driving a vehicle are moved to a high salience region by using the salience index described later. We verified the method of detecting the tendency of the driver and detecting the driver's abnormality when the salience index exceeds a predetermined threshold.

As a result of this verification, the inventors of the present application have found that there are many cases in which even dysfunctional persons are not attracted to areas with high salience, and even healthy persons are attracted to areas with high salience. For example, if the area with high salience is a point of caution that should be seen while driving at the same time, even a healthy person may look at the area with high salience, and as a result, it may be erroneously determined that a disease has occurred. I found out.

FIGS. 20 and 21 show the driver's line of sight detected using a driving simulator and plotted as a time change of the salience index. FIG. 20 is a measurement result of a healthy person, and FIG. 21 is a measurement result of a dysfunctional patient. However, since the traveling speeds are different from each other, the time axis and the traveling place do not always match.

The details will be described later, but the salience index is an index in which the higher the degree of attraction to the high salience region, the higher the numerical value. In the examples of FIGS. 20 and 21, a threshold value is set to 0.6, which is judged to have a relatively high degree of attraction to the high salience region, and when the salience index exceeds 0.6, there is an impairment of attention function. It was supposed to be estimated.

Then, it was found that there were some scenes in which the salience index exceeded 0.6 even in healthy subjects (see AR1 in FIG. 20). For example, when overtaking a vehicle with high salience such as a fire engine, even a healthy person tends to be attracted to a place with high salience. Further, even if the patient has a dysfunction, it may be confirmed even in a place where the salience index is low in a place where the risk is high such as an intersection, and a region where the salience index is low is confirmed (see AR2 in FIG. 21).

Therefore, the inventors of the present application have conducted further diligent studies and found that an abnormality of the driver can be detected by observing the spatial spread of salience in addition to the salience index.

Specifically, the inventors of the present application use a salience index to detect whether the driver's gaze point tends to move toward a high salience region, and at the same time, the spatial spread of salience and the spread of the driver's line-of-sight distribution. Detects the degree of correlation with. Then, when the driver's gaze tends toward the high salience region and the degree of correlation between the spatial spread of salience and the spread of the line-of-sight distribution is high, it is determined that the driver has an abnormality. , It was found that the accuracy of driver's abnormality detection can be improved.

Hereinafter, exemplary embodiments will be specifically described with reference to the drawings. In the following description, the forward traveling side of the vehicle is simply referred to as the front side, and the backward traveling side is simply referred to as the rear side. The left side when looking at the front side from the rear side is called the left side, and the opposite is called the right side.

FIG. 1 schematically shows the interior of an automobile as a vehicle. This vehicle is a right-hand drive vehicle, and the steering wheel 58 is arranged on the right side.

In the vehicle interior, the front window glass 51 is arranged on the front side of the vehicle when viewed from the driver's seat. The front window glass 51 is partitioned by a plurality of vehicle components when viewed from the vehicle interior side. Specifically, the front window glass 51 is partitioned by the left and right front pillar trims 52, the roof trim 53, and the instrument panel 54.

The left and right front pillar trims 52 form boundaries on the outer side of the front window glass 51 in the vehicle width direction. Each front pillar trim 52 is arranged along each front pillar. The roof trim 53 constitutes the upper boundary of the front window glass 51. The roof trim 53 covers the vehicle interior side of the roof panel of the vehicle. A rear-view mirror 55 is attached to the center of the front window glass 51 in the vehicle width direction and slightly below the roof trim 53. In the vicinity of the rear-view mirror 55 in the roof trim 53, an in-vehicle camera 28 (see FIG. 3) for photographing the vehicle interior, particularly the driver's face, is provided. The in-vehicle camera 28 will be described in detail later. The instrument panel 54 constitutes the lower boundary of the front window glass 51. The instrument panel 54 is provided with a meter box and a display 57.

Further, the vehicle has side mirrors 56 on the outer side in the vehicle width direction with respect to the left and right front pillars. Each side mirror 56 is arranged so that the driver seated in the driver's seat can see through the window of the side door.

As shown in FIG. 2, the vehicle is provided with a camera (hereinafter, referred to as a front camera 21a) for photographing the external environment on the front side of the vehicle. The front camera 21a is a front end portion of the vehicle and is arranged slightly below the bonnet 59 of the vehicle. The front camera 21a is an example of an out-of-vehicle information acquisition means for acquiring environmental information on the front side of the vehicle.

<Vehicle control system>
FIG. 3 illustrates the configuration of the vehicle control system 10 according to the embodiment. The vehicle control system 10 is provided in a vehicle (specifically, a four-wheeled vehicle). The vehicle can be switched between manual driving, assisted driving, and automatic driving. Manual driving is driving in which the vehicle travels in response to a driver's operation (for example, accelerator operation). Assisted driving is driving that assists the driver's operation. Autonomous driving is driving that runs without the operation of the driver. The vehicle control system 10 controls the vehicle in assisted driving and automatic driving. Specifically, the vehicle control system 10 controls the operation (particularly traveling) of the vehicle by controlling the actuator 11 provided in the vehicle.

The vehicle control system 10 includes an information acquisition unit 20, a control unit 30, and a notification unit 40. In the following description, the vehicle provided with the vehicle control system 10 will be referred to as "own vehicle", and other vehicles existing around the own vehicle will be referred to as "other vehicle".

-Actuator-
The actuator 11 includes a drive system actuator, a steering system actuator, a braking system actuator, and the like. Examples of drive system actuators include engines, motors, and transmissions. An example of a steering system actuator is steering. An example of a braking system actuator is a brake.

-Information acquisition department-
The information acquisition unit 20 acquires various information used for vehicle control (particularly travel control). In this example, the information acquisition unit 20 includes a plurality of cameras 21, a plurality of radars 22, a position sensor 23, an external input unit 24, a vehicle state sensor 25, a driving operation sensor 26, and a driver state sensor 27. And include.

〔camera〕
The plurality of cameras 21 have the same configuration as each other. The plurality of cameras 21 are provided on the vehicle so as to surround the periphery of the vehicle. Each of the plurality of cameras 21 acquires image data showing a part of the external environment of the vehicle by capturing a part of the environment spreading around the vehicle (external environment of the vehicle). The image data obtained by each of the plurality of cameras 21 is transmitted to the control unit 30. The plurality of cameras 21 include the above-mentioned front camera 21a.

In this example, the camera 21 is a monocular camera with a wide-angle lens. For example, the camera 21 is configured by using a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary metal-oxide-semiconductor). The camera 21 may be a monocular camera having a narrow-angle lens, or a stereo camera having a wide-angle lens or a narrow-angle lens.

[Radar]
The plurality of radars 22 have similar configurations to each other. The plurality of radars 22 are provided on the vehicle so as to surround the vehicle. Each of the plurality of radars 22 detects a part of the external environment of the vehicle. Specifically, the radar 22 detects a part of the external environment of the vehicle by transmitting radio waves to a part of the external environment of the vehicle and receiving reflected waves from a part of the external environment of the vehicle. To do. The detection results of the plurality of radars 22 are transmitted to the control unit 30.

For example, the radar 22 may be a millimeter-wave radar that transmits millimeter waves, a lidar (Light Detection and Ranging) that transmits laser light, or an infrared radar that transmits infrared rays. It may be an ultrasonic sensor that transmits ultrasonic waves.

[Position sensor]
The position sensor 23 detects the position of the vehicle (eg, latitude and longitude). For example, the position sensor 23 receives GPS information from the Global Positioning System and detects the position of the vehicle based on the GPS information. The information (vehicle position) obtained by the position sensor 23 is transmitted to the control unit 30.

[External input section]
The external input unit 24 inputs information through an external network (for example, the Internet) provided outside the vehicle. For example, the external input unit 24 receives communication information from another vehicle (not shown) located around the vehicle, car navigation data from a navigation system (not shown), traffic information, high-precision map information, and the like. The information obtained by the external input unit 24 is transmitted to the control unit 30.

[Vehicle status sensor]
The vehicle condition sensor 25 detects the condition of the vehicle (for example, speed, acceleration, yaw rate, etc.). For example, the vehicle condition sensor 25 includes a vehicle speed sensor that detects the speed of the vehicle, an acceleration sensor that detects the acceleration of the vehicle, a yaw rate sensor that detects the yaw rate of the vehicle, and the like. The information (vehicle condition) obtained by the vehicle condition sensor 25 is transmitted to the control unit 30.

[Driving operation sensor]
The driving operation sensor 26 detects a driving operation applied to the vehicle. For example, the driving operation sensor 26 includes an accelerator opening degree sensor, a steering angle sensor, a brake oil pressure sensor, and the like. The accelerator opening sensor detects the amount of operation of the accelerator of the vehicle. The steering angle sensor detects the steering angle of the steering wheel of the vehicle. The brake oil sensor detects the amount of operation of the vehicle brake. The information (driving operation of the vehicle) obtained by the driving operation sensor 26 is transmitted to the control unit 30.

[Driver status sensor]
The driver state sensor 27 detects the state of the driver driving the vehicle (for example, the driver's health state, emotions, physical behavior, etc.). The information (driver's state) obtained by the driver state sensor 27 is transmitted to the control unit 30. In this example, the driver status sensor 27 includes an in-vehicle camera 28 and a biometric information sensor 29.

《Camera in the car》
The in-vehicle camera 28 is provided inside the vehicle. The in-vehicle camera 28 acquires image data including the driver's eyes by capturing an image of a region including the driver's eyes. The image data obtained by the in-vehicle camera 28 is transmitted to the control unit 30. For example, the in-vehicle camera 28 is arranged in front of the driver, and the imaging range is set so that the driver's eyeball is within the imaging range. The in-vehicle camera 28 may be provided on goggles (not shown) worn by the driver.

《Biological information sensor》
The biological information sensor 29 detects the driver's biological information (for example, sweating). The information (driver's biological information) obtained by the biological information sensor 29 is transmitted to the control unit 30.

-Control unit-
In assisted driving or automatic driving, the control unit 30 determines a target route, which is a route to be traveled by the vehicle, based on various information acquired by the information acquisition unit 20, and is required to travel on the target route. Determine the target movement, which is the movement of the vehicle. Then, the control unit 30 controls the operation of the actuator 11 so that the movement of the vehicle becomes the target movement. For example, the control unit 30 is composed of an electronic control unit (ECU) having one or more arithmetic chips. In other words, the control unit 30 is composed of an electronic control unit (ECU) having one or more processors, one or more memories for storing programs and data for operating the one or more processors, and the like. To.

In this example, as shown in FIG. 3, the control unit 30 includes an image processing unit 31, an external environment recognition unit 32, a candidate route generation unit 33, a vehicle behavior recognition unit 34, and a driver behavior recognition unit 35. It has a target route determining unit 36 and a motion control unit 37.

[Image processing unit]
The image processing unit 31 receives images captured by a plurality of cameras 21 and performs image processing. The image processing performed by the image processing unit 31 includes a first image processing for an image used by the external environment recognition unit 32 for recognizing an external environment such as an object, and a second image processing for generating a saliency map. Image processing and is included. The first image processing is an image processing conventionally used for automatic driving and the like, and detailed description thereof will be omitted here.

<< About the second image processing >>
The image processing unit 31 deletes, for example, pixels unnecessary for processing (for example, generation of a salency map) of the map generation unit 301, which will be described later, among the elements constituting the image, with respect to the image captured by the front camera 21a. Perform processing. Further, the image processing unit 31 performs a process of synthesizing another image with respect to the image showing the external environment on the front side of the vehicle taken by the front camera 21a to generate a composite image.

FIG. 5 is an example of an image D11 showing the external environment on the front side of the vehicle taken by the front camera 21a in the traveling scene of the vehicle. The external environment shown in the image D11 includes a roadway 150 and a white line 151 on the roadway 150. Further, in the external environment shown in this image D11, a wall 162 formed on the left side of the roadway 150, a forest 163 formed in an area on the left side of the wall 162, and a hill extending on the right side area of the roadway 150. 164 and forest 165 formed on the hill 164 are included. Further, the external environment shown in the image D11 includes the roadway 150 and the sky 167 extending above the forests 163 and 165. In the following description, it is assumed that the sky 167 of image D11 is a sunset sky and a reddish sky is spreading. In other words, as shown in FIG. 7, the image D11 is an image in which the spread RW of the high salience region is relatively large. Note that FIG. 7 will be described in detail later.

The image processing unit 31 synthesizes with the image D11 an image showing a vehicle component that enters the driver's field of view when the vehicle is traveling. Specifically, as shown in FIG. 6, the image processing unit 31 synthesizes an image of a vehicle component (hereinafter referred to as a vehicle image) when the driver seated in the driver's seat looks at the front side of the vehicle with the image D11. Then, the composite image D12 is generated. The vehicle components photographed in the vehicle image are, for example, the front pillar trim 52 on the right side (driver's seat side), the right part of the roof trim 53, the right part of the instrument panel 54, the rearview mirror 55, and the right side. Side mirror 56 and steering wheel 58. For the vehicle image, for example, the structure inside the vehicle that enters the driver's field of view from the driver's seat side can be photographed in advance, and the photographed image data can be stored as a layer in the memory (not shown) of the control unit 30. .. Then, when synthesizing the vehicle image with the image D11, the image processing unit 31 may read the vehicle image from the memory into a layer different from the image D11, align the image D11 with the vehicle image, and superimpose the vehicle image. A part of the bonnet 59 may be further considered as a vehicle component. Further, in creating the composite image D12, information other than the front camera 21a, for example, the detection result of the radar 22 or the input information from the external input unit 24 may be used.

FIG. 9 is an example of a composite image D22 in which a vehicle image is combined with an image of the external environment on the front side of the vehicle taken by the front camera 21a in a driving scene different from that of FIGS. 5 to 8. The composite image D22 is generated by synthesizing the captured image of the front camera 21a and the vehicle image, as in the case of the composite image D12. The external environment shown in the composite image D22 includes the roadway 150 and the white line 151 on the roadway 150. Further, the external environment shown in the composite image D22 includes another vehicle 161 and walls 162 formed on both sides of the roadway 150. Trees 168 are planted in the area to the left of the left wall 162, and trees 169 are planted on the hill 164 extending to the area to the left of the left wall 162. Similarly, 168 trees are planted on a hill 164 that extends to the area to the left of the right wall 162. The external environment shown in image D22 includes the roadway 150, the hills 164, and the sky 170 extending above the trees 168,169. In the following description, it is assumed that the sky 170 of the composite image D22 is a blue sky. In other words, it is assumed that the image D22 is an image in which the spread RW of the high salience region is relatively small, as shown in FIG. Note that FIG. 10 will be described in detail later.

As shown in FIGS. 6 and 9, the composite images D12 and D22 generated by the image processing unit 31 are in a state where a part of the images is blocked by the vehicle constituent members. The image data (hereinafter referred to as composite image data) of the composite images D12 and D22 generated by the image processing unit 31 is transmitted to the map generation unit 301 described later.

[External Environment Recognition Department]
The external environment recognition unit 32 recognizes the external environment of the vehicle based on the data output from the plurality of cameras 21 and the radar 22. The external environment of the vehicle recognized by the external environment recognition unit 32 includes an object. Examples of objects include a moving body that displaces with the passage of time and a stationary body that does not displace with the passage of time. Examples of moving objects include motorcycles, motorcycles, bicycles, pedestrians and the like. Examples of stationary bodies include signs, roadside trees, medians, centerpoles, buildings and the like.

[Candidate route generator]
The candidate route generation unit 33 generates one or a plurality of candidate routes based on the output of the external environment recognition unit 32. The candidate route is a route on which the vehicle can travel and is a candidate for the target route.

[Vehicle behavior recognition unit]
The vehicle behavior recognition unit 34 recognizes the behavior of the vehicle (for example, speed, acceleration, yaw rate, etc.) based on the output of the vehicle condition sensor 25. For example, the vehicle behavior recognition unit 34 recognizes the behavior of the vehicle from the output of the vehicle state sensor 25 using the learning model generated by the deep learning.

[Driver behavior recognition unit]
The driver behavior recognition unit 35 recognizes the driver's behavior (for example, the driver's health condition, emotion, physical behavior, etc.) based on the output of the driver condition sensor 27. For example, the driver behavior recognition unit 35 recognizes the driver's behavior from the output of the driver state sensor 27 using the learning model generated by deep learning. In this example, the driver behavior recognition unit 35 has a driver state detection unit 300. The driver state detection unit 300 will be described in detail later.

[Target route determination unit]
The target route determination unit 36 sets the target route from one or a plurality of candidate routes generated by the candidate route generation unit 33 based on the output of the vehicle behavior recognition unit 34 and the output of the driver behavior recognition unit 35. Select a candidate route. For example, the target route determination unit 36 selects a candidate route that the driver feels most comfortable with among the plurality of candidate routes.

[Motion control unit]
The motion control unit 37 determines the target motion based on the candidate path selected as the target path by the target path determination unit 36, and controls the actuator 11 based on the determined target motion. For example, the motion control unit 37 derives the target driving force, the target braking force, and the target steering amount, which are the driving force, the braking force, and the steering amount for achieving the target motion, respectively. Then, the motion control unit 37 sets the drive command value indicating the target driving force, the braking command value indicating the target braking force, and the steering command value indicating the target steering amount to the actuator of the drive system, the actuator of the braking system, and the steering system. It sends to the actuator and each. The drive system actuator, the braking system actuator, and the steering system actuator are examples of the actuator 11.

《Notification section》
The notification unit 40 notifies the driver of the vehicle of various information. In this example, the notification unit 40 includes a display unit 41 and a speaker 42. The display unit 41 outputs various information as an image. The speaker 42 outputs various information by voice.

<< Driver status detector >>
The driver state detection unit 300 detects an abnormality of the driver of the vehicle. Specifically, as shown in FIG. 4, the driver state detection unit 300 includes a map generation unit 301, a gazing point detection unit 302, and an abnormality detection unit 303.

The driver's abnormality is an abnormality that causes the driver's attention function to deteriorate. Examples of such driver abnormalities include brain diseases such as stroke, heart diseases such as myocardial infarction, epilepsy, hypoglycemia, and drowsiness.

[Map generator]
The map generation unit 301 receives the composite images D12 and D22 (hereinafter collectively referred to as composite images D2) from the image processing unit 31 and generates a surrency map D3 for the composite images D12 and D22. Specifically, the map generation unit 301 calculates the salience of the portion of the composite image data representing the external environment, that is, the portion other than the vehicle image synthesized by the image processing unit 31. At this time, the map generation unit 301 does not calculate the salience of the vehicle image used for the composition in the image processing unit 31, but uses it for calculating the salience of the portion of the composite image D2 representing the external environment. That is, the map generation unit 301 generates the saliency map D3 in consideration of the vehicle components that enter the driver's field of view when the vehicle is running.

As described above, salience changes depending on the color, brightness, movement, etc. of the target. Therefore, in the present embodiment, the map generation unit 301 calculates the salience for each feature such as color-based salience, brightness-based salience, and motion-based salience, and after generating the salience map for each feature, they are used. Are added together to generate a salience map D13 (see FIG. 7) based on the final composite image D12 and a salience map D23 (see FIG. 10) based on the composite image D22. The saliency map D13 and the saliency map D23 are examples of the saliency map D3.

For example, regarding color-based salience, the map generation unit 301 determines that when the color difference between the neighborhood region and the vehicle component is large in the vicinity region of the vehicle component in the composite image data, the color difference is smaller than when the color difference is small. Increase the salience of nearby areas. The color difference is calculated by the following formula when the RGB of the color of a certain pixel is (R1, G1, B1) and the RGB of the color of another pixel is (R2, G2, B2).
(Color difference) = {(R2-R1) ² + (G2-G1) ² + (B2-B1) ² } ^1/2
The map generation unit 301 may calculate so that the larger the color difference, the higher the salience continuously, or may set a plurality of threshold values and calculate so that the salience increases by a certain value each time the threshold value is exceeded. ..

Further, for example, regarding the salience based on the brightness, the map generation unit 301 determines that when the brightness difference between the neighborhood region and the vehicle constituent member is large in the vicinity region of the vehicle constituent member in the composite image data, the brightness difference is small. In comparison, the salience of the neighboring region is increased. For example, when the combined vehicle component is black, the map generation unit 301 increases the salience of the portion of the vicinity region that is close to white because the brightness difference becomes larger as the portion is closer to white. The map generation unit 301 may calculate so that the larger the luminance difference is, the higher the salience is continuously, or may set a plurality of threshold values and calculate so that the salience increases by a certain value each time the threshold value is exceeded. Good.

For the calculation of salience itself, a known computer program published on the Internet or the like can be used. In addition, a known computer program can be used for calculating the saliency map for each feature and integrating each saliency map.

[Gaze point detector]
The gazing point detection unit 302 calculates the driver's line-of-sight direction from the driver's eyeball image taken by the in-vehicle camera 28. The gazing point detection unit 302 calculates the driver's line-of-sight direction by detecting a change in the driver's pupil from the state in which the driver looks into the lens of the in-vehicle camera 28, for example. The line-of-sight direction may be calculated from either the driver's left eye or the right eye, or may be the average value of the line-of-sight directions (line-of-sight vectors) obtained from each of the driver's eyes. Further, when it is difficult to calculate the driver's line-of-sight direction from the change in the driver's pupil, the line-of-sight direction may be calculated in consideration of the direction of the driver's face. Further, a learning model (learning model for detecting the line of sight) generated by deep learning may be used for calculating the line-of-sight direction of the driver.

<< Detection of gazing point and saccade >>
The gazing point detection unit 302 detects the driver's saccade based on the movement of the driver's line of sight. The saccade is a jumping eye movement in which the driver intentionally moves the line of sight, and is an eye movement in which the line of sight is moved from a gaze point where the line of sight is stagnant for a predetermined time to the next gaze point. As shown in FIG. 12, the period sandwiched between two adjacent gaze periods is the saccade period. The gaze period is a period in which the line of sight is considered to be stagnant. The amplitude ds of the saccade is the moving distance of the line of sight during the saccade period.

The gazing point detection unit 302 calculates the moving speed of the line of sight based on the change in the moving distance of the line of sight, and a state in which the moving speed of the line of sight is less than a predetermined speed threshold value (for example, 2 deg / s) is predetermined. The period during which the stagnation time (for example, 0.1 second) continues is extracted as the "gaze period", and the point ahead of the driver's line of sight during the gaze period is extracted as the "gaze point". Further, in the gazing point detection unit 302, among the line-of-sight movements in the period sandwiched between two adjacent gazing periods, the moving speed is equal to or higher than the speed threshold value (for example, 2deg / s), and the moving distance is predetermined. The movement of the line of sight that is equal to or greater than the distance threshold (for example, 0.1 deg) is extracted as a “saccade”.

The gaze point detection unit 302 may perform noise removal processing on the extracted saccade. Specifically, as shown in FIG. 13, the gazing point detection unit 302 derives the regression curve L10 based on a plurality of saccade candidates. The gazing point detection unit 302 derives the regression curve L10 from a plurality of saccade candidates by, for example, the least squares method. Next, the gazing point detection unit 302 derives the first reference curve L11 by shifting the regression curve L10 in the direction in which the moving speed increases (the increasing direction in the vertical axis of FIG. 13), and moves the regression curve L10 in the moving speed. The second reference curve L12 is derived by shifting in the direction of decrease (decrease direction on the vertical axis of FIG. 13), and the sackde range R10 is defined between the first reference curve L11 and the second reference curve L12. Then, the gazing point detection unit 302 extracts the saccade candidates included in the saccade range R10 from the plurality of saccade candidates as saccades.

Next, the gazing point detection unit 302 calculates the amplitude ds of the saccade, which is an index of the saccade, and the frequency fs of the saccade. Specifically, the gazing point detection unit 302 calculates the average value of the saccade amplitude ds included in the predetermined cycle (for example, every 10 seconds) as the “saccade amplitude ds”. The value obtained by dividing the number of saccades included in the cycle by the time of the cycle is calculated as "saccade frequency fs".

[Abnormality detection unit]
The abnormality detection unit 303 detects the driver's abnormality based on the saliency map D3 generated by the map generation unit 301 and the driver's gaze point and saccade amplitude ds detected by the gaze point detection unit 302. To do. The operation of the abnormality detection unit 303 will be specifically described in the following "Operation of the vehicle control system".

-Vehicle control system operation-
Hereinafter, the operation of the vehicle control system will be described with reference to FIG.

(Step ST11)
First, in step ST11, the vehicle control system executes a gaze point detection step of detecting the gaze point of the driver of the vehicle. Specifically, in step ST11, for example, as described in the above-mentioned section “Detection of gazing point and saccade”, the gazing point detection unit 302 notes based on the driver's line of sight detected by the in-vehicle camera 28. The amplitude ds of the viewpoint and the saccade is detected.

(Step ST12)
In the next step ST12, the vehicle control system executes a map generation step of generating a saliency map based on the image data received from the front camera 21a. Specifically, in step ST12, for example, as described in the section of the above-mentioned "map generation unit", the map generation unit 301 creates the saliency map D3 based on the composite image D2 synthesized by the image processing unit 31. Generate. The execution order of step ST11 and step ST12 is not particularly limited. For example, step S12 may be executed before step S11, or step ST11 and step ST12 may be executed at the same time.

(Step ST13)
In step ST13, it is determined whether or not there is a correlation between the extent of the high salience region of the salience map generated in step ST12 and the magnitude of the driver's saccade amplitude ds detected in step ST11.

Specifically, for example, the anomaly detection unit 303 increases the spread RW of the high-salience region of the salience map generated by the map generation unit 301 and the amplitude ds of the driver's saccade detected by the gazing point detection unit. Determine if there is a correlation with. More specifically, the abnormality detection unit 303 prepares, for example, a plurality of salience maps D3 having different spread RWs in the high salience region. Then, the anomaly detection unit 303 expands the high correlation region of each salience map D3 and the amplitude of the saccade in a predetermined time range (for example, 5 seconds) in which the image underlying the salience map D3 is taken. The presence or absence of mutual correlation is determined by comparing with ds.

FIG. 14 is a conceptual diagram showing the relationship between “expansion RW of high salience region” and “saccade amplitude ds” for a healthy person and a person with attention dysfunction. In FIG. 14, the thick solid line schematically shows the characteristics of a healthy person, and the fine solid line schematically shows the characteristics of a person with attention dysfunction.

Here, if the driver is in a normal state (for example, a healthy person), he / she drives while looking at a parked vehicle, a pedestrian, a side road, or other visible object to be watched when the vehicle is running. That is, a healthy person drives while looking at a visual object to be watched regardless of the extent of the spread of salience. Therefore, as shown by the thick solid line in FIG. 14, the degree of correlation between the magnitude of the spread RW of the high salience region and the amplitude ds of the saccade tends to be low. On the other hand, when the driver becomes unable to drive actively due to an abnormal physical condition or the like, the driver naturally turns his / her gaze to an area with high salience. That is, when the driver frequently sees the area with a high degree of attraction, there is a high possibility that the driver has an abnormality. Then, when the driver has an abnormality and the spread RW of the high salience region is large, the amount of movement of the driver's line of sight increases, and the amplitude ds of the saccade tends to increase. On the other hand, when there is an abnormality in the driver and the spread RW of the high salience region is small, the amount of movement of the driver's line of sight tends to decrease, and the amplitude ds of the saccade tends to decrease. That is, when there is an abnormality in the driver, as shown by the fine solid line in FIG. 14, the degree of correlation between the magnitude of the spread RW of the high salience region and the magnitude of the amplitude ds of the saccade tends to be high. .. Therefore, in step ST11, the degree of correlation between the spread RW of the high salience region of the salience map generated by the map generation unit 301 and the amplitude ds of the driver's saccade detected by the gaze point detection unit is a predetermined first degree. If it is higher than one criterion, the process proceeds to the next step ST14. On the other hand, in step ST13, when the degree of correlation between the spread RW of the high salience region and the amplitude ds of the saccade is equal to or less than the first reference, the process returns to step ST11. Here, the predetermined first criterion may be any reference as long as it can determine the high degree of correlation, and can be arbitrarily set. For example, as shown by a broken line in FIG. 14, a reference line that serves as a reference indicating that the degree of correlation is high is drawn, and the correlation is based on the relationship with the reference line (for example, the degree of approximation or the degree of agreement of the slope). The degree may be determined. Step ST13 is, for example, an example of processing that constitutes a part of the determination step.

(Step ST14)
In step ST14, it is determined whether or not the driver's gaze point detected in step ST11 has a higher tendency to see the high salience region than a predetermined reference value. Specifically, for example, the tendency of the line of sight to move to the high salience region is detected by using the salience index, and it is determined whether or not the salience index exceeds a predetermined threshold value.

Specifically, the abnormality detection unit 303 has a region in which the driver has high salience based on the salience map generated by the map generation unit 301 and the driver's gaze point detected by the gaze point detection unit 302. Determine if the tendency to see is higher than a given second criterion. In this embodiment, it is determined whether the salience index is higher than a predetermined second criterion (for example, 0.6).

In the determination of step ST14, a numerical value other than 0.6 may be used as the second criterion of the salience index, or an index other than the salience index may be used.

<< Calculation of Salience Index >>
The method of calculating the salience index will be described with reference to FIGS. 15 to 18. The calculation of this salience index is mainly performed by the abnormality detection unit 303.

FIG. 15 is a plot of the time change of the salience of the driver's gaze point. In this graph, for example, as shown in FIGS. 8 and 11, the change in the gazing point is applied to the salience map, and the height of the salience of the gazing point is plotted each time a saccade occurs. On the other hand, in FIG. 16, coordinates (hereinafter referred to as random coordinates) are randomly specified from the same salience map as in FIG. 15 (for example, FIGS. 8 and 11), and the salience of random coordinates is obtained each time a saccade occurs. It is generated by.

After creating the graphs of FIGS. 15 and 16, respectively, the ROC (Receiver Operating Characteristic) curve of the ratio exceeding the threshold value at the random point and the ratio exceeding the threshold value at the driver's gaze point is obtained. Specifically, first, as a first step, the threshold value is set to a value larger than the maximum value in the graphs of FIGS. 15 and 16. Next, as a second step, while lowering the threshold value, the ratio of points exceeding the threshold value is obtained for each threshold value. Then, the process of this second step is repeated until the threshold value becomes equal to or less than the minimum value in the graphs of FIGS. 15 and 16.

After the first and second steps, as the third step, the ratio exceeding the threshold value at the random point (referred to as the first probability) is taken on the horizontal axis and the ratio exceeding the threshold value at the driver's gaze point (referred to as the second probability). For the vertical axis, a graph as shown in FIG. 17 is created. The graph of FIG. 17 represents the gaze probability for a random probability at the same threshold. In the graph of FIG. 17, since both the horizontal axis and the vertical axis are probabilities, the minimum value of the curve is 0 and the maximum value is 1.

In FIG. 17, the dashed line represents how the driver's line of sight moves with respect to salience. When the curve is convex upward as in the curve C1 of FIG. 17, it means that the driver's gaze point exceeds the threshold value at a higher rate than the random point. When a curve having a shape like the curve C1 in FIG. 17 is calculated, it means that the driver's line of sight tends to see a portion having high salience, that is, the degree of attraction to the high salience region is high. There is. On the other hand, when the curve is convex downward as in the curve C2 of FIG. 17, it means that the ratio of the driver's gaze point exceeding the threshold value is lower than the random point. When a curve having a shape like the curve C2 in FIG. 17 is calculated, it means that the driver's line of sight is not affected by the high salience region.

In the present embodiment, the AUC (Area Under Curve) is obtained as the fourth step. AUC is the integral value of the lower right part of this curve. That is, the AUC is the area of the region surrounded by the curve C1 (or curve C2) and the alternate long and short dash line in FIG. 17, as shown by the diagonal lines in FIG. In this embodiment, this integrated value is called a salience index. By using the salience index, it is possible to reduce the dependence on the driving scene. Specifically, for example, if the driver simply calculates the ratio of seeing the high salience area, the high salience area is often seen when the driving scene includes the high salience area as a whole. Results may be obtained. On the other hand, by using the salience index compared with the random points as in the present embodiment, it is possible to reduce the dependence on the driving scene such as the number and spread of the high salience region. You will be able to make highly accurate judgments.

If the salience index is higher than the second criterion in step ST14, the process proceeds to the next step ST15. On the other hand, in step ST14, when the salience index is equal to or less than the second criterion, the process returns to step ST11. Step ST14 is, for example, an example of processing that constitutes a part of the determination step. It should be noted that steps ST11 and ST12 are continuously executed while the vehicle is running, regardless of the processing of steps ST13 and 14.

(Step ST15)
In step S15, the control unit 30 performs control corresponding to the driver's abnormality. For example, when the degree of abnormality of the driver is high (for example, when the driver recognizes that a disease has developed), the control unit 30 controls the actuator 11 via the driving motion control unit 37 or the like to perform automatic driving. Perform the switching process. In addition, the control unit 30 carries out an actuation for the driver, such as asking the driver "Is it okay" or "Would you like to take a rest?" Through the notification unit 40, and the driver. You may try to confirm the reaction of. For example, the control unit 30 may control the actuator 11 to switch to automatic operation when the driver's reaction is weak as a result of the above-mentioned actuation.

In this way, by carrying out actions such as asking the driver and calling attention to the driver, it is possible to improve the accuracy of the driver's abnormality determination and encourage the driver to take safe actions.

As described above, according to the present embodiment, it is possible to detect the driver's abnormality as accurately as possible. Specifically, for example, as shown in FIG. 19, when only the salience index is viewed, even in a situation where a healthy person is erroneously determined to be abnormal, as shown by the thick solid line in FIG. If the correlation between the spread RW of the high salience region and the amplitude ds of the saccade is low, it is not determined that the driver is abnormal. As a result, the tendency that the line of sight is attracted to salience can be extracted more accurately, and as a result, the driver's abnormality can be accurately determined.

In the above embodiment, in step ST14 of FIG. 19, the second criterion may be changed according to the spread RW of the high salience region. When the extent of the salience region is large, the high salience region also increases, so it is expected that even a healthy person will have a high probability of seeing the high salience region. Therefore, by changing the second criterion according to the expansion of the high salience region, it is possible to reduce the erroneous determination of the driver's abnormality determination.

Further, in the above embodiment, in step ST13 of FIG. 19, it is determined whether or not there is a correlation between the extent of the high salience region of the salience map and the magnitude of the amplitude ds of the driver's saccade. However, it is not limited to this. For example, the frequency fs of the saccade may be used as a parameter in addition to or changed by the amplitude ds of the saccade.

The technology disclosed herein is useful as a driver condition determination device mounted on an automobile.

21a Front camera (means for acquiring information outside the vehicle)
30 Control unit (vehicle control unit)
302 Gaze point detector

Claims (3)

A vehicle control device installed in a vehicle
A gaze point detection unit that detects the gaze point of the driver of the vehicle,
The outside environment information is received from the outside information acquisition means for acquiring the environment information on the front side of the vehicle, the expansion of the high salience region based on the outside environment information and the amplitude of the driver's sacked detected by the gazing point detection unit. When the degree of correlation with is higher than the predetermined first criterion, and the driver's gaze point detected by the gaze point detection unit has a higher tendency to see a region with high salience than the predetermined second criterion. In addition, a vehicle control device including a vehicle control unit that performs an operation corresponding to an abnormality of the driver. In the vehicle control device according to claim 1,
A vehicle control device characterized in that the second criterion is changed according to the expansion of the high salience region. It is a driver state determination method for determining the state of the driver of a vehicle.
A gaze point detection step for detecting the gaze point of the driver of the vehicle and the saccade amplitude based on the movement of the gaze point, and the gaze point detection step.
A map generation step of generating a survivor map based on the vehicle exterior environment information received from the vehicle exterior information acquisition means for acquiring the environment information on the front side of the vehicle, and a map generation step.
When there is a correlation between the extent of the high salience region of the salience map generated in the map generation step and the magnitude of the driver's saccade amplitude detected in the gaze point detection step, the gaze point detection step The driver state is characterized by comprising a determination step of determining that the driver has an abnormality when the gaze point of the driver detected in the above is higher than a predetermined reference value. Judgment method.

Images