JP7548847B2 - Driving Support Devices - Google Patents

Description

This disclosure relates to a driving assistance device that reflects the driver's driving characteristics in automated driving control.

In recent years, vehicles equipped with driving assistance functions and autonomous driving functions have been put into practical use, mainly for the purpose of reducing traffic accidents and easing the burden on drivers. For example, there is known a device that detects obstacles around the vehicle based on information detected by various sensors such as an exterior camera and LiDAR (Light Detection and Ranging) installed in the vehicle, and assists the driving of the vehicle to avoid collisions between the vehicle and the obstacles.

Various technologies have been proposed for such driving assistance devices to eliminate driver discomfort. For example, Patent Document 1 discloses a driving assistance device that, during manual driving control, identifies the driving environment by a combination of environmental item values detected for environmental items, learns the driver's driving operation in association with the driving environment, and, during automatic driving control, detects environmental item values to identify the driving environment and executes automatic driving control by referring to the learning results of the driving operation in that driving environment. Patent Document 2 also discloses a driving assistance device that acquires information indicating the driving operation and information indicating the driving situation during the driving operation, determines whether the driving situation is appropriate for learning based on the acquired information, and determines that driving operations in driving situations determined to be inappropriate are not to be learned.

JP 2015-089801 A JP 2019-127207 A

However, although safety can be improved by excluding driving situations that are inappropriate for learning from the learning target, there is a risk of increasing discomfort to the driver. For example, suppose that the visibility of obstacles that cause blind spots changes between day and night, and there are situations at night where the driver has difficulty noticing the obstacle. In such a situation, if the driver notices an obstacle and decelerates in front of the obstacle during manual driving in the daytime, the driving assistance device learns the driving operation. On the other hand, if the driver does not notice an obstacle and does not decelerate during manual driving at night, the driving assistance device excludes the driving operation from the learning target. If automatic driving control of the vehicle is executed at night based on the results of such repeated learning, the daytime driving operation that was the learning target is referenced, and the vehicle decelerates even though the driver does not recognize the obstacle, which may cause the driver to feel uncomfortable.

This disclosure has been made in consideration of the above problems, and the purpose of this disclosure is to provide a driving assistance device that can reduce the discomfort felt by the driver when automatic driving control is executed by narrowing down the driving operations to be learned and reflecting the learned driving characteristics of the driver.

In order to solve the above problem, according to one aspect of the present disclosure, there is provided a driving assistance device that learns the driving characteristics of a driver during manual driving of a vehicle, thereby reflecting the driving characteristics of the driver in automatic driving control of the vehicle, and the driving environment information, including information on the vehicle's running state and information on the surrounding environment, acquired during automatic driving control is constructed by learning using a learning target dataset, which is driving environment information when it is determined that the driving operation acquired during manual driving is suitable for learning the driving characteristics, and a non-learning target dataset, which is driving environment information when it is determined that the driving operation is not suitable for learning, as learning data, and estimates whether the driver will perform a driving operation that is not suitable for learning. A driving assistance device is provided that includes an acquisition unit that acquires an estimation result obtained by inputting the estimation result into a driving operation estimation model for learning, an estimation unit that, when the estimation result is an estimation result that the driver will perform a driving operation unsuitable for learning, identifies the conditions of the driving environment in which the driver is estimated to perform a driving operation unsuitable for learning by the driving operation estimation model based on accumulated information of the conditions of the driving environment determined to be a driving operation unsuitable for learning, which is extracted by comparing a learning target dataset with a learning non-target dataset, and estimates an unrecognized object that the driver does not recognize, and a notification control unit that notifies the driver of information related to the estimated unrecognized object.

As described above, according to the present disclosure, it is possible to reduce the discomfort felt by the driver when autonomous driving control is executed by narrowing down the driving operations to be learned and reflecting the learned driving characteristics of the driver.

1 is a schematic diagram illustrating a configuration example of a vehicle equipped with a driving assistance device according to an embodiment of the present disclosure. 2 is a block diagram showing a configuration example of a driving assistance device according to the embodiment; FIG. 5 is a flowchart showing a process of a learning phase performed by the driving assistance device according to the embodiment; 5 is a flowchart showing a process of an execution phase performed by the driving assistance device according to the embodiment; FIG. 2 is an explanatory diagram showing a driving environment during manual driving in a first application example. FIG. 1 is an explanatory diagram showing a driving environment during autonomous driving in a first application example. FIG. 13 is an explanatory diagram showing a driving environment during manual driving in a second application example. FIG. 11 is an explanatory diagram showing a driving environment during autonomous driving in a second application example.

Below, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals to avoid redundant description.

<1. Overall configuration of the vehicle>
First, an example of the overall configuration of a vehicle equipped with a driving assistance device according to an embodiment of the present disclosure will be described.

FIG. 1 is a schematic diagram showing an example of the configuration of a vehicle 1 equipped with a driving assistance device 50.
1 is configured as a four-wheel drive vehicle in which a driving torque output from a driving force source 9 that generates a driving torque for the vehicle is transmitted to a left front wheel 3LF, a right front wheel 3RF, a left rear wheel 3LR, and a right rear wheel 3RR (hereinafter collectively referred to as "wheels 3" unless a distinction is required). The driving force source 9 may be an internal combustion engine such as a gasoline engine or a diesel engine, a driving motor, or may include both an internal combustion engine and a driving motor.

The vehicle 1 may be an electric vehicle equipped with two drive motors, for example a front-wheel drive motor and a rear-wheel drive motor, or an electric vehicle equipped with drive motors corresponding to each wheel 3. In addition, if the vehicle 1 is an electric vehicle or a hybrid electric vehicle, the vehicle 1 is equipped with a secondary battery that stores the power supplied to the drive motors, and a generator such as a motor or fuel cell that generates power to charge the battery.

The vehicle 1 is equipped with a driving force source 9, an electric steering device 15, and brake devices 17LF, 17RF, 17LR, 17RR (hereinafter collectively referred to as "brake devices 17" unless a distinction is required) as devices used to control the operation of the vehicle 1. The driving force source 9 outputs a driving torque that is transmitted to the front wheel drive shaft 5F and the rear wheel drive shaft 5R via a transmission, a front wheel differential mechanism 7F, and a rear wheel differential mechanism 7R (not shown). The operation of the driving force source 9 and the transmission is controlled by a vehicle control unit 41 that includes one or more electronic control units (ECUs: Electronic Control Units).

The front-wheel drive shaft 5F is provided with an electric steering device 15. The electric steering device 15 includes an electric motor and a gear mechanism (not shown), and is controlled by the vehicle control unit 41 to adjust the steering angle of the left front wheel 3LF and the right front wheel 3RF. During manual driving, the vehicle control unit 41 controls the electric steering device 15 based on the steering angle of the steering wheel 13 by the driver. During automatic driving control, the vehicle control unit 41 also controls the electric steering device 15 based on the set driving trajectory.

Brake devices 17LF, 17RF, 17LR, and 17RR apply braking forces to front, rear, left, and right drive wheels 3LF, 3RF, 3LR, and 3RR, respectively. Brake devices 17 are configured, for example, as hydraulic brake devices, and a predetermined braking force is generated by controlling the hydraulic pressure supplied to each brake device 17 by the vehicle control unit 41. When vehicle 1 is an electric vehicle or hybrid electric vehicle, brake devices 17 are used in conjunction with regenerative braking using a drive motor.

The vehicle control unit 41 includes one or more electronic control devices that control the drive of the driving force source 9 that outputs the driving torque of the vehicle 1, the electric steering device 15 that controls the steering angle of the steering wheel or steered wheels, and the brake device 17 that controls the braking force of the vehicle 1. The vehicle control unit 41 may also have a function of controlling the drive of a transmission that changes the speed of the output output from the driving force source 9 and transmits it to the wheels 3. The vehicle control unit 41 is configured to be able to acquire information transmitted from the driving assistance device 50, and is configured to be able to execute automatic driving control of the vehicle 1.

The vehicle 1 also includes forward-facing cameras 31LF, 31RF, a rearward-facing camera 31R, an interior camera 33, a vehicle state sensor 35, a GPS (Global Positioning System) sensor 37, and an HMI (Human Machine Interface) 43. Note that in this embodiment, the rearward-facing camera 31R may be omitted.

The front photographing cameras 31LF, 31RF and the rear photographing camera 31R constitute an ambient environment sensor for acquiring information on the ambient environment of the vehicle 1. The front photographing cameras 31LF, 31RF and the rear photographing camera 31R photograph the front or rear of the vehicle 1 and generate image data. The front photographing cameras 31LF, 31RF and the rear photographing camera 31R are equipped with imaging elements such as CCD (Charged-Coupled Devices) or CMOS (Complementary Metal-Oxide-Semiconductor), and transmit the generated image data to the driving assistance device 50. In the vehicle 1 shown in FIG. 1, the front photographing cameras 31LF, 31RF are configured as a stereo camera including a pair of left and right cameras, and the rear photographing camera 31R is configured as a so-called monocular camera, but each may be either a stereo camera or a monocular camera.

In addition to the front imaging cameras 31LF, 31RF and rear imaging camera 31R, the vehicle 1 may also be equipped with cameras mounted on the side mirrors 11L, 11R, for example, to capture images of the left rear or right rear. In addition, the vehicle 1 may also be equipped with one or more sensors as surrounding environment sensors for acquiring information about the surrounding environment, such as LiDAR (Light Detection And Ranging), a radar sensor such as a millimeter wave radar, or an ultrasonic sensor.

The in-vehicle camera 33 captures images of the interior of the vehicle and generates image data. The in-vehicle camera 33 is equipped with an imaging element such as a CCD or CMOS, and transmits the generated image data to the driving assistance device 50. In this embodiment, the in-vehicle camera 33 is positioned so that it can capture an image of at least the driver. Only one in-vehicle camera 33 may be installed, or multiple cameras may be installed.

The vehicle state sensor 35 is composed of at least one sensor that detects the operation state and behavior of the vehicle 1. The vehicle state sensor 35 includes at least one of a steering angle sensor, an accelerator position sensor, a brake stroke sensor, a brake pressure sensor, or an engine speed sensor, and detects the operation state of the vehicle 1, such as the steering angle of the steering wheel or steering wheels, the accelerator opening, the amount of brake operation, or the engine speed. The vehicle state sensor 35 also includes at least one of a vehicle speed sensor, an acceleration sensor, or an angular velocity sensor, and detects the behavior of the vehicle, such as the vehicle speed, longitudinal acceleration, lateral acceleration, and yaw rate. The vehicle state sensor 35 transmits a sensor signal including the detected information to the driving assistance device 50.

The GPS sensor 37 receives satellite signals from GPS satellites. The GPS sensor 37 transmits the position information on the map data of the vehicle 1 contained in the received satellite signals to the driving assistance device 50. Note that instead of the GPS sensor 37, an antenna that receives satellite signals from other satellite systems that identify the position of the vehicle 1 may be provided.

The HMI 43 is driven by the driving assistance device 50 and presents various information to the driver by means of image display, audio output, and the like. The HMI 43 includes, for example, a display device provided in the instrument panel and a speaker provided in the vehicle. The display device may be a display device of a navigation system. The HMI 43 may also include a HUD (head-up display) that displays information on the front window, superimposed on the scenery around the vehicle 1.

<2. Driving support device>
Next, the driving assistance device 50 according to this embodiment will be described in detail.

(2-1. Configuration example)
FIG. 2 is a block diagram showing an example of the configuration of the driving support device 50 according to this embodiment.
The driving support device 50 is connected to an ambient environment sensor 31, an in-vehicle camera 33, a vehicle state sensor 35, and a GPS sensor 37 directly or via a communication means such as a controller area network (CAN) or a local inter net (LIN). A vehicle control unit 41 and an HMI 43 are also connected to the driving support device 50. The driving support device 50 is not limited to an electronic control device mounted on the vehicle 1, and may be a terminal device such as a smartphone or a wearable device.

The driving assistance device 50 includes a control unit 51, a memory unit 53, a driving characteristics database 55, a non-learned data execution condition database 57, and an object impact database 59. The control unit 51 includes one or more processors such as a CPU (Central Processing Unit) and various peripheral components. A part or all of the control unit 51 may be configured with updatable firmware or the like, or may be a program module executed by commands from the CPU or the like.

The storage unit 53 is composed of a storage element such as a RAM (Random Access Memory) or a ROM (Read Only Memory), or one or more storage media such as a HDD (Hard Disk Drive), a CD (Compact Disk), a DVD (Digital Versatile Disk), an SSD (Solid State Drive), a USB flash, a storage device, etc. However, the type of the storage unit 53 is not particularly limited. The storage unit 53 stores information such as computer programs executed by the control unit 51, various parameters used in the calculation process, detection data, and calculation results.

The driving characteristic database 55, the non-learned data execution condition database 57, and the object influence database 59 are each composed of a memory element such as a RAM or ROM, or a memory medium such as a HDD, CD, DVD, SSD, USB flash, or storage device. The driving characteristic database 55 is a database that stores the driver's driving operation in association with information on the surrounding environment of the vehicle 1 when the driving operation is performed. The non-learned data execution condition database 57 is a database that stores data on the driving operation when non-learned data is executed, as identified by the control unit 51. The object influence database 59 is a database that stores data on the influence of the surrounding environment on the driving operation determined to be non-learned data, as identified by the control unit 51.

(2-2. Functional Configuration)
The control unit 51 of the driving assistance device 50 according to this embodiment is configured to be capable of executing a process of learning the driving characteristics of the driver while the vehicle 1 is being manually driven, and thereby reflecting the driving characteristics of the driver in the automatic driving control of the vehicle 1. The control unit 51 is also configured to be capable of executing a process of notifying the driver of information on the surrounding environment that is presumably not recognized by the driver, while executing the automatic driving control of the vehicle 1 reflecting the learned driving characteristics.

As shown in FIG. 2, the control unit 51 includes a surrounding environment detection unit 61, a driver detection unit 63, a driving state detection unit 65, a judgment unit 67, a learning unit 69, an estimation unit 71, a driving condition setting unit 73, and a notification control unit 75.

(Ambient environment detection unit)
The surrounding environment detection unit 61 detects information on the surrounding environment of the vehicle 1. The information on the surrounding environment of the vehicle 1 includes, for example, information on obstacles and obstructions that cause obstacles on and beside the travel path of the vehicle 1, as well as meteorological environment information such as weather and brightness. Specifically, the surrounding environment detection unit 61 detects obstacles on and beside the travel path based on detection data transmitted from the surrounding environment sensor 31. The surrounding environment detection unit 61 also calculates the position of the obstacle as seen from the vehicle 1, the distance from the vehicle 1 to the obstacle, and the relative speed of the obstacle to the vehicle 1. The detected obstacles or obstructions include other vehicles, pedestrians, bicycles, buildings, utility poles, traffic signs, traffic signals, natural objects, and any other detection targets that exist around the vehicle. The surrounding environment detection unit 61 may also have a lane recognition function, such as detecting white lines on the road. The surrounding environment detection unit 61 may detect obstacles using information on the position on the map data of the vehicle 1 acquired via the GPS sensor 37 and road information ahead in the traveling direction.

The surrounding environment detection unit 61 detects meteorological environment information such as weather and brightness based on image data transmitted from the front imaging cameras 31LF, 31RF or the rear imaging camera 31R. The meteorological environment information may be detected based on information acquired from a telematics service via wireless communication means such as mobile communication, or may be detected using a sensor such as a raindrop sensor or an illuminometer. The surrounding environment detection unit 61 stores the detected information on the surrounding environment in the memory unit 53 as time-series data. The information on the surrounding environment, together with the driver information, constitutes driving environment information.

(Driver detection unit)
The driver detection unit 63 detects information about the driver of the vehicle 1. Specifically, the driver detection unit 63 performs face recognition processing of the driver sitting in the driver's seat based on the detection data transmitted from the in-vehicle camera 33, and identifies the driver. The driver may be identified when the in-vehicle camera 33 detects an occupant sitting in the driver's seat, for example, when the driving assistance device 50 is started. The driver detection unit 63 also detects the direction of the driver's line of sight based on the detection data transmitted from the in-vehicle camera 33. The driver detection unit 63 stores the detected direction of the driver's line of sight in the storage unit 53 as time-series data. The driver's information constitutes driving environment information together with information about the surrounding environment. The driver may be identified by an occupant's operation input to a touch panel or the like.

(Running condition detection unit)
The driving condition detection unit 65 detects information on the operation state and behavior of the vehicle 1 based on the detection data transmitted from the vehicle condition sensor 35. The driving condition detection unit 65 obtains information on the operation state of the vehicle 1, such as the steering angle of the steering wheel or steering wheels, the accelerator opening, the amount of brake operation, or the engine speed, and information on the behavior of the vehicle, such as the vehicle speed, the longitudinal acceleration, the lateral acceleration, and the yaw rate, and stores this information in the memory unit 53 as time-series data.

(Determination unit)
The determination unit 67 determines whether the driver's driving operation is suitable for learning based on the driving environment information acquired during manual driving. For example, the determination unit 67 may determine whether the driving operation is suitable for learning by determining one or more of whether the behavior of the vehicle 1 is stable, whether the running state of the vehicle 1 complies with traffic laws, whether meteorological conditions such as weather and brightness are good, and whether the driver's behavior is natural.

Specifically, the determination unit 67 may determine that the driving operation is not suitable for learning when the acceleration/deceleration or angular velocity of the vehicle 1 detected by the driving state detection unit 65 exceeds a predetermined threshold value, or when the vehicle speed exceeds or falls below the legal speed by a predetermined amount. The determination unit 67 may also determine that the driving operation is not suitable for learning when it is determined that the amount of rainfall or snowfall exceeds a predetermined amount, the surrounding brightness is determined to be equal to or less than a predetermined amount, or the visibility is determined to be less than a predetermined distance due to fog or mist, based on the information detected by the surrounding environment detection unit 61. The determination unit 67 may also determine that the driving operation is not suitable for learning when it is determined that the driver has his/her eyes closed or is looking away from the road, based on the information of the driver detected by the driver detection unit 63. The determination unit 67 adds information on whether the driving operation is suitable for learning to the driving environment information at each time and stores it.

(Learning Department)
The learning unit 69 executes a driving characteristic learning process for learning the driving characteristics of the driver using a learning target data set, which is driving environment information determined to be a driving operation suitable for learning, as learning data. For example, the learning unit 69 associates information on the driver's driving operation in the learning target data set with information on the behavior of the vehicle 1 and information on the surrounding environment when the driving operation is performed, and stores the information in the driving characteristic database 55. The learning unit 69 may use a known machine learning model to construct a driving characteristic learning model using the learning target data set as learning data. The machine learning model may be, for example, a random forest, or a calculation model using a neural network such as a support vector machine, a nearest neighbor method, or a deep learning, or a Bayesian network. The learning data of the driving characteristics is used to set the driving conditions of the vehicle 1 using data on the surrounding environment of the vehicle 1 detected during the automatic driving control of the vehicle 1 as input data.

The learning unit 69 also executes a learning process to learn driving environment information in which the driver performs driving operations that are not suitable for learning, using the learning target dataset and the non-learning target dataset, which is driving environment information determined to be driving operations that are not suitable for learning, as learning data. For example, the learning unit 69 compares the learning target dataset with the non-learning target dataset to extract driving environment conditions that caused the driving operations to be determined to be not suitable for learning, and stores the driving environment conditions in the non-learning target data execution condition database 57 in association with the non-learning target dataset. As driving environment conditions that are determined to be driving operations that are not suitable for learning, for example, rainfall or snowfall exceeding a predetermined amount, ambient brightness being equal to or less than a predetermined amount, visibility being less than a predetermined distance due to fog or mist, the driver having their eyes closed or driving while looking away, etc. are extracted.

The learning unit 69 uses the above-mentioned publicly known machine learning model to construct a driving operation estimation model for estimating whether the driver will perform driving operations that are not suitable for learning in each driving environment, using the extracted condition information and the non-learning data set as learning data. The driving operation estimation model is stored in a non-learning data execution condition database. The learning data and the driving operation estimation model are used to estimate whether the driver will perform driving operations that are not suitable for learning in the current driving environment, using information on the surrounding environment of the vehicle 1 detected in the automatic driving control of the vehicle 1 and information on the driver as input data.

In addition, when constructing the driving operation estimation model, the learning unit 69 determines the degree of influence on the driving operation that is determined to be unsuitable for learning for each object included in the surrounding environment, and stores information on the influence of the object in the object influence database 59. The objects included in the surrounding environment are various obstacles and obstructions that exist on the driving path of the vehicle 1 or beside the driving path. The influence of the object is the influence that may lead to a decrease in driving safety, such as the difficulty of seeing each object and the degree of reduced attention. The learning unit 69 determines the difficulty of seeing from the driver and the degree of reduced attention for each type of object under the driving operation conditions that are determined to be unsuitable for learning. The influence of the object can be evaluated, for example, at multiple levels set in advance, but the method of evaluating the influence is not particularly limited.

For example, the learning unit 69 uses a permutation importance technique to analyze which object the driving operation estimation model has determined to be unsuitable for learning, taking into account the risk of not being able to recognize the object, and stores the analysis result in the object influence database 59 in association with the conditions of the driving environment in which the driving operation is determined to be unsuitable for learning. However, the method of analyzing the influence of the object on the determination that the driving operation is unsuitable for learning is not limited to the example using the permutation importance technique.

(Estimation section)
In the case where it is estimated that the driver will perform a driving operation that is not suitable for learning under the current driving environment in the automatic driving control of the vehicle 1, the estimation unit 71 specifies the conditions under which the driver will perform the driving operation that is not suitable for learning, and estimates an object in the surrounding environment that the driver does not recognize (hereinafter, also referred to as a "non-recognized object"). Specifically, the estimation unit 71 inputs information on the surrounding environment of the vehicle 1 and information on the driver acquired during the automatic driving control of the vehicle 1 into a driving operation estimation model, and determines whether or not the driver will perform a driving operation that is not suitable for learning under the current driving environment based on the obtained estimation result. In other words, in the automatic driving control of the vehicle 1, the estimation unit 71 determines whether or not the current driving environment corresponds to a driving environment in which the driver would perform a driving operation that is not suitable for learning if the vehicle 1 were being manually driven.

In addition, when it is estimated that the driver will perform a driving operation that is not suitable for learning in the current driving environment, the estimation unit 71 refers to the learning data stored in the non-learning data execution condition database 57 to identify the conditions of the driving environment in which the driver is estimated to perform the driving operation that is not suitable for learning. For example, when the amount of rainfall or snowfall exceeds a predetermined amount, the surrounding brightness is below a predetermined level, visibility is less than a predetermined distance due to fog or mist, or the driver has his eyes closed or is looking away while driving, these conditions are identified as the conditions of the driving environment in which the driver is estimated to perform the driving operation that is not suitable for learning.

Furthermore, the estimation unit 71 estimates non-recognized objects that the driver does not recognize among the surrounding vehicles, obstacles, etc. detected as the surrounding environment based on the information stored in the object influence database 59. Specifically, the estimation unit 71 identifies objects in the surrounding environment that have a large influence on the conditions of the driving environment in which it is estimated that the driver will perform a driving operation that is not suitable for learning, based on the information stored in the object influence database 59. Furthermore, the estimation unit 71 sets the obstacles, etc. that correspond to the identified objects among the surrounding vehicles, obstacles, etc. detected as the surrounding environment as non-recognized objects.

In this embodiment, the estimation unit 71 functions as an acquisition unit that acquires information on the estimation results based on the driving operation estimation model.

(Operation condition setting section)
In the automatic driving control of the vehicle 1, the driving condition setting unit 73 basically sets the driving conditions of the vehicle 1 so as to avoid a collision with an obstacle present ahead in the traveling direction of the vehicle 1. For example, in the automatic driving control of the vehicle 1, the driving condition setting unit 73 sets a traveling trajectory capable of avoiding a collision between the vehicle itself and an obstacle, and sets a target steering angle for driving the vehicle 1 along the traveling trajectory. For example, the driving condition setting unit 73 may set the traveling trajectory of the vehicle 1 using a risk potential that is an index indicating the possibility that the vehicle 1 will collide with a pedestrian, a surrounding vehicle, or other obstacle. In this case, the risk potential is set so that the closer the distance to the obstacle is, the higher the collision risk is, and the driving condition setting unit 73 sets the traveling trajectory so that the vehicle 1 runs on a trajectory where the collision risk is smaller. In addition, when the risk potential is set so that the collision risk is smaller as the vehicle speed is lower, the driving condition setting unit 73 may reduce the collision risk by setting the traveling trajectory and the vehicle speed.

In addition, in this embodiment, the driving condition setting unit 73 sets driving conditions that reflect the driving characteristics of the driver using the driving characteristic learning model learned by the learning unit 69. For example, the driving condition setting unit 73 reflects the driver's driving characteristics in the realized automated driving control by correcting the level or setting range of the risk potential based on the learned driving characteristics of the driver. The driving condition setting unit 73 transmits information on the set driving conditions to the vehicle control unit 41. The vehicle control unit 41, which has received the information on the driving conditions, controls the operation of each control device based on the information on the set driving conditions. In this way, automated driving control that reflects the driver's driving characteristics is realized.

(Notification control section)
The notification control unit 75 notifies the occupants of the vehicle 1 by controlling the driving of the HMI 43. In this embodiment, when the estimation unit 71 estimates the presence of an unrecognized object, the notification control unit 75 notifies the driver of information on the estimated unrecognized object. The notification control unit 75 notifies the driver of the information on the unrecognized object by outputting a warning sound or voice, or by displaying an image or text. The specific notification method is not particularly limited, and may be to output a certain warning sound or voice, display an image or text, or to notify the position of the unrecognized object. In addition, the notification control unit 75 may notify the driver of information on the driving conditions set due to the presence of the estimated unrecognized object together with the information on the unrecognized object.

3. Operation of the driving support device
Next, an example of the operation of the driving support device according to the present embodiment will be specifically described with reference to a flowchart. Below, the example of the operation of the driving support device will be described by dividing the process into a learning phase executed during manual driving and an execution phase executed during automatic driving.

(3-1. Learning Phase Processing)
FIG. 3 is a flowchart showing an example of a process executed in the learning phase.
First, when the in-vehicle system including the driving assistance device 50 is started (step S11), the driving condition setting unit 73 of the control unit 51 determines whether or not the vehicle 1 is being manually driven (step S13). For example, the driving condition setting unit 73 determines whether or not the driving mode changeover switch is set to the manual driving mode. For example, the driving mode is configured to be switched based on an operation input by the occupant of the vehicle 1. If the vehicle 1 is not being manually driven (S13/No), the control unit 51 transitions to processing of the execution phase.

On the other hand, if the vehicle 1 is being manually driven (S13/Yes), the control unit 51 acquires driving environment information of the vehicle 1 (step S15). Specifically, the surrounding environment detection unit 61 of the control unit 51 acquires detection data transmitted from the surrounding environment sensor 31, and detects information on the surrounding environment of the vehicle 1 based on the detection data. In this embodiment, the surrounding environment detection unit 61 detects at least other vehicles, people, buildings, traffic signs, white lines, etc. that are present on or beside the driving path of the vehicle 1 based on the detection data transmitted from the surrounding environment sensor 31. In addition, the driver detection unit 63 of the control unit 51 detects information on the driver's face and line of sight based on image data transmitted from the in-vehicle camera 33. In addition, the driving state detection unit 65 of the control unit 51 detects information on the operation state and behavior of the vehicle 1 based on the detection data transmitted from the vehicle state sensor 35.

Next, the determination unit 67 of the control unit 51 determines whether the acquired driving environment information is data detected when a driving operation suitable for learning is performed (step S17). For example, the determination unit 67 may determine whether the driving operation is suitable for learning by determining one or more of whether the behavior of the vehicle 1 is stable, whether the running state of the vehicle 1 complies with traffic laws, whether meteorological conditions such as weather and brightness are good, or whether the driver's behavior is natural. The data of the driving environment information detected when a driving operation suitable for learning is performed is stored as a learning target dataset.

Specifically, the determination unit 67 may determine that the driving operation is not suitable for learning when the acceleration/deceleration or angular velocity of the vehicle 1 detected by the driving state detection unit 65 exceeds a predetermined threshold value, or when the vehicle speed exceeds or falls below the legal speed by a predetermined amount. The determination unit 67 may also determine that the driving operation is not suitable for learning when it is determined that the amount of rainfall or snowfall exceeds a predetermined amount, the surrounding brightness is determined to be equal to or lower than a predetermined amount, or the visibility is determined to be less than a predetermined distance due to fog or mist, based on the information detected by the surrounding environment detection unit 61. The determination unit 67 may also determine that the driving operation is not suitable for learning when it is determined that the driver has his/her eyes closed or is looking away from the road, based on the information of the driver detected by the driver detection unit 63.

Next, the learning unit 69 of the control unit 51 determines whether the acquired driving environment information is a learning target dataset based on the judgment result of step S17 (step S19). If the acquired driving environment information is a learning target dataset (S19/Yes), the learning unit 69 executes a driving characteristic learning process using the learning target dataset as learning data (step S21). For example, the learning unit 69 stores information on the driver's driving operation in the driving characteristic database 55 in association with information on the behavior of the vehicle 1 when the driving operation is performed and information on the surrounding environment. The learning unit 69 may use a known machine learning model to construct a driving characteristic learning model using the learning target dataset, which is driving environment information determined to be a driving operation suitable for learning, as learning data.

If the acquired driving environment information is the learning target data set (S19/Yes), the driving characteristic learning process is executed in step S21. If the acquired driving environment information is not the learning target data set (S19/No), the driving characteristic learning process is not executed. Instead, the learning unit 69 uses the acquired driving environment information and the driving environment information stored so far to compare the learning target data set with the learning target data set and extracts the difference (step S23). For example, in the case of a driving environment in which the visibility of an obstacle that creates a blind spot changes between day and night, the difference between day and night and whether the line of sight is directed to the obstacle or not is extracted as the difference in the driving environment. The learning unit 69 may compare the learning target data set and the learning target data set acquired when the vehicle travels through the same driving position on the map data. This allows the learning target data set and the learning target data set under at least the same road conditions to be compared, and the estimation accuracy of the conditions under which driving operations that are not suitable for learning are performed can be improved.

Next, the learning unit 69 extracts the conditions of the driving environment that caused the driving operation to be determined to be unsuitable for learning based on the difference between the learning data set and the non-learning data set, and stores the extracted conditions in the non-learning data execution condition database 57 (step S25). For example, in the case of a driving environment in which the visibility of the obstruction that creates the above-mentioned blind spot changes between day and night, nighttime and not directing the gaze toward the obstruction are extracted as the conditions of the driving environment that caused the driving operation to be determined to be unsuitable for learning.

Next, the learning unit 69 uses the extracted information on the conditions of the driving environment and the non-learning dataset as learning data to construct a driving operation estimation model for estimating whether the driver will perform driving operations that are not suitable for learning (step S27). Furthermore, the learning unit 69 determines the degree of influence on driving operations determined to be not suitable for learning for each object included in the surrounding environment, and stores the information on the influence of the object in the object influence database 59 (step S29). For example, the learning unit 69 uses a permutation importance method to analyze which obstacles the driving operation estimation model has determined to be not suitable for learning in consideration of the risk of not being able to recognize them, and determines the influence for each type of obstacle or obstruction.

Next, the learning unit 69 associates the determined information on the impact of the object with the conditions of the driving environment during the driving operation that was determined to be unsuitable for learning, and stores the information in the object impact database 59 (step S31). Next, the control unit 51 determines whether the in-vehicle system has stopped (step S33). If the in-vehicle system has stopped (S33/Yes), the control unit 51 ends the control process. On the other hand, if the in-vehicle system has not stopped (S33/No), the process returns to step S13 and repeats the processing of each step described so far.

In this way, when the vehicle 1 is being manually driven, a driving characteristic learning model is constructed based on driving environment information when a driving operation suitable for learning is performed. Also, when the vehicle 1 is being manually driven, a driving operation estimation model is constructed that estimates whether the driving environment of the vehicle 1 corresponds to a driving environment in which a driving operation unsuitable for learning may be performed. Furthermore, when the vehicle 1 is being manually driven, the conditions of the driving environment in which the driver may perform a driving operation unsuitable for learning are learned, and the objects in the surrounding environment that may affect a driving operation unsuitable for learning and the degree of influence of each object are learned. The driving assistance device 50 can store this learning data for each driver.

(3-2. Processing in the Execution Phase)
FIG. 4 is a flowchart showing an example of a process executed in the execution phase.
First, when the in-vehicle system including the driving assistance device 50 is started (step S41), the driving condition setting unit 73 determines whether the vehicle 1 is under automatic driving control (step S43), similar to the process of step S13. If the vehicle 1 is not under automatic driving control (S43/No), the control unit 51 proceeds to the learning phase process.

On the other hand, if the vehicle 1 is under automatic driving control (S43/Yes), similar to the process of step S15, the control unit 51 acquires driving environment information of the vehicle 1 (step S45). Specifically, the surrounding environment detection unit 61 of the control unit 51 acquires detection data transmitted from the surrounding environment sensor 31, and detects information on the surrounding environment of the vehicle 1 based on the detection data. In addition, the driver detection unit 63 of the control unit 51 detects information on the driver's face and line of sight direction based on image data transmitted from the in-vehicle camera 33. In addition, the driving state detection unit 65 of the control unit 51 detects information on the operation state and behavior of the vehicle 1 based on the detection data transmitted from the vehicle state sensor 35.

Next, the estimation unit 71 uses the surrounding environment information and the driver information from the acquired driving environment information as input data and estimates whether the driver will perform a driving operation that is not suitable for learning in the current driving environment based on the driving operation estimation model (step S47). In other words, in the automatic driving control of the vehicle 1, if the vehicle 1 is being manually driven, the estimation unit 71 determines whether the driver will perform a driving operation that is not suitable for learning in the current driving environment.

Next, the estimation unit 71 determines whether the driver will perform a driving operation that is not suitable for learning in the current driving environment based on the estimation result of step S47 (step S49). If it is not determined that the driver will perform a driving operation that is not suitable for learning (S49/No), the process returns to step S45 and repeats the processing of each step described above. On the other hand, if it is determined that the driver will perform a driving operation that is not suitable for learning (S49/Yes), the estimation unit 71 extracts objects that affect the driving operation from among the objects in the surrounding environment (step S51). Specifically, the estimation unit 71 refers to the learning data stored in the non-learning data execution condition database 57 to identify the conditions of the driving environment in which it is estimated that the driver will perform a driving operation that is not suitable for learning. Furthermore, the estimation unit 71 estimates objects that the driver does not recognize from among surrounding vehicles, obstacles, etc. detected as the surrounding environment based on the information stored in the object influence database 59.

Next, the estimation unit 71 evaluates the impact of each of the extracted objects based on the information stored in the object impact database 59 (step S53). In this embodiment, the estimation unit 71 sets the object with the greatest impact as a non-recognized object not recognized by the driver. Next, the notification control unit 75 of the control unit 51 notifies the driver of the extracted objects and information on the non-recognized objects estimated based on the impact of the objects (step S55). For example, the notification control unit 75 notifies the driver of the object and position information of the non-recognized object by outputting a warning sound or voice, or by displaying an image or text. In addition to the information on the non-recognized object, the notification control unit 75 may also notify the driver of the driving conditions set due to the presence of the estimated non-recognized object.

Next, the driving condition setting unit 73 of the control unit 51 sets driving conditions for the automatic driving control of the vehicle 1 based on the driving characteristic learning data (step S57). For example, the driving condition setting unit 73 uses a risk potential that sets the risk of collision with an obstacle or the risk of deviation from the driving path to set a driving trajectory and vehicle speed so that the vehicle 1 drives on a trajectory that reduces the risk. At that time, the level or setting range of the risk potential is corrected based on the driving characteristic learning data learned using the learning target dataset. This allows the driver's driving characteristics to be reflected in the automatic driving control that is realized.

Next, the control unit 51 determines whether the in-vehicle system has stopped (step S59). If the in-vehicle system has stopped (S59/Yes), the control unit 51 ends the control process. On the other hand, if the in-vehicle system has not stopped (S59/No), the control unit 51 returns to step S43 and repeats the process of each step described so far.

In this way, during automatic driving control of vehicle 1, the learning data of the driving characteristics of each driver is reflected and automatic driving control of vehicle 1 is executed. At that time, if the driving environment corresponds to one in which it may be determined that the driver performs driving operations that are not suitable for learning if the driving environment is manual, the conditions of the driving environment that are determined to be driving operations that are not suitable for learning are identified, and unrecognized objects that the driver does not recognize in such a driving environment are estimated, and information related to the unrecognized objects is notified to the driver. Therefore, when the driving conditions of vehicle 1 are set due to unrecognized objects, the driver can know the cause, and the discomfort of the driver can be reduced.

＜4. Application Examples＞
The driving support device 50 according to this embodiment has been described above. Below, some examples of driving scenes to which the driving support device 50 according to this embodiment is applied will be described.

(4-1. First Application Example)
5 and 6 are diagrams for explaining the first application example, each showing a driving scene in which the vehicle 1 passes straight through a T-junction with poor visibility. Fig. 5 shows the driving environment during manual driving in the daytime and at night, and Fig. 6 shows the driving environment during automatic driving control in the daytime and at night.

During manual driving as shown in FIG. 5, since it is bright during the day and it is relatively easy to see ahead, the driver notices the presence of a blind spot caused by the side wall 101 and sufficiently decelerates the vehicle 1 to pass through the T-junction. The driving assistance device 50 determines that the daytime driving operation is suitable for learning, and learns the driving characteristics of the driver using the driving environment information acquired at this time as a learning target dataset.

On the other hand, at night, it is dark and relatively difficult to see ahead, so the driver does not notice the blind spot caused by the side wall 101 and passes through the T-junction without slowing down. The driving assistance device 50 determines that the night driving operation is not suitable for learning, and sets the driving environment information acquired at this time as a non-learning dataset. In this case, the driving assistance device 50 extracts "daytime or nighttime" as the difference between the learning dataset and the non-learning dataset, and specifies that the condition determined to be the driving operation not suitable for learning is "nighttime". Furthermore, since the driver passes through the T-junction without noticing the side wall 101 and without slowing down the vehicle 1, the driving assistance device 50 highly evaluates the influence of the side wall 101 present in the surrounding environment in the night driving environment.

During the automatic driving control shown in FIG. 6, both during the day and at night, the driving assistance device 50 detects the presence of a side wall 101 that creates a blind spot, and uses automatic driving control to sufficiently decelerate the vehicle 1 to pass through the T-junction. During the day, the driver will also notice the presence of the blind spot caused by the side wall 101, so will not feel uncomfortable even if the vehicle 1 decelerates due to automatic driving control. On the other hand, at night, if the driver does not notice the side wall 101, the driver will feel uncomfortable when the vehicle 1 decelerates due to automatic driving control, as the cause is unknown.

However, the driving assistance device 50 determines that the driver will perform driving operations that are not suitable for learning in the nighttime driving environment, and estimates that the driver does not recognize the side wall 101 that exists in the surrounding environment based on the information stored in the object impact database 59. The driving assistance device 50 notifies the driver of the side wall 101 that is estimated to be unrecognized or the presence of a blind spot caused by the side wall 101. The driving assistance device 50 issues a voice notification such as, for example, "There is a difficult-to-see jump-out point ahead to the right, so please slow down." This allows the driver to know the reason why the vehicle 1 has decelerated due to the automatic driving control, and the discomfort felt by the driver can be reduced.

(4-2. Second application example)
7 and 8 are diagrams for explaining the second application example, each of which is an explanatory diagram showing a driving scene in which the vehicle 1 turns right at an intersection and there is a pedestrian about to cross a pedestrian crossing ahead of the right turn. Fig. 7 shows the driving environment during manual driving during the day and at night during rain, and Fig. 8 shows the driving environment during automatic driving control during the day and at night during rain.

During manual driving as shown in FIG. 7, because it is bright during the day and the road ahead is relatively easy to see, the driver notices the presence of a pedestrian 103 attempting to cross the crosswalk, stops the vehicle 1 temporarily until the pedestrian 103 has finished crossing, and then continues the vehicle 1 after the pedestrian 103 has finished crossing. The driving assistance device 50 determines that the daytime driving operation is suitable for learning, and learns the driver's driving characteristics using the driving environment information acquired at this time as a learning target dataset.

On the other hand, at night, it is dark and relatively difficult to see ahead, so the driver does not notice the presence of the pedestrian 103 trying to cross the crosswalk and turns right at the intersection without stopping. The driving assistance device 50 determines that the night driving operation is not suitable for learning, and sets the driving environment information acquired at this time as a non-learning dataset. In this case, the driving assistance device 50 extracts "daytime or nighttime" and "sunny or raining" as the difference between the learning dataset and the non-learning dataset, and specifies that the condition determined to be the driving operation not suitable for learning is "nighttime and raining". Furthermore, since the driver turns right at the intersection without noticing the pedestrian 103, resulting in a near miss with the pedestrian, the driving assistance device 50 highly evaluates the influence of the pedestrian 103 present in the surrounding environment in the night driving environment.

During the automatic driving control shown in FIG. 8, both during the day and at night, the driving assistance device 50 detects the presence of a pedestrian 103 attempting to cross the crosswalk, stops the vehicle 1 under automatic driving control, and allows the vehicle 1 to proceed after the pedestrian has finished crossing. During the day, the driver will also notice the presence of the pedestrian 103, so will not feel uncomfortable even if the vehicle 1 is stopped under automatic driving control. On the other hand, at night, if the driver does not notice the pedestrian 103, the driver will feel uncomfortable when the vehicle 1 is stopped under automatic driving control because the reason is unknown.

However, the driving assistance device 50 determines that the driver will perform driving operations that are not suitable for learning in the nighttime driving environment, and estimates that the driver does not recognize the pedestrian 103 present in the surrounding environment based on the information stored in the object impact database 59. The driving assistance device 50 notifies the driver of the presence of the pedestrian 103 that is estimated to be unrecognized. The driving assistance device 50 issues a voice notification such as, for example, "There is a pedestrian ahead where you will turn right, so we will stop temporarily." This allows the driver to know the reason why the vehicle 1 has stopped temporarily due to the automatic driving control, and the discomfort felt by the driver can be reduced.

<5. Summary>
As described above, the driving assistance device 50 according to the present embodiment learns the driving characteristics of the driver using the learning target data set acquired when the driver performs a driving operation suitable for learning during manual driving as learning data. Therefore, the driving conditions during automatic driving control of the vehicle 1 can be set based on the results of learning an appropriate driving operation by the driver, such as a stable behavior of the vehicle 1 and a low risk of overlooking a blind spot due to an obstacle or obstruction in front of the vehicle 1. Therefore, when performing automatic driving control that reflects the driving characteristics of the driver during manual driving, safe and stable automatic driving control can be realized.

The driving support device 50 according to this embodiment learns the conditions of the driving environment in which the driver may perform driving operations that are not suitable for learning, and the objects in the surrounding environment that affect the driving operations that are not suitable for learning, and the influence of the objects, based on the non-learning data set acquired when a driving operation that is not suitable for learning is performed during manual driving. Therefore, during automatic driving control of the vehicle 1, it is possible to estimate that the driver will perform driving operations that are not suitable for learning, such as overlooking obstacles or decreasing attention, if the driver is assumed to be driving manually, and to estimate unrecognized objects that the driver does not recognize in the driving environment. The driving support device 50 then notifies the driver of information related to the unrecognized objects, thereby reducing the discomfort felt by the driver when the steering angle of the vehicle 1 changes or the vehicle 1 accelerates or decelerates due to the unrecognized objects.

In addition, in the driving assistance device 50 according to this embodiment, by notifying the driver of information on driving conditions set due to unrecognized objects together with information related to unrecognized objects, the driver can be informed of the behavior exhibited by the vehicle 1. This can further reduce the sense of discomfort felt by the driver.

Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the present disclosure is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field to which the present invention pertains can conceive of various modified or revised examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

For example, in the above embodiment, the driving assistance device 50 mounted on the vehicle 1 has a learning function, but the present disclosure is not limited to such an example. For example, a part or all of the learning function of the driving assistance device 50 may be provided in a server device that can communicate via mobile communication means, and the driving assistance device 50 may be configured to transmit and receive data to and from the server device.

In addition, in the above embodiment, a driving operation estimation model is used to estimate whether the driver will perform a driving operation that is not suitable for learning, and the control unit 51 identifies the conditions of the driving environment in which it is estimated that the driver will perform a driving operation that is not suitable for learning using the driving operation estimation model, and estimates unrecognized objects that the driver does not recognize, but the present disclosure is not limited to such an example. The driving operation estimation model may be a driving environment estimation model for estimating whether the driving environment is one in which the driver can perform a driving operation that is not suitable for learning. In this case, the driving environment estimation model outputs an estimation result of whether the driving environment is one in which the driver can perform a driving operation that is not suitable for learning, for example.

In addition, a storage medium that stores a computer program that causes a processor constituting the control unit 51 to execute operations including inputting driving environment information, including information on the vehicle's driving state and information on the surrounding environment acquired during automatic driving control, into a driving operation estimation model for estimating whether the driver will perform a driving operation that is not suitable for learning, constructed by learning using a learning target dataset that is driving environment information when the driving operation is determined to be suitable for learning the driving characteristics acquired during manual driving and a learning non-target dataset that is driving environment information when the driving operation is determined to be unsuitable for learning as learning data, and acquiring an estimation result obtained; and, when the estimation result is an estimation result that the driver will perform a driving operation that is not suitable for learning, identifying the conditions of the driving environment in which the driver is estimated to perform a driving operation that is not suitable for learning by the driving operation estimation model based on accumulated information of the conditions of the driving environment that is determined to be an unsuitable driving operation, extracted by comparing the learning target dataset and the learning non-target dataset, and estimating an unrecognized object that the driver does not recognize; and notifying the driver of information related to the estimated unrecognized object.

1...vehicle, 31...surrounding environment sensor, 33...in-vehicle camera, 35...vehicle condition sensor, 50...driving support device, 51...control unit, 55...driving characteristics database, 57...non-learning data execution condition database, 59...object influence degree database, 61...surrounding environment detection unit, 63...driver detection unit, 65...driving condition detection unit, 67...determination unit, 69...learning unit, 71...estimation unit, 73...driving condition setting unit, 75...notification control unit

Claims (5)

A driving assistance device that learns driving characteristics of a driver during manual driving of a vehicle and reflects the driving characteristics of the driver in automatic driving control of the vehicle,
Driving environment information including vehicle driving state information and surrounding environment information acquired during autonomous driving control,
an acquisition unit that acquires an estimation result obtained by inputting a learning target data set, which is the driving environment information when the driving operation is determined to be suitable for learning the driving characteristics acquired during manual driving, and a non-learning target data set, which is the driving environment information when the driving operation is determined to be unsuitable for the learning, into a driving operation estimation model for estimating whether the driver will perform the driving operation that is unsuitable for the learning; and
When the estimation result is an estimation result that the driver performs a driving operation that is not suitable for the learning,
Based on accumulated information on conditions of a driving environment determined to be a driving operation not suitable for the learning, which is extracted by comparing the learning target dataset with the learning non-target dataset,
an estimation unit that specifies a driving environment condition in which the driver is estimated to perform a driving operation that is not suitable for the learning by the driving operation estimation model, and estimates a non-recognized object that is not recognized by the driver;
a notification control unit that notifies the driver of information related to the estimated unrecognized object;
A driving assistance device equipped with the above. a determination unit that determines whether a driving operation is suitable for learning based on the driving environment information acquired during manual driving;
A database that accumulates information on the conditions of the driving environment that are determined to be driving operations that are not suitable for the learning, the information being extracted by comparing the learning target dataset with the learning non-target dataset; and
a learning unit that performs learning using the training data set and the non-training data set as training data to generate the driving operation estimation model;
The driving assistance device of claim 1 , further comprising: The database accumulates information on the conditions of the driving environment that are determined to be driving operations that are not suitable for the learning for each driver,
The driving assistance device according to claim 2 , wherein the learning unit generates the driving performance estimation model for each driver. the learning unit calculates, for each object included in the surrounding environment of the vehicle, a degree of influence on the driving operation determined to be unsuitable for the learning based on the learned driving operation estimation model;
4. The driving assistance device according to claim 2, wherein the notification control unit notifies the driver of information related to the non-recognized object having a large influence among the information of the estimated non-recognized object during the autonomous driving control. The driving assistance device according to any one of claims 1 to 4, wherein the notification control unit notifies information of driving conditions set due to the non-recognized object together with information related to the estimated non-recognized object.

Images