WO2022050200A1

WO2022050200A1 - Information processing device, information processing method, and information processing program

Info

Publication number: WO2022050200A1
Application number: PCT/JP2021/031604
Authority: WO
Inventors: 英史大場; ヴァルーンアローラ; ウラディミールズロコリッツァ
Original assignee: ソニーセミコンダクタソリューションズ株式会社
Priority date: 2020-09-07
Filing date: 2021-08-27
Publication date: 2022-03-10
Also published as: JPWO2022050200A1

Abstract

This information processing device comprises an acquisition unit (101) that acquires the state of an operator of a vehicle and an automatic operation control unit (10112) that controls automatic operation that makes the vehicle travel autonomously. On the basis of the state of the operator as acquired by the acquisition unit, the automatic operation control unit finds a return quality that is the quality of behavior when the vehicle returns to manual operation by the operator from automatic operation. The automatic operation control unit quantifies the return quality found and thereby monitors the operator.

Description

Information processing equipment, information processing methods and information processing programs

This disclosure relates to information processing devices, information processing methods and information processing programs.

In recent years, the development of automatic driving technology in which a vehicle control system (information processing system) controls a vehicle has been actively carried out. However, even if such automatic driving technology becomes widespread, there are still technical issues before it becomes possible to drive at a running speed equivalent to that of existing manual driving by automatic driving with only autonomous independent control. It is believed that many will remain. Therefore, it is considered to try cooperative automatic driving that utilizes the advance monitoring information by limiting the automatic driving to, for example, the driving section where the road environment maintenance and the constant advance monitoring information of the road environment can be obtained. Has been done.

In that case, depending on the actual development status of the road infrastructure, etc., there are a section where autonomous driving is possible, which is a road section where autonomous driving control is possible by the vehicle control system, and a manual driving section, which is a road section where autonomous driving is not allowed. It is expected that a mixed situation will occur. That is, not only is the situation in which the vehicle control system completely autonomously and continuously performs automatic driving, but the driving control is not handed over from the above-mentioned automatic driving to manual driving in which the driver steers or the like. There can be situations where it shouldn't be.

Patent Document 1 describes a technique relating to the transfer of control from automatic operation to manual operation.

Japanese Unexamined Patent Publication No. 2018-180594

If the transfer of control from automatic driving to manual driving is executed in a state where the driver is not sufficiently prepared for manual driving, there is a risk of causing social adverse effects such as inducing an accident to the following vehicle. there were.

The object of the present disclosure is to provide an information processing device, an information processing method, and an information processing program capable of appropriately performing a transfer from automatic operation to manual operation.

The information processing apparatus according to the present disclosure includes an acquisition unit for acquiring the state of the driver of the vehicle and an automatic driving control unit for controlling automatic driving for autonomous driving of the vehicle. Based on the acquired driver's condition, the return quality, which is the quality of action when the vehicle returns from automatic driving to manual driving by the driver's driving, is obtained, and the obtained return quality is quantified. Monitor the driver for the driver.

It is a block diagram which shows the structural example of the schematic function of the vehicle control system applicable to the embodiment of this disclosure. It is a block diagram which shows the structure of an example of the information processing apparatus which comprises the automatic operation control part applicable to an embodiment. It is a schematic diagram for demonstrating the case where each automatic operation level of SAE is seen from the user's point of view as a usage state. It is a schematic diagram for schematically explaining the application of the automatic operation level 3. It is an example flowchart which outlines the transfer process from the automatic operation to the manual operation by the existing technology. It is an example flowchart which schematically shows the transfer process from the automatic operation to the manual operation which concerns on embodiment. It is an example flowchart which shows the flow from the itinerary setting to the transition to the automatic operation mode which concerns on embodiment. It is an example flowchart which shows the flow of the process by the automatic operation mode which concerns on embodiment. It is an example flowchart which shows the correspondence to the event which occurred in the automatic operation by the automatic operation level 4 which concerns on embodiment. It is a schematic diagram schematically showing an example of the bird's-eye view display of the itinerary applicable to the embodiment. It is a schematic diagram which shows the example of the bird's-eye view display which color-coded each section which concerns on embodiment. It is a schematic diagram which shows the example of the bird's-eye view display configured in an annular shape which concerns on embodiment. It is a schematic diagram which shows the example of the bird's-eye view display including the road information which concerns on embodiment. It is a functional block diagram of an example for demonstrating the function of the control by HCD in the automatic operation control part which concerns on embodiment. It is a functional block diagram of an example for demonstrating the function of the driver return delay evaluation unit which concerns on embodiment. It is a schematic diagram for demonstrating the high-precision update LDM applicable to an embodiment. It is a schematic diagram for demonstrating the acquisition of information by the remote support control I / F applicable to an embodiment. It is a functional block diagram of an example for explaining the function of the driver behavior change achievement level estimation part which concerns on embodiment. It is a schematic diagram for demonstrating the basic structure of the automatic operation level 4 applicable to an embodiment. It is a schematic diagram for demonstrating the ODD in the automatic operation level 4 which concerns on embodiment. It is an example flowchart for demonstrating the operation example of the automatic operation level 4 which concerns on embodiment. It is an example flowchart for demonstrating the operation example of the automatic operation level 4 which concerns on embodiment. It is a figure which shows typically the mode that the driver who is traveling on the road 7 by his own vehicle extends the section of the automatic driving level 3. It is an example flowchart which shows the process in the conditional automatic operation level 3 available section which concerns on embodiment. It is an example flowchart which showed the flow of the process of automatic operation applicable to an embodiment paying attention to ODD. It is an example flowchart which shows the example of the ODD setting process applicable to an embodiment in more detail. It is a schematic diagram for more concretely explaining the setting example of the ODD section applicable to an embodiment. It is a functional block diagram of an example for demonstrating the function of the driver behavior evaluation part applicable to DMS which concerns on embodiment. It is a functional block diagram of an example for demonstrating the function of the learning part configured offline which is applicable to an embodiment. It is a schematic diagram for schematically explaining the generation of the 3D head model applicable to an embodiment. It is a schematic diagram for schematically explaining the generation of a body model applicable to an embodiment. It is a schematic diagram for demonstrating the method of determining the wakefulness of a driver applicable to an embodiment. It is an example flowchart which shows the process which evaluates the quality of behavior which concerns on embodiment.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

Hereinafter, embodiments of the present disclosure will be described in the following order.
0. Outline of the present disclosure 1. Configuration applicable to embodiments of the present disclosure 2. Outline of the level of automatic operation by SAE 3. Embodiment 3-1 according to the present disclosure. Outline of Embodiment 3-2. HCD (Human Centered Design) according to the embodiment 3-2-1. Outline of HCD according to the embodiment 3-2-2. Advantages of HCD in autonomous driving 3-2-2-1. Overdependence 3-2-2-2. About HCD 3-2-2-3. Benefits for drivers 3-2-2-4. About the driver's working memory and thinking during driving 3-2-2-5. About the "contract" between the system and the driver 3-2-2-6. Operation of automatic operation level 4 3-2-2-7. Effect of adopting HCD 3-2-3. Specific Examples of HCD According to the Embodiment 3-2-3-1. Operation example of automatic operation to which HCD according to the embodiment is applied 3-2-3-2. Evaluation of driver's return behavior 3-2-3-3. About the bird's-eye view display of the itinerary applicable to the embodiment 3-2-4. HCD control configuration example according to the embodiment 3-3. Regarding automatic operation level 4 applicable to the embodiment 3-3-1. Basic structure 3-3-2. About ODD at automatic operation level 4 3-3-3. Operation example of automatic operation level 4 according to the embodiment 3-4. Application example of HCD for automatic operation level 3 3-5. Determinants of ODD 3-6. About DMS (Driver Monitoring System) according to the Embodiment 3-6-1. Outline of DMS according to the embodiment 3-6-2. More specific description of DMS according to the embodiment 3-6-3. Quantification of behavioral quality (QoA) according to the embodiment 3-6-4. Configuration applicable to DMS according to the embodiment 3-6-5. Specific example of evaluation of quality of behavior according to an embodiment 3-6-6. DMS summary according to the embodiment

<< 0. Summary of the present disclosure >>
In the present disclosure, when the driving control of a vehicle is taken over from the automatic driving in which the vehicle is autonomously driven to the manual driving in which the driver steers the vehicle, etc., the automatic driving system of the vehicle is changed to the manual driving for the driver. Regarding the processing when taking over. More specifically, in this disclosure, we support the self-learning of the driver naturally through the repeated use of the driver so that the transfer of driving from the vehicle to the driver can be performed smoothly. Provide a mechanism.

In other words, for the driver, the automatic driving function is only knowledge information obtained from desk explanations and materials at the beginning of its use, and it is an unknown function as a physical experience, so it is used due to psychological anxiety. It is still considered skeptical of inexperienced systems. In addition, as a normal action judgment, when a person takes a certain risk to take an action in order to obtain something, he / she makes a selection judgment so as to balance with the risk.

Therefore, drivers who have begun to use autonomous driving will remain aware of a certain level of caution even while using the autonomous driving function in order to reduce their sense of risk while they still have a sense of anxiety about driving. , The necessary consciousness required when using the automatic driving function is not completely lost and is maintained.

Here, when the event coping performance of the autonomous driving system is gradually improved and the anxiety that the user feels as a risk due to repeated use of the autonomous driving is reduced, the anxiety that the user of the autonomous driving is over-dependent on the autonomous driving disappears. It will be like. In particular, the advanced automatic driving function, which is about to be introduced in recent years, is an accident even if the driver is required to manually drive from automatic driving and it is difficult for the driver to return to manual driving under the conditions. It is required to have a function to avoid the problem and take countermeasures and minimize the influence even in the situation where an accident is unavoidable.

When autonomous driving with such advanced functions is realized, the driver's anxiety as a risk of excessive dependence on autonomous driving gradually disappears, and the driver's preparation is insufficient for the request to take over for manual driving. There is a risk that a situation will occur. Therefore, when it is difficult for the driver to take necessary measures within the deadline, the introduction of emergency stop by automatic driving as a safe measure is being considered as a process for minimizing the risk of impact.

However, when a vehicle frequently decelerates or makes an emergency stop in many road environments, there are other factors such as sudden deceleration of the following vehicle, stop in a road environment with poor visibility, and blockage of narrow roads such as bridges with narrow traffic bands. A situation that hinders the driving of a vehicle may occur in a form in which the driver himself / herself cannot directly see the effect, and the effect may lead to a decrease in the efficiency of the road environment, which is an artery of social activity. In other words, the existing autonomous driving control mechanism did not have a means for the driver to reflect these social impacts as a sense of risk in the behavioral judgment when using the autonomous driving function.

The purpose of this disclosure is to provide a mechanism that allows the driver to reflect the above-mentioned social impact as a sense of risk in the behavioral judgment when using the automatic driving function.

<< 1. Configuration applicable to embodiments of the present disclosure >>
First, the configuration applicable to the embodiment of the present disclosure will be described.

FIG. 1 is a block diagram showing a configuration example of a schematic function of a vehicle control system 10100, which is an example of a mobile control system applicable to the embodiment of the present disclosure.

Hereinafter, when a vehicle provided with the vehicle control system 10100 is distinguished from other vehicles, it is referred to as an own vehicle or an own vehicle.

The vehicle control system 10100 includes an input unit 10101, a data acquisition unit 10102, a communication unit 10103, an in-vehicle device 10104, an output control unit 10105, an output unit 10106, a drive system control unit 10107, and a drive system system 10108. , A body system control unit 10109, a body system system 10110, a storage unit 10111, and an automatic operation control unit 10112.

Of these, the input unit 10101, the data acquisition unit 10102, the communication unit 10103, the output control unit 10105, the drive system control unit 10107, the body system control unit 10109, the storage unit 10111, and the automatic operation control unit 10112 are via the communication network 10121. And are interconnected. The communication network 10121 is, for example, from an in-vehicle communication network or bus compliant with any standard such as CAN (Controller Area Network), LIN (Local Interconnect Network), LAN (Local Area Network), or FlexRay (registered trademark). Become. In addition, each part of the vehicle control system 10100 may be directly connected without going through the communication network 10121.

Hereinafter, when each part of the vehicle control system 10100 communicates via the communication network 10121, the description of the communication network 10121 shall be omitted. For example, when the input unit 10101 and the automatic operation control unit 10112 communicate with each other via the communication network 10121, it is described that the input unit 10101 and the automatic operation control unit 10112 simply communicate with each other.

The input unit 10101 is provided with a device used by the passenger to input various data, instructions, and the like. For example, the input unit 10101 includes an operation device such as a touch panel, a button, a switch, and a lever, and an operation device such as a microphone and a camera, which can be input by a method other than manual operation by voice or gesture. Further, the input unit 10101 may be, for example, a remote control device using infrared rays or other radio waves, or an externally connected device such as a mobile device or a wearable device corresponding to the operation of the vehicle control system 10100. The input unit 10101 generates an input signal based on data, instructions, and the like input by the passenger (for example, the driver) and supplies the input signal to each unit of the vehicle control system 10100.

The data acquisition unit 10102 includes various sensors for acquiring data used for processing of the vehicle control system 10100, and supplies the acquired data to each unit of the vehicle control system 10100.

For example, the data acquisition unit 10102 includes various sensors for detecting the state of the own vehicle and the like. Specifically, for example, the data acquisition unit 10102 includes a gyro sensor, an acceleration sensor, an inertial measurement unit (IMU), an accelerator pedal operation amount, a brake pedal operation amount, a steering wheel steering angle, an engine speed, and the like. It is equipped with a sensor or the like for detecting the rotation speed of the motor, the rotation speed of the wheels, or the like.

Further, for example, the data acquisition unit 10102 is provided with various sensors for detecting information outside the own vehicle. Specifically, for example, the data acquisition unit 10102 includes an image pickup device such as a ToF (Time Of Flight) camera, a stereo camera, a monocular camera, an infrared camera, and other cameras. Further, for example, the data acquisition unit 10102 includes an environment sensor for detecting the weather or the weather, and a surrounding information detection sensor for detecting an object around the own vehicle. The environment sensor includes, for example, a raindrop sensor, a fog sensor, a sunshine sensor, a snow sensor, and the like. Ambient information detection sensors include, for example, ultrasonic sensors, radars, LiDAR (Light Detection and Ringing, Laser Imaging Detection and Ringing), sonar, and the like.

Further, for example, the data acquisition unit 10102 is provided with various sensors for detecting the current position of the own vehicle. Specifically, for example, the data acquisition unit 10102 includes a GNSS receiver or the like that receives a GNSS signal from a GNSS (Global Navigation Satellite System) satellite.

Further, for example, the data acquisition unit 10102 is provided with various sensors for detecting information in the vehicle. Specifically, for example, the data acquisition unit 10102 includes an image pickup device that captures an image of the driver, a biosensor that detects the driver's biological information, a microphone that collects sound in the vehicle interior, and the like. In this case, it is preferable that the image pickup apparatus can image the front surface of the driver's head, the upper body, the lower back and the lower body, and the feet. It is also possible to provide a plurality of image pickup devices so as to image each part. The biosensor is provided on, for example, on the seat surface or the steering wheel, and detects the biometric information of the passenger sitting on the seat or the driver holding the steering wheel.

The communication unit 10103 communicates with the in-vehicle device 10104 and various devices, servers, base stations, etc. outside the vehicle, transmits data supplied from each unit of the vehicle control system 10100, and uses the received data as the vehicle control system. It is supplied to each part of 10100. The communication protocol supported by the communication unit 10103 is not particularly limited, and the communication unit 10103 can also support a plurality of types of communication protocols.

For example, the communication unit 10103 wirelessly communicates with the in-vehicle device 10104 by wireless LAN, Bluetooth (registered trademark), NFC (Near Field Communication), WUSB (Wireless USB), or the like. Further, for example, the communication unit 10103 may be connected to a USB (Universal Serial Bus), HDMI (High-Definition Multimedia Interface) (registered trademark), or MHL (registered trademark) via a connection terminal (and a cable if necessary) (not shown). Wired communication is performed with the in-vehicle device 10104 by Mobile High-definition Link) or the like.

Further, for example, the communication unit 10103 with a device (for example, an application server or a control server) existing on an external network (for example, the Internet, a cloud network, or a business-specific network) via a base station or an access point. Communicate. Further, for example, the communication unit 10103 uses P2P (Peer To Peer) technology to communicate with a terminal (for example, a pedestrian or store terminal, or an MTC (Machine Type Communication) terminal) existing in the vicinity of the own vehicle. Further, for example, the communication unit 10103 performs vehicle-to-vehicle (Vehicle to Vehicle) communication, road-to-vehicle (Vehicle to Infrastructure) communication, vehicle-to-home (Vehicle to Home) communication, and pedestrian-to-vehicle (Vehicle to Home) communication. V2X communication such as Vehicle to Peer-to-Pedestrian) communication. For example, the communication unit 10103 is provided with a beacon receiving unit, receives radio waves or electromagnetic waves transmitted from a radio station or the like installed on the road, and has a current position. Obtain information such as traffic congestion, traffic restrictions, or required time.

The in-vehicle device 10104 includes, for example, a mobile device or a wearable device owned by a passenger, an information device carried in or attached to the own vehicle, a navigation device for searching a route to an arbitrary destination, and the like.

The output control unit 10105 controls the output of various information to the passengers of the own vehicle or the outside of the vehicle. For example, the output control unit 10105 generates an output signal including at least one of visual information (for example, image data) and auditory information (for example, audio data) and supplies it to the output unit 10106 to output the output unit. It controls the output of visual and auditory information from 10106. Specifically, for example, the output control unit 10105 synthesizes image data captured by different image pickup devices of the data acquisition unit 10102 to generate a bird's-eye view image, a panoramic image, or the like, and outputs a signal including the generated image. It is supplied to the output unit 10106. Further, for example, the output control unit 10105 generates voice data including a warning sound or a warning message for dangers such as collision, contact, and entry into a danger zone, and outputs an output signal including the generated voice data to the output unit 10106. Supply.

The output unit 10106 is provided with a device capable of outputting visual information or auditory information to the passengers of the own vehicle or the outside of the vehicle. For example, the output unit 10106 includes a display device, an instrument panel, a HUD (Head Up Display), an audio speaker, headphones, a wearable device such as a spectacle-type display worn by a passenger, a projector, a lamp, and the like. The display device included in the output unit 10106 displays visual information in the driver's field of view, such as a head-up display, a transmissive display, and a device having an AR (Augmented Reality) display function, in addition to the device having a normal display. It may be a display device.

The drive system control unit 10107 controls the drive system system 10108 by generating various control signals and supplying them to the drive system system 10108. Further, the drive system control unit 10107 supplies control signals to each unit other than the drive system system 10108 as necessary, and notifies the control state of the drive system system 10108.

The drive system system 10108 includes various devices related to the drive system of the own vehicle. For example, the drive system system 10108 includes a drive force generator for generating a drive force of an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle, and the like. It is equipped with a braking device that generates braking force, ABS (Antilock Brake System), ESC (Electronic Stability Control), and an electric power steering device.

The body system control unit 10109 controls the body system system 10110 by generating various control signals and supplying them to the body system system 10110. Further, the body system control unit 10109 supplies a control signal to each unit other than the body system 10110 as necessary, and notifies the control state of the body system 10110 and the like.

The body system 10110 is equipped with various body devices equipped on the vehicle body. For example, the body system 10110 includes a keyless entry system, a smart key system, a power window device, a power seat, a steering wheel, an air conditioner, and various lamps (for example, headlamps, back lamps, brake lamps, winkers, fog lamps) and the like. To prepare for.

The storage unit 10111 includes a storage medium for storing data and a controller for controlling reading and writing of data to the storage medium. The storage medium included in the storage unit 10111 includes, for example, a magnetic storage device such as a ROM (Read Only Memory), a RAM (Random Access Memory), and an HDD (Hard Disc Drive), a semiconductor storage device, an optical storage device, and an optical device. One or more of the magnetic storage devices can be applied. The storage unit 10111 stores various programs, data, and the like used by each unit of the vehicle control system 10100. For example, the storage unit 10111 has map data such as a three-dimensional high-precision map such as a dynamic map, a global map which is less accurate than the high-precision map and covers a wide area, and a local map including information around the own vehicle. Remember.

Note that one of the maps stored in the storage unit 10111 is a local dynamic map (hereinafter, LDM). Conceptually, the LDM treats the data to be handled as static data (type 1), quasi-static data (type 2), quasi-dynamic data (type 3), and dynamic data (type 3), depending on the speed of change. It consists of four layers of type 4).

In FIG. 1, the automatic operation control unit 10112 includes a detection unit 10131, a self-position estimation unit 10132, a situation analysis unit 10133, a planning unit 10134, and an operation control unit 10135. The detection unit 10131, the self-position estimation unit 10132, the situation analysis unit 10133, the planning unit 10134, and the operation control unit 10135 are realized by operating a predetermined program on the CPU (Central Processing Unit). Not limited to this, a part or all of the detection unit 10131, the self-position estimation unit 10132, the situation analysis unit 10133, the planning unit 10134, and the operation control unit 10135 may be realized by a hardware circuit that operates in cooperation with each other. Is also possible.

The automatic driving control unit 10112 controls automatic driving such as autonomous driving or driving support. Specifically, for example, the automatic driving control unit 10112 issues collision avoidance or impact mitigation of the own vehicle, follow-up running based on the inter-vehicle distance, vehicle speed maintenance running, collision warning of the own vehicle, lane deviation warning of the own vehicle, and the like. Collision control is performed for the purpose of realizing the functions of ADAS (Advanced Driver Assistance System) including. Further, for example, the automatic driving control unit 10112 performs coordinated control for the purpose of automatic driving that autonomously travels without depending on the operation of the driver.

The detection unit 10131 detects various information necessary for controlling automatic operation. The detection unit 10131 includes an outside information detection unit 10141, an inside information detection unit 10142, and a vehicle state detection unit 10143.

The outside information detection unit 10141 performs detection processing of information outside the own vehicle based on data or signals from each part of the vehicle control system 10100. For example, the vehicle outside information detection unit 10141 performs detection processing, recognition processing, tracking processing, and distance detection processing for an object around the own vehicle. Objects to be detected include, for example, vehicles, people, obstacles, structures, roads, traffic lights, traffic signs, road markings, and the like. Further, for example, the vehicle outside information detection unit 10141 performs detection processing of the environment around the own vehicle. The surrounding environment to be detected includes, for example, weather, temperature, humidity, brightness, road surface condition, and the like.

The vehicle outside information detection unit 10141 uses the self-position estimation unit 10132, the map analysis unit 10151 of the situation analysis unit 10133, the traffic rule recognition unit 10152, the situation recognition unit 10153, and the operation control unit to obtain data indicating the result of the detection process. It is supplied to the emergency situation avoidance unit 10171 and the like of 10135.

The in-vehicle information detection unit 10142 performs in-vehicle information detection processing based on data or signals from each unit of the vehicle control system 10100. For example, the vehicle interior information detection unit 10142 performs driver authentication processing and recognition processing, driver status detection processing, passenger detection processing, vehicle interior environment detection processing, and the like. The state of the driver to be detected includes, for example, physical condition, arousal degree, concentration degree, fatigue degree, line-of-sight direction, and the like. The environment inside the vehicle to be detected includes, for example, temperature, humidity, brightness, odor, and the like. The in-vehicle information detection unit 10142 supplies data indicating the result of the detection process to the situation recognition unit 10153 of the situation analysis unit 10133, the emergency situation avoidance unit 10171 of the operation control unit 10135, and the like.

The vehicle state detection unit 10143 performs detection processing of the state of the own vehicle based on data or signals from each part of the vehicle control system 10100. The states of the vehicle to be detected include, for example, speed, acceleration, steering angle, presence / absence and content of abnormality, driving operation state, power seat position and tilt, door lock state, and other in-vehicle devices. The state etc. are included. The vehicle state detection unit 10143 supplies data indicating the result of the detection process to the situation recognition unit 10153 of the situation analysis unit 10133, the emergency situation avoidance unit 10171 of the operation control unit 10135, and the like.

The self-position estimation unit 10132 estimates the position and attitude of the own vehicle based on data or signals from each unit of the vehicle control system 10100 such as the vehicle exterior information detection unit 10141 and the situation recognition unit 10153 of the situation analysis unit 10133. Perform processing. Further, the self-position estimation unit 10132 generates a local map (hereinafter, referred to as a self-position estimation map) used for self-position estimation, if necessary. The map for self-position estimation is, for example, a highly accurate map using a technique such as SLAM (Simultaneous Localization and Mapping). The self-position estimation unit 10132 supplies data indicating the result of the estimation process to the map analysis unit 10151 of the situation analysis unit 10133, the traffic rule recognition unit 10152, the situation recognition unit 10153, and the like. Further, the self-position estimation unit 10132 stores the self-position estimation map in the storage unit 10111.

The situation analysis unit 10133 analyzes the situation of the own vehicle and the surroundings. The situational analysis unit 10133 includes a map analysis unit 10151, a traffic rule recognition unit 10152, a situational awareness unit 10153, and a situational awareness unit 10154.

The map analysis unit 10151 uses data or signals from each unit of the vehicle control system 10100 such as the self-position estimation unit 10132 and the vehicle exterior information detection unit 10141 as necessary, and stores various maps stored in the storage unit 10111. Perform analysis processing and build a map containing information necessary for automatic operation processing. The map analysis unit 10151 uses the constructed map as a traffic rule recognition unit 10152, a situation recognition unit 10153, a situation prediction unit 10154, and a route planning unit 10161, an action planning unit 10162, an operation planning unit 10163, etc. of the planning unit 10134. Supply to.

The traffic rule recognition unit 10152 determines the traffic rules around the vehicle based on data or signals from each unit of the vehicle control system 10100 such as the self-position estimation unit 10132, the vehicle outside information detection unit 10141, and the map analysis unit 10151. Perform recognition processing. By this recognition process, for example, the position and state of the signal around the own vehicle, the content of the traffic regulation around the own vehicle, the lane in which the vehicle can travel, and the like are recognized. The traffic rule recognition unit 10152 supplies data indicating the result of the recognition process to the situation prediction unit 10154 and the like.

The situation recognition unit 10153 can be used for data or signals from each unit of the vehicle control system 10100 such as the self-position estimation unit 10132, the vehicle exterior information detection unit 10141, the vehicle interior information detection unit 10142, the vehicle state detection unit 10143, and the map analysis unit 10151. Based on this, the situation recognition process related to the own vehicle is performed. For example, the situational awareness unit 10153 performs recognition processing such as the situation of the own vehicle, the situation around the own vehicle, and the situation of the driver of the own vehicle. Further, the situational awareness unit 10153 generates a local map (hereinafter referred to as a situational awareness map) used for recognizing the situation around the own vehicle, if necessary. The situational awareness map is, for example, an occupied grid map (Occupancy Grid Map).

The status of the own vehicle to be recognized includes, for example, the position, posture, movement (for example, speed, acceleration, moving direction, etc.) of the own vehicle, and the presence / absence and content of an abnormality. The surrounding conditions of the vehicle to be recognized include, for example, the type and position of surrounding stationary objects, the type, position and movement of surrounding animals (eg, speed, acceleration, direction of movement, etc.), and the surrounding road. The composition and road surface condition, as well as the surrounding weather, temperature, humidity, brightness, etc. are included. The state of the driver to be recognized includes, for example, physical condition, arousal level, concentration level, fatigue level, eye movement, driving operation, and the like.

The situational awareness unit 10153 supplies data indicating the result of the recognition process (including a situational awareness map, if necessary) to the self-position estimation unit 10132, the situation prediction unit 10154, and the like. Further, the situational awareness unit 10153 stores the situational awareness map in the storage unit 10111.

The situation prediction unit 10154 performs a situation prediction process regarding the own vehicle based on data or signals from each unit of the vehicle control system 10100 such as the map analysis unit 10151, the traffic rule recognition unit 10152, and the situation recognition unit 10153. For example, the situation prediction unit 10154 performs prediction processing such as the situation of the own vehicle, the situation around the own vehicle, and the situation of the driver.

The situation of the own vehicle to be predicted includes, for example, the behavior of the own vehicle, the occurrence of an abnormality, the mileage, and the like. The situation around the own vehicle to be predicted includes, for example, the behavior of the animal body around the own vehicle, the change in the signal state, the change in the environment such as the weather, and the like. The driver's situation to be predicted includes, for example, the driver's behavior and physical condition.

The situation prediction unit 10154, together with the data indicating the result of the prediction process, together with the data from the traffic rule recognition unit 10152 and the situation recognition unit 10153, has the route planning unit 10161, the action planning unit 10162, and the operation planning unit 10163 of the planning unit 10134. And so on.

The planning unit 10134 includes a route planning unit 10161, an action planning unit 162, and an operation planning unit 163.

The route planning unit 10161 plans a route (itinerary) to the destination based on data or signals from each unit of the vehicle control system 10100 such as the map analysis unit 10151 and the situation prediction unit 10154. For example, the route planning unit 10161 sets a route from the current position to the specified destination based on the global map. Further, for example, the route planning unit 10161 appropriately changes the route based on the conditions such as traffic congestion, accidents, traffic restrictions, construction work, and the physical condition of the driver. The route planning unit 10161 supplies data indicating the planned route to the action planning unit 10162 and the like.

The action planning unit 10162 can safely route the route planned by the route planning unit 10161 based on the data or signals from each unit of the vehicle control system 10100 such as the map analysis unit 10151 and the situation prediction unit 10154. Plan your vehicle's actions to drive. For example, the action planning unit 10162 plans starting, stopping, traveling direction (for example, forward, backward, left turn, right turn, turning, etc.), traveling lane, traveling speed, and overtaking. The action planning unit 10162 supplies data indicating the planned behavior of the own vehicle to the operation planning unit 10163 and the like.

The motion planning unit 10163 is an operation of the own vehicle for realizing the action planned by the action planning unit 10162 based on the data or signals from each unit of the vehicle control system 10100 such as the map analysis unit 10151 and the situation prediction unit 10154. Plan. For example, the motion planning unit 10163 plans acceleration, deceleration, traveling track, and the like. The motion planning unit 10163 supplies data indicating the planned operation of the own vehicle to the acceleration / deceleration control unit 10172, the direction control unit 10173, and the like of the motion control unit 10135.

The motion control unit 10135 controls the motion of the own vehicle. The operation control unit 10135 includes an emergency situation avoidance unit 10171, an acceleration / deceleration control unit 10172, and a direction control unit 10173.

Based on the detection results of the outside information detection unit 10141, the inside information detection unit 10142, and the vehicle condition detection unit 10143, the emergency situation avoidance unit 10171 may collide, contact, enter a danger zone, have a driver abnormality, or have a vehicle. Performs emergency detection processing such as abnormalities. When the emergency situation avoidance unit 10171 detects the occurrence of an emergency situation, the emergency situation avoidance unit 10171 plans the operation of the own vehicle to avoid an emergency situation such as a sudden stop or a sharp turn. The emergency situation avoidance unit 10171 supplies data indicating the planned operation of the own vehicle to the acceleration / deceleration control unit 10172, the direction control unit 10173, and the like.

The acceleration / deceleration control unit 10172 performs acceleration / deceleration control for realizing the operation of the own vehicle planned by the motion planning unit 10163 or the emergency situation avoidance unit 10171. For example, the acceleration / deceleration control unit 10172 calculates a control target value of a driving force generator or a braking device for realizing a planned acceleration, deceleration, or sudden stop, and drives a control command indicating the calculated control target value. It is supplied to the system control unit 10107.

The direction control unit 10173 performs direction control for realizing the operation of the own vehicle planned by the motion planning unit 10163 or the emergency situation avoidance unit 10171. For example, the direction control unit 10173 calculates a control target value of the steering mechanism for realizing a traveling track or a sharp turn planned by the motion planning unit 10163 or the emergency situation avoidance unit 10171, and controls to indicate the calculated control target value. The command is supplied to the drive system control unit 10107.

FIG. 2 is a block diagram showing a configuration of an example of an information processing device in which the automatic operation control unit 10112 of FIG. 1 is configured.

In FIG. 2, the information processing apparatus 10000 is connected to each other by a bus 10020 so as to be communicable with each other. It includes an I / F 10014 and a control I / F 10015.

The storage device 10013 is a storage medium that stores data non-volatilely, and a hard disk drive, a flash memory, or the like can be applied. The CPU 10010 uses the RAM 10012 as a work memory according to the programs stored in the storage devices 10013 and the ROM 10011 to control the operation of the information processing device 10000.

The input / output I / F 10014 is an interface that controls the input / output of data to the information processing apparatus 10000. The control I / F 10015 is an interface for a device to be controlled by the information processing apparatus 10000. For example, the input / output I / F 10014 and the control I / F 10015 are connected to the communication network 10121.

For example, by executing the CPU 10010 and the information processing program according to the embodiment, the above-mentioned detection unit 10131, self-position estimation unit 10132, situation analysis unit 10133, planning unit 10134, and operation control unit 10135 are placed on the main storage area in the RAM 10012. Each is configured as, for example, a module.

The information processing program is installed in the information processing device 10000 in advance when, for example, the information processing device 10000 is incorporated in a vehicle and shipped. Not limited to this, the information processing program may be installed in the information processing apparatus 10000 after the information processing apparatus 10000 is incorporated in the vehicle and shipped. Further, the information processing program can be supplied from the input / output I / O I / F 10014 via communication with an external device (server or the like) by the communication unit 10103 and installed in the information processing device 10000.

<< 2. Outline of the level of automatic driving by SAE >>
Next, the automatic driving of the vehicle applied to the embodiment will be described. Regarding the automatic driving of vehicles, the automatic driving level is defined by SAE (Society of Automotive Engineers). Table 1 shows the automated driving levels defined by SAE.

Hereinafter, the automatic operation level defined by SAE shown in Table 1 will be basically referred to and explained. However, in the examination of the automatic driving level shown in Table 1, the issues and validity when the automatic driving technology has become widespread have not been fully examined. Therefore, in the following explanation, based on these issues, etc. There are some parts that are not necessarily explained by the interpretation as defined by SAE.

As shown in Table 1, according to SAE, the automatic driving levels that a person needs to intervene in steering are classified into five stages, for example, from level 0 (Level 0) to level 4 (Level 4). In SAE, automatic driving level 5 (Level 5) is defined assuming only unmanned automatic driving, but in this disclosure, this automatic driving level 5 is because the driver is not involved in steering at all. It is out of scope.

Automatic driving level 0 (Level 0) is manual driving (driver's direct driving steering) without driving assistance by the vehicle control system, in which the driver performs all driving tasks and performs safe driving (for example, danger). Always monitor the actions to be avoided).

Automatic driving level 1 (Level 1) is manual driving (direct driving steering) in which driving support (automatic braking, ACC (Adaptive Cruise Control), LKAS (Lane Keeping Assistant System), etc.) can be executed by the vehicle control system. The driver performs all driving tasks other than the assisted single function and also performs safe driving monitoring.

Automatic driving level 2 (Level 2) is also referred to as "automatic driving function under specific conditions", and under specific conditions, the vehicle control system subtasks the driving task related to vehicle control in both the front-rear direction and the left-right direction of the vehicle. Execute. For example, at the automatic driving level 2, the vehicle control system controls both steering operation and acceleration / deceleration in cooperation with each other (for example, cooperation between ACC and LKAS). However, even in the automatic driving level 2, the execution subject of the driving task is basically the driver, and the monitoring subject related to safe driving is also the driver.

Automatic driving level 3 (Level 3) is also called "conditional automatic driving", and the vehicle control system can execute all driving tasks within a limited area. In the automatic driving level 3, the driving task is executed by the vehicle control system, and the monitoring subject related to safe driving is basically the vehicle control system.

In this automatic driving level 3 defined by SAE, what kind of secondary task the driver can actually perform is not clearly defined. The "secondary task" refers to an operation other than the operation related to driving performed by the driver while driving, and is also called NDRA (Non-driving related activity).

More specifically, while driving at autonomous driving level 3, the driver performs tasks and actions other than steering, such as operating a mobile terminal, telephone conference, watching video, playing games, thinking, and talking with other passengers. It is considered possible to perform secondary tasks such as. On the other hand, within the scope of the definition of SAE's automatic driving level 3, it is appropriate for the driver to perform driving operations in response to requests from the vehicle control system side due to system failures, deterioration of the driving environment, etc. Is expected to be done. Therefore, at the automatic driving level 3, the driver can immediately return to the manual driving even in the situation where the secondary task as described above is being executed in order to ensure safe driving. It is expected that they will always be in a good state of preparation.

Automatic driving level 4 (Level 4) is also called "fully automatic driving under specific conditions", and the vehicle control system executes all driving tasks within a limited area. At the automatic driving level 4, the driving task is executed by the vehicle control system, and the monitoring subject related to safe driving is also the vehicle control system.

However, at the automatic driving level 4, unlike the above-mentioned automatic driving level 3, the driver performs a driving operation (manual driving) in response to a request from the vehicle control system side due to a system failure or the like. No action is expected. Therefore, at the automatic driving level 4, the driver can perform the secondary task as described above, and depending on the situation, for example, can take a nap.

As described above, from the automatic driving level 0 to the automatic driving level 2, the driver runs in the manual driving mode in which all or some of the driving tasks are independently executed. Therefore, at these three levels of autonomous driving, it is permissible for the driver to engage in secondary tasks other than manual driving and related movements that may reduce attention or impair forward attention while driving. It has not been.

On the other hand, at the automatic driving level 3, the vehicle control system runs in the automatic driving mode in which all driving tasks are independently executed. However, as described above, at the automatic driving level 3, there may be a situation in which the driver performs a driving operation. Therefore, at the automatic driving level 3, when the driver is allowed to perform the secondary task, the driver is required to be in a ready state to return to the manual driving from the secondary task.

Furthermore, even at the automatic driving level 4, the vehicle control system will drive in the automatic driving mode in which all driving tasks are executed. Here, in the section to which the automatic driving level 4 is originally applied, there may be a section to which the automatic driving level 4 cannot be applied to a part of the section due to the actual development status of the road infrastructure or the like. Since it is assumed that such a section is set to, for example, the automatic driving level 2 or lower, the driver is required to independently execute the driving task. Therefore, even if the automatic driving by the automatic driving level 4 is being used at the planning stage of the traveling itinerary or after the start of the itinerary, if a situation occurs such that the conditions for which the use is permitted are not met, the automatic driving as described above occurs. A transition request to operation level 2 or lower may occur. Therefore, the driver is required to be ready to return to manual operation from the secondary task depending on the situation, even if it was not planned at the beginning of the itinerary plan, when these changes in conditions are found. Become.

Here, the actual usage range for each automatic driving level allowed for each of these different automatic driving levels is called ODD (Operation Design Domain). More specifically, ODD is a driving environment condition that is a prerequisite for the operation of the automatic driving system by design, and when all the conditions shown in the ODD are satisfied, the automatic driving system operates normally and the vehicle is automatically operated. Driving is done. Further, when the condition shown in the ODD is lacking during driving, it is necessary to transfer the driving control of the vehicle from the automatic driving to the manual driving. The conditions indicated by ODD generally differ depending on each automatic driving system, deterioration and dirt of sensors, etc., and performance fluctuations at that time due to the self-diagnosis result of the on-board device for controlling automatic driving. It will be a thing.

FIG. 3 is a schematic diagram for explaining a case where each automatic operation level of SAE is viewed from the user's point of view as a usage state.

The environment to which automatic driving level 0 (Level 0) can be applied is considered to be general social road infrastructure such as private roads, general roads, and highways. The environment to which the automatic driving level 1 (Level 1) is applicable is a road equipped with a device for driving support and an environment, for example, some arterial roads and highways. In this automatic driving level 1, it is necessary that the driving support system such as ACC and LKAS described above by the vehicle control system is applied to the existing manually driven vehicle. In this case, the driver's attention loss is an intuitive risk because the driver assistance system does not provide comprehensive support.

In addition, the environment to which the automatic driving level 2 (Level 2) can be applied is a fixed road section such as an expressway if the equipment and environment for driving support are in place. In the section to which automatic driving level 2 is applied, in addition to acceleration / deceleration in the driving direction by driving control such as ACC, driving along the lane such as LKAS and lateral control with respect to the driving direction are also automatically performed. It is a generally acceptable classification and requires continued attention to the driver's driving. On the other hand, in the section to which the automatic driving level 2 is applied, the driving itself is established as it is if there is no obstructive factor, and if the driving support becomes too sophisticated, the driver's sense of risk may be lowered. Therefore, it can be said that the automatic driving level 2 is an automatic driving level for which preventive measures for lowering the driver's attention are required.

As described above, in these sections of automatic driving level 0 to 2, the running of the vehicle is controlled by the manual driving of the driver.

On the other hand, the section of automatic driving level 3 (Level 3) and the section of automatic driving level 4 (Level 4) are defined as sections where autonomous driving control is possible by the automatic driving system of the vehicle, as described above. Of these, the environment to which the automatic driving level 4 can be applied may be realized by, for example, securing a section in which the information of each type in LDM is constantly updated and the road predictability is guaranteed.

On the other hand, the environment to which the automatic operation level 3 (Level 3) can be applied is, for example, a section where automatic operation is possible at the automatic operation level 4, but for some reason, the ODD corresponding to the automatic operation level 4 It may be a section that does not satisfy the conditions. For example, it is a section where only quasi-static data can be obtained in LDM, or a section where the running conditions of automatic driving level 4 cannot be continuously secured due to deterioration or lack of environmentally friendly performance of the system. As the section where the automatic operation level 4 cannot be secured, a temporary construction section, a flooded section, a complicated intersection section, an LDM missing section, a communication band temporary shortage section, a section in which a risk report is issued from a vehicle in front, and the like can be considered.

Further, the environment to which the automatic driving level 3 can be applied is a section functionally passable under the control of the automatic driving level 4, but may include a section in which the application of the automatic driving level 4 is canceled for some reason. can. As an example of such a section, a section, a construction section, or a railroad crossing crossing where there is a risk of causing a flow stack that stops the smooth flow of transportation infrastructure when the vehicle is stopped by MRM (Minimum Risk Maneuver) or the emergency evacuation is decelerated. , Etc. are conceivable. Furthermore, automatic driving level 3 can also be applied to the institutional driver-free passage prohibited section (which is subject to penalties for use violated by institutional preventive operation), which is fixedly or actively set. It can be an environment.

In addition, as shown by the black arrow in FIG. 3, automatic driving up to automatic driving level 2 at high speed under the condition that the normal road usage range is smooth from the user's point of view and no traffic jam occurs. Even in sections where only automatic driving by level is allowed, the average vehicle speed will drop due to traffic jams, etc., and if the conditions for temporarily allowing automatic driving by automatic driving level 3 etc. are met, ACSF (Automatically Commanded Steering Function) etc. Operation that enables the use of the automatic driving function is being considered. For example, even in a section where automatic driving is not originally assumed, such as a congested section on a highway to which automatic driving level 2 is applied, automatic driving by automatic driving level 3 or automatic driving level 4 may be possible. In particular, if automatic operation according to automatic operation level 4 is permitted, NDRA can be safely executed. In this case, it is necessary to be able to predict the end point of traffic congestion and manage the operation of returning to manual operation.

Here, as a usage pattern of automatic driving, the vehicle enters the applicable section of automatic driving level 2 from the applicable section of automatic driving level 2, and the driving control of the vehicle is switched from manual driving by the driver to automatic driving by the vehicle control system. Think about the case. In this case, the driver does not have to concentrate on driving the vehicle, and attention maintenance is reduced. That is, it can be said that switching from manual driving to automatic driving is a usage mode that causes a decrease in the continuous attention maintenance of the driver.

In addition, considering the switching from automatic driving level 4 to automatic driving level 2 (return to manual driving), the driver's attention maintenance is reduced in the section where automatic driving is performed by automatic driving level 4, and the driver's attention maintenance is reduced in advance. It can be said that this is a usage area where it is required to formulate the details of the time budget from the driver's monitoring information in the steady state to the return. In the existing technology, at the automatic driving level 4, the driver is uniquely notified of the return to the automatic driving level of the automatic driving level 2 or lower, that is, the manual driving.

The role of the function corresponding to the automatic driving level 3 is to avoid the division between the section where the automatic driving is performed by the automatic driving level 4 and the section where the manual driving is performed by the automatic driving level of the automatic driving level 2 or lower. It can be said that it is the role of the connection. That is, the usage mode of automatic driving according to the automatic driving level 3 is a usage mode in which the driver continues to maintain attention to driving and is expected to return in a short time (for example, several seconds). In this automatic driving level 3, the DMS (Driver Monitoring System) that monitors the driver detects the decrease in the driver's attention maintenance and the continuation of the driver's attention maintenance is the automatic driving level of 3 or less. It is an essential requirement to use it.

FIG. 4 is a schematic diagram for schematically explaining the application of the automatic operation level 3. In the map shown in FIG. 4, a case is shown in which the vehicle travels from the start point ST to the end point EP in the direction indicated by the arrow (counterclockwise, counterclockwise) in the figure according to the itinerary TP (filled in the figure). There is.

In FIG. 4, sections RA ₁ , RA ₂ and RA ₃ indicate sections corresponding to, for example, automatic operation levels 0 to 2, and manual operation is indispensable in these sections. When the vehicle's automatic driving system enters these sections RA ₁ to RA ₃ while driving, for example, by automatic driving according to automatic driving level 4, the driving control is changed from automatic driving by the automatic driving system to manual driving by the driver's steering. It is necessary to take over to driving. On the other hand, sections RB ₁ to RB ₅ indicate sections in which passing driving can be performed while the automatic driving is performed under the caution monitoring of the posture for returning from the automatic driving to the manual driving. Sections RB ₁ to RB ₅ are sections corresponding to, for example, automatic operation level 3.

In order to enter each section RA ₁ , RA ₂ and RA ₃ where manual driving is essential, it is necessary for the driver to be in a position to return from automatic driving to manual driving. Therefore, for example, each section RB ₁ , RB ₄ and RB ₅ corresponding to the automatic operation level 3 are set on the approach side of each section RA ₁ , RA ₂ and RA ₃ , respectively.

On the other hand, the sections RB ₂ and RB ₃ are functionally passable sections under the control of the automatic driving level 4, but are set as sections for canceling the application of the automatic driving level 4 for some reason. Section RB ₂ is, for example, a temporary construction section or a flooded section, and section RB ₃ is a section that requires attention to the running of the own vehicle due to, for example, a sharp curve.

In this way, if a return to manual operation occurs while driving at automatic operation level 4, if there is time to reach the relevant location where the return request is issued from the system and the transfer is required to be completed. Before that, the vehicle goes through the section of automatic driving level 3 or the state where the driver's driving ability is awakened and attention to the surrounding area is possible and the driving ability is restored. At autonomous driving level 3, the driver is not directly involved in driving, but it is necessary to maintain attention to the situation, so it is required to wait only with caution without actual driving steering for a long time. In operation, the driver may feel distressed.

<< 3. Embodiments of the present disclosure >>
Next, an embodiment according to the present disclosure will be described. In the following, unless otherwise specified, the ODD is an ODD corresponding to the automatic operation level 4 and indicates a condition for the automatic operation level 4. Further, a traveling section satisfying the conditions indicated by ODD is simply referred to as an ODD section.

<3-1. Outline of embodiment>
First, the outline of the embodiment will be described in comparison with the existing technology. FIG. 5 is a flowchart of an example schematically showing a transfer process from automatic operation to manual operation by the existing technology. Prior to the start of the process according to the flowchart of FIG. 5, it is assumed that the vehicle on which the driver is boarding is traveling in the ODD section corresponding to the automatic driving level 4.

When the end point of the ODD section is approaching, the automatic driving system mounted on the vehicle notifies the driver in step S10 that the end point of the ODD is approaching. The driver prepares to take over the operation control from the automatic operation to the manual operation, for example, in response to this notification. The automatic driving system issues an alarm to the driver when the transfer of the driving control from the automatic driving to the manual driving is delayed more than a predetermined time (step S11).

In step S12, the automatic operation system determines whether or not the transfer of the operation control from the automatic operation to the manual operation has been performed within a predetermined time after notifying as the ODD end notice point in step S10. When the driver determines that the transfer is completed within a predetermined time (step S12, "OK"), the automatic operation system shifts the process to step S13 and evaluates the transfer completion.

On the other hand, when the automatic operation system determines in step S12 that the transfer of the operation control from the automatic operation to the manual operation is not completed within a predetermined time (step S12, "NG"), the process shifts to step S14. .. In step S14, the autonomous driving system applies the MRM to the control of the vehicle and causes an evacuation run, for example, an emergency stop on the shoulder.

Here, the concept of automatic driving control of a vehicle by the existing technology is that the level at which the vehicle can run in the automatic driving level classification of SAE is the range of the design assumption of the equipment mounted on the vehicle as ODD. It will be decided. The driver is required to always follow all the requirements and deal with all the requirements according to the level of autonomous driving that the vehicle can drive autonomously.

For example, it is assumed that a specific highway allows automatic driving of automatic driving level 4, and the automatic driving performance of the on-board equipment of the vehicle allows automatic driving equivalent to automatic driving level 4. In this case, the driver can use and drive the automatic driving level of the vehicle as level 4 in the section. When the vehicle approaches a situation outside the ODD section where driving at automatic driving level 4 is possible, the automatic driving system urges the driver to return to manual driving (FIG. 5, step S10), and the response is delayed. If so, a warning is simply issued (FIG. 5, step S11). If the automatic driving system does not return to the manual driving at an appropriate timing even though the warning is issued in step S11, the automatic driving system is urgent in the ODD section where the driving can be performed by the automatic driving of the automatic driving level 4. By shifting to the forced evacuation steering, so-called MRM control (FIG. 5, step S14), it is assumed that the system will prevent entry into a section that cannot be dealt with by automatic driving.

In vehicle control using such existing technology, depending on the performance limit of the automatic driving system, if the automatic driving system is in a section where driving at the automatic driving level 4 is permitted, the driver can specify 2 other than driving. It is expected to engage in the next task (NDRA). On the other hand, when the automatic driving system reaches its own coping limit, it issues a compulsory return request from automatic driving to manual driving to the driver. From the user's point of view, this will force a forced return from the engagement of the secondary task.

In this way, when using the automatic driving system of the vehicle by the existing technology, the driver is forced to have a subordinate relationship with the automatic driving system. Therefore, for the driver, the engagement of the secondary task using the automatic driving function has been a stressful use or control form.

FIG. 6 is a flowchart of an example schematically showing the transfer process from the automatic operation to the manual operation according to the embodiment. Prior to the start of the process according to the flowchart of FIG. 6, it is assumed that the vehicle on which the driver is boarding is traveling in the ODD section corresponding to the automatic driving level 4.

In step S20, the automatic driving system notifies the driver in advance of the end of the ODD section. For example, the autonomous driving system notifies the driver of the end of ODD at a time earlier than the time expected to be required from the notification to the return of the manual driving.

In the next step S21, a "contract" regarding the transfer start point is signed between the automatic driving system and the driver. The "contract" here refers to a series of flows in which the driver explicitly responds to the notification issued by the automatic driving system. At this time, the automatic driving system presents to the driver information indicating the points where the transfer must be completed and the risk when the transfer is not completed. The "contract" here is to share information about the transfer between the autonomous driving system and the driver, and does not impose any obligation on the driver, so it is actually a "provisional contract". It should be called.

In this way, by concluding a temporary contract regarding the transfer start point between the automatic driving system and the driver by the driver's explicit response, the transfer work can be imprinted on the driver's working memory. Working memory, also called working memory, refers to the memory capacity of the human brain that temporarily stores and processes information necessary for work and operation.

In the next step S22, the automatic driving system manages the process of handing over to manual driving by the driver. For example, the automatic driving system monitors the driver's condition, and based on the monitoring result and the vehicle condition at that time, in step S22, the margin from the current time (local point) to the transfer start point and the transfer start. Judge whether or not the grace period can be extended. The automatic driving system performs further notification and control such as transition to MRM according to the determination result.

The "margin" mentioned here is a vehicle around the road on which the vehicle is traveling compared to the time estimated for the driver to return to driving from the driver's status analysis detected through continuous passive monitoring. It is a time that can be secured longer than the time required to reach the takeover completion limit point when traveling at the cruising speed estimated from the flow of. In addition, the above-mentioned "extension of the grace time until the start of takeover" means, for example, deceleration of the traveling speed, shoulder, temporary evacuation, or low-speed traveling lane without obstructing the flow of the surrounding cruising vehicle. It refers to extending the time to reach the transfer completion limit point by moving, etc.

In the next step S23, the automatic operation system determines whether or not the transfer of the operation control from the automatic operation to the manual operation is performed within a predetermined time set in, for example, the transfer process management in step S22. When the automatic driving system determines that the takeover has not been completed within a predetermined time, the vehicle is evacuated and traveled in the same manner as in step S14 of FIG. 5, and an emergency stop or the like by MRM is executed.

In the next step S24, the automatic driving system evaluates the completion of handing over to manual driving by the driver. At this time, the automatic driving system calculates the evaluation points according to the response of the driver involved in the takeover. For example, the automatic driving system adds evaluation points to operations that are preferable for taking over, such as when the takeover process is voluntarily performed by the driver or when a provisional contract is executed. In addition, even if the driver gives up taking over in advance and chooses a break, etc., it is equivalent to selecting a means to prevent the driving of surrounding vehicles, which is a suitable process as an impact on social infrastructure, and is an evaluation point. Is added. On the other hand, in the automatic driving system, if the takeover process is started in response to repeated warnings, or if the takeover process is started after an imminent situation occurs, the risk of the vehicle controlling by MRM and failing to take over increases. For operations that are not desirable for taking over, etc., evaluation points are deducted according to the degree of influence.

In the next step S25, the automatic driving system gives an insensitive or a penalty to the driver according to the evaluation points calculated in the step S24. As an example, when the automatic driving system adjusts the evaluation points based on the evaluation points = 0, for example, if the evaluation points calculated in step S24 exceed 0, the automatic driving system gives the driver insensitivity. .. On the other hand, if the evaluation score calculated in step S24 is lower than 0, the automatic driving system imposes a penalty on the driver. The lower the rating, the heavier the penalty. Penalties may be, for example, restrictions on the use of the driver for automatic driving, restrictions on the driver's engagement in secondary tasks, and the like.

In this way, by giving the driver an incentive or a penalty according to the evaluation of the takeover process, the risk to the driver's working memory of the takeover work is included in the provisional contract in step S21 described above. Imprinting is possible (step S26).

In the embodiment of the present disclosure, the automatic driving system constantly monitors and observes the driver's condition, and evaluates the degree of possibility of the driver returning to manual driving. The automatic driving system gives the driver a notice of advance return request necessary for returning to manual driving while the vehicle is being controlled by steering by automatic driving so that the return to manual driving can be performed properly without delay. Information is always presented to the driver. Prior to the actual return start timing, the automatic operation system concludes a "contract" with the driver regarding the determination of an appropriate return start timing, and manages the takeover process for the return start timing.

In the embodiment of the present disclosure, in this way, human-centered interactive control of automatic driving is performed in order to realize a smooth return to manual driving with few failures, thereby aiming at comfortable use of automatic driving. That is, in the embodiment of the present disclosure, the automatic driving system unilaterally notifies the transition demand, which is a request for returning to manual driving, based only on the status in which the vehicle is placed, and prompts the driver to return. Instead, we aim for takeover control in which the system and the driver cooperate, sharing prior knowledge of takeover with the driver and working on the driver's memory.

That is, the automatic driving system according to the embodiment of the present disclosure informs the driver of each takeover start point at a time earlier than the predicted time required for returning to manual operation after the driver is notified of the takeover start point. Notify (step S20 in FIG. 6). Then, the driver makes a "temporary contract" with the automatic driving system regarding the actual takeover start point (step S21 in FIG. 6). Based on the provisional contract, the autonomous driving system manages the takeover process, budgets the takeover sequence, and distributes the risk (step S22 in FIG. 6). As a result, the driver can appropriately prepare for the end of the secondary task and grasp the situation necessary for manual driving in advance (for example, grasp the surrounding environment necessary for driving control by manual driving).

The "contract" (FIG. 6, step S21) that the system makes with the driver in the embodiment of the present disclosure can be described later in the visual cortex memory of the driver by devising a method of providing the "contract" (FIG. 6, step S21). It also has the role of keeping that sense of time in memory.

By concluding the advance contracts between the autonomous driving system and the driver necessary for the transfer from automatic driving to manual driving, the autonomous driving system can "reliably" convey the importance of the transfer to the driver. Can be realized. The automatic driving system informs the driver in advance of the transmission of the required point for completion of the transfer and the risk when the transfer is incomplete, so that the judgment information related to the driver's behavior judgment is "notified" in the working memory. It can be reliably captured as "information". As a result, unlike the situation in which the driver starts grasping the situation after receiving the notification by the existing technology, a certain amount of notice information to be noted in advance is once taken into the memory, and as a result, the driver inadvertently makes a mistake in the decision to start taking over. That can be prevented or at least reduced.

Further, according to the embodiment of the present disclosure, even if there is a risk of oversight due to an error in the decision to start taking over due to dementia or dementia of the driver, the driver's repeated return habits are monitored each time. However, it is possible to detect signs of such errors. Therefore, according to the automatic driving system according to the embodiment, the effect based on this prior "contract" (that is, the execution in which the return work is expected) is reduced, and as a side effect, dementia such as elderly people and the like It can also be used to grasp signs of dementia and the like.

<3-2. About HCD (Human Centered Design) according to the embodiment>
The automatic driving system according to the embodiment is centered on people (drivers, etc.) instead of MCD (Machine Centered Design), which is a design concept centered on devices and systems generally used in existing systems. HCD (Human Centered Design), which is a design concept, is applied. In other words, the automatic driving system according to the embodiment performs cooperative control that incorporates human behavior characteristics into vehicle control.

<3-2-1. Outline of HCD according to the embodiment>
First, the outline of the HCD according to the embodiment will be described. In the existing MCD, the automatic driving system mechanically determines the ODD that can be used for automatic driving according to the performance of the on-board equipment mounted on the vehicle, and uniquely permits the use of the automatic driving function within the limited range. do. In this case, even if the user makes excessive dependence on the automatic driving function, the control performed by the system on the user is a one-sided notification of a control instruction, an alarm, or a case where it is impossible to respond. Only in MRM (Minimal Risk Maneuver).

On the other hand, in the HCD according to the present disclosure, the availability of the automatic driving function by the user when using the automatic driving is controlled so that the use proceeds within a socially acceptable range. That is, in the control of availability, the characteristics obtained from the behavioral habits of the driver are taken into consideration, and if there is an appropriate behavioral habit, the use is permitted. On the other hand, for inappropriate behavioral habits (not responding to requests for returning to manual driving, delaying returning to manual driving, deterioration of quality of returning behavior, etc.), HMI (HMI) that encourages users to change their behavior. Human-Machine Interface) is adopted, and the usable range of the automatic driving function is actively applied to individuals according to the degree of adaptation.

It is difficult to realize such HCD by simply introducing a function, and it is necessary to provide multi-step hierarchical, multi-dimensional and dynamic information feedback that promotes changes in human usage behavior. In this disclosure, the concept of "contract" is adopted, and feedback of information for embodying HCD is provided by a contract between the system and a person (driver).

Specifically, it is as follows.
-Conclude a "contract" between the system and the driver when starting to use each allowable section of the automatic driving function.
-Before reaching the end point of the section used by the contract, the system and the driver confirm the "incidental contract" that completes the transfer to prompt and safe manual driving without sudden stop or slowing down. ..
-Fulfillment by the driver of the incidental obligation regarding reconfirmation of the change status with the passage of usage time.
-Give credit to the driver as an individual effectiveness evaluation of the incidental obligation in the "contract" that accompanies the repeated use of autonomous driving (driver credit).
-Redefine the ODD of the allowable range of automatic driving on the road currently in use, taking into account the personal characteristics of the driver's return from the evaluation history of performance regarding the obligation to return to manual driving in the past, that is, the driver's credit. ..

The system provides these as visual information to the driver, and also provides changes after the start of each permitted section (as ancillary contracts) as visual information for confirming changes in the situation. For example, if the condition of the end point of automatic driving is different from the condition at the start of use, the system will be a factor to end the use of automatic driving and a risk if it does not correspond to the end of automatic driving. Visual feedback to the driver is given using the visual information that describes. Based on the information presented to the driver through the HMI, the carrier disregards the return request as a behavioral judgment psychology because the information material of the risk judgment according to the situation that disregards the return behavior is reflected in the memory. Being affected, the memory becomes clearer.

By the way, the memory that remains in the working memory declines with the passage of time. In particular, if NDRA (Non-driving Related Activity) such as watching TV broadcasts continues in a situation where there is no urgent need to deal with manual driving, the transfer to manual driving will not be the main concern during that time. .. Therefore, it is the information that is processed in the subconscious by the activity of the brain that occurs outside the consciousness, which is responsible for the re-recognition of the necessary event. Therefore, it is important to have an HMI that incorporates risk memory into the subliminal consciousness of the brain by a subliminal method or the like and restores interest in taking over driving.

In addition, the system may request the driver to point and call for confirming the direction of travel and evaluate the result, forcibly looking ahead and urging the driver to confirm.

In order to realize the control based on this HCD, it is necessary to continuously reconstruct the memory necessary for recovery according to the individual difference of the person and the memory ability in the working memory of the situation. The problem here is that human memory is not information that can be directly observed from the outside. Therefore, in the present disclosure, the system realizes the continuous reconstruction of the memory necessary for restoration by the "contract" made with the driver.

<3-2-2. About the superiority of HCD in autonomous driving>
Next, the superiority of the HCD according to the present disclosure in automatic driving will be described.

The social introduction of autonomous driving technology has a great impact on long-term users and how they are involved, depending on the introduction procedure. In order for the introduction of autonomous driving technology to be successfully introduced without any socially adverse side effects, it is necessary to appropriately suppress the harmful effects of autonomous driving and proceed with the introduction. That is, when the functions made possible by technological development are provided indefinitely without considering human behavioral psychology, the user does not necessarily use the technology to the extent that it is socially acceptable.

The existing concept of social introduction of autonomous driving is to gradually introduce the autonomous driving function according to the achievement level of the technology (SAE autonomous driving level, etc.), that is, the function that can be automatically performed according to the result of technological development. The purpose is to gradually expand the use from the range and promote the introduction to society. In other words, the concept of social introduction of existing autonomous driving is based on the results of technological development, that is, according to the achievement performance of machine development, the autonomous driving function defined by SAE etc. at autonomous driving levels 2 to 4. It refers to the idea of gradually promoting the social introduction of autonomous driving at the performance stage.

On the other hand, in the technology according to the present disclosure, it depends on the ability of the driver to adapt to the device using the automatic driving function, that is, whether the driver has a passive ability to appropriately use the technology. Therefore, the function of the provided automatic driving is dynamically changed, and the allowable degree of the automatic driving control is dynamically controlled.

That is, in the present disclosure, even if the mechanical and functional configurations of the automatic driving are exactly the same, the actual provision of the automatic driving function is behavioral suitability so that the driver who is the user can safely use the function. Dynamically change the automatic driving function that the user can actually use, depending on whether or not the vehicle is equipped with. The present disclosure relates to a technique for applying such an idea of HCD to the control operation of an automatic driving system.

<3--2-2-1. About overdependence>
In order to use the appropriate automatic driving function, it is necessary for the driver not to be overly dependent on the automatic driving function. The simplest case of excessive reliance on autonomous driving functions is, for example, the driver neglecting the forward attention and surrounding monitoring obligations necessary for driving, even though the design clearly assumes that the driver is involved in control. There are cases. In this case, the driver's attention to the vehicle in front may be reduced, and the driver may be inattentive to the inter-vehicle distance. That is, while using an automatic driving function limited to functions such as a lane keeping support system, other vehicles or obstacles that interfere with the own vehicle before and after the own vehicle even though it is a limited auxiliary function. If they are not close to each other, it may happen that the driver depends on this automatic driving function. If the driver feels reassured by these types of support, it may lead to a decrease in driving attention, which may lead to delays in judgment and excessive avoidance behavior in emergency coping behavior.

When such a situation related to a decrease in attention occurs, the driver may fall into a situation such as a delay in dealing with an emergency situation, an excessive avoidance action, or an inability to deal with it. There is also a risk that the deceleration or rushed operation will cause secondary damage such as a rear-end collision with a following vehicle or the occurrence of traffic congestion. If it becomes possible to automatically handle steering control even in more complex and complicated situations such as automatic driving of automatic driving level 2, the frequency with which the driver is directly involved in steering is reduced. The driver feels reassured by these driving support functions, and the necessary attention tends to be neglected even though continuous caution is required in case the driver originally needs to take measures.

When the introduction of autonomous driving higher than the autonomous driving level 2 is introduced, the driver is not always required to pay attention to the front, so this problem becomes more complicated. If it is judged that it is dangerous to continue driving with the system under automatic control, the limit of the situation judgment coping ability of the on-board equipment is exceeded, if the vehicle is running at automatic driving level 4, the system should continue automatic driving. It is necessary to give up and start the procedure to have the driver take over the driving properly. Alternatively, if it is difficult for the driver to take over due to an urgent sudden event, it is necessary to automatically start the risk minimization procedure. In this case, the driver does not promptly start the return to the manual operation according to the instruction from the system, and neglects the measures such as the return to the manual operation to allow the system to reach the limit point. In order to delay, the vehicle may be decelerated against the traveling speed of surrounding vehicles, MRM may be started, or the system may lead to a situation where accident prevention measures are taken. This is a good example of overdependence in autonomous driving.

Similarly, even in autonomous driving level 2 and autonomous driving level 3, the advanced support allows the system to be able to operate under many driving conditions without frequent steering intervention by the driver. You will experience the appropriate and safe implementation of event handling. The driver is content with the situation in which the system deals with the event, as it is possible to complete the itinerary without a direct decrease in attention when the vehicle is running, which is not directly linked to the sense of risk. And the driver may become accustomed to the dependent use of autonomous driving, and the driver's skepticism about system imperfections may diminish.

In the case of automatic driving level 2 or automatic driving level 3, the driver is required to take immediate action in an abnormal situation, so a decrease in attention is not allowed, whereas in automatic driving level 4, this caution is not allowed. It can fall into a decline. Furthermore, while this automatic driving level 2 and automatic driving level 3 require continuous attention to the driver, in reality, for example, from an ergonomic point of view, the driver can always fulfill this duty of care. Is not guaranteed.

In other words, it is a situation in which the user is unilaterally forced to understand and use the individual design limits correctly according to the performance that the design function of the machine can achieve in the development, and the design performance is affected. If the technology is introduced into society on the assumption that the user will be able to use the technology as expected, the problem of overdependence on the technology remains. In general, human psychology is skeptical of new and unknown technologies and unconsciously takes preventive measures to deal with them. However, as the development and spread of autonomous driving progresses, the functions of autonomous driving become more sophisticated and diversified, and anxiety and skepticism in using it gradually decrease, and this problem of overdependence becomes even greater and can become a problem.

This disclosure does not deal with this essential problem as a system that prevents the user's attention loss, etc. by simply considering it as a problem of lowering consciousness or attention (warning, return to awakening, etc.), but by the user. It relates to the technology necessary for introducing a series of mechanisms necessary for behavioral psychology to naturally self-learn and adapt to the limit performance of this automatic driving. In other words, in this disclosure, a series of controls necessary to encourage the user to improve / change the behavior that gradually changes the repeated use behavior by the user himself / herself are performed, and the hierarchy is used to promote the improvement of the use behavior. It provides a mechanism that acts on the driver with a modified mechanism.

<3-2-2-2. About HCD>
The point of this disclosure is to change the relationship between the vehicle and the driver from the existing MCD to the HCD, and determine the operating area of the system depending on how the person behaves. Then, the determined operating area influences the benefits that the user can obtain when using the vehicle, depending on the behavior routine of the person. The system is weighted so that the user can feel comfortable and the feedback loop is not unintentionally disturbed (traffic jam, rear-end collision, road blockage, etc.). I do. The present disclosure relates to an HMI in which the system is effective in maintaining such weighted system control and fostering human behavioral habits in a virtuous cycle.

In other words, users are required to move away from the simple dependence on the functions provided by the use of vehicles equipped with existing autonomous driving functions, and to change their behavior in a coordinated manner. In order to create this change in usage behavior, a mechanism to create that change is necessary. This disclosure proposes the mechanism of each element that produces the behavior change of this cooperative use and the technology related to its overall operation.

In order to enable this usage pattern of HCD, behavior change of the user is indispensable, and HMI for producing the behavior change is also required. HCD is not just a mechanism that allows users to use functions as they desire, but a mechanism that is expressed to encourage users to take necessary coping actions naturally in order to use the functions comfortably. The whole. Human behavior is not designed to allow the use of animal instincts in humans as desired, but to adhere to (or to be followed) the rules required to maintain social order in modern society. It can be considered as a functional design required for that purpose after redefining it with a design that incorporates the necessary spontaneous behavior and a mechanism that involves behavior change.

I will explain more specifically. First, when introducing autonomous driving into society, the availability will change depending on the conditions, but it is a prerequisite that the system has a function to automatically control the steering of the vehicle.

The function of the vehicle to acquire information from the outside, supplement the information, grasp the environment, plan the driving of the own vehicle, and proceed with the planning is required at the minimum. In addition, if the system cannot confirm the guarantee that this series of processes can be executed under all conditions, it depends on whether the driver is required to return to manual operation promptly or not. Therefore, automatic driving of automatic driving level 3 or automatic driving level 4 exceeding the automatic driving level 2 is permitted.

Furthermore, for example, in automatic driving of automatic driving level 4, there is a possibility that one or more automatic driving ends will occur before the vehicle in automatic driving reaches the end point of the itinerary driving. In this case, inevitably, the transfer sequence to the manual operation at the end of the automatic operation will be included in the itinerary multiple times.

Here, SAE defines automatic operation level 5, which is more sophisticated than automatic operation level 4. Autonomous driving level 5 is operated after starting in a closed environment or investing a large amount of infrastructure in acquiring environment and environmental information and preparing an LDM that updates information with higher definition and higher refresh rate than the surroundings, for example, a robot. Applies to taxis, etc. Unless it is operated like a robot taxi at this automatic driving level 5, in a vehicle that enables the use of automatic driving level 4 for general users, the user who will be the driver will be asked to start automatic driving during the itinerary. There will be various demands for returning to manual operation.

The requirements for the automatic driving function of a vehicle are determined by the achievement limits that can be dealt with by the design and development of the vehicle. In this case, it is equipped with more cost, such as a large amount of information acquisition resources that can be given to the optimum processing, a large amount of resources for autonomous or external acquisition and income, and power and cost resources that can be given to operations. It is possible to extend the limit of autonomous driving by constructing and improving the infrastructure. On the other hand, although the frequency of requests for return and the available range in which automatic operation can be used are different, it is extremely difficult to eliminate the situation where return from automatic operation to manual operation is required. Therefore, an HCD that realizes appropriate driver intervention in response to these takeover requests that occur when using a vehicle is required in place of the existing MCD. It was

When the system configuration is changed from MCD that depends on the performance of the device to HCD that emphasizes cooperation with humans, the user is encouraged to use it appropriately without overdependence. For that purpose, the system allows the driver to self-learn the balance between the benefits obtained by using autonomous driving for the user and the loss and risk incurred to enjoy the benefits, and the user can use it comfortably. It is necessary to have a mechanism to bring out the benefits in the balance while bearing the necessary obligations.

<3-2-2-3. Benefits for drivers>
Then, from the viewpoint of what the benefits for vehicle users will be, as shown below, the benefits to be aimed at will be one or more complex.

First, examples of actions to obtain benefits are described below.
(1) Purely achieve the movement from the starting point A to the target point B.
(2) Achieve the movement of the two points with comfortable movement.
(3) Achieve movement with a lower budget.
(4) Achieve movement in a shorter time.
(5) Achieve the move at the scheduled time.
(6) Achieve movement with less fatigue.
(7) Achieve the desired movement even in a state of exhaustion or poor physical condition.
(8) Leave the site anyway.
(9) Achieve the purpose of transporting necessary goods.
(10) As entertainment, drive outdoors in good weather or in a scenic view.
(11) During the movement, it is possible to appropriately engage in work other than driving (secondary task, that is, NDRA).

Examples of the NDRA executed during the movement in the above item (11) include the following.
(11-1) Eating and drinking (11-2) Browsing using mobile terminals, etc. (11-3) Email texting (11-4) Implementation of teleconference (11-5) Execution of leaving seats, sorting deliveries, etc. (11-6) Make up and adjust your personal taste.
(11-7) Execution of karaoke, watching movies, watching sports TV broadcasts, etc. (11-8) Operation of terminal devices such as smartphones, mobile phones, tablet computers, notebook computers, etc. (11-9) On the move Appreciating the scenery (11-10) Checking the contents of bags, searching for lost items (11-11) Conversations and exchanges with attendees, games such as crosswords (11-12) Other e-sports (11-13) Congestion Benefits of using automatic driving to reduce the burden only at times (11-14) Primary measures for temporary poor physical condition (foot swelling, etc.) (11-15) Temporary deterioration of eyesight (11-16) Eye drops Use of eye drops, support for temporary deterioration of eyesight due to eye drops (11-17) Response to seizures of asthma and epilepsy (11-18) If you are driving in a section where you can drive safely at automatic driving level 4, during that period Temporary nap (11-19) Implementation of undefined processing

Further examples of actions to gain benefits include:
(12) The function of automatic driving can be used continuously as much as possible (13) There is no usage loss due to the use (14) It does not bother interested parties.

As mentioned above, if there is a need to return from automatic driving to manual driving while driving using automatic driving, the driver cannot properly return to manual driving when a return request is issued by the system. In addition, it becomes necessary to perform emergency deceleration and evacuation stop as MRM. Therefore, there is a risk that the negative aspects of autonomous driving, such as the interruption of social people and distribution arterial roads, traffic congestion, and the induction of rear-end collisions, will be exposed.

From the individual user's point of view, the secondary losses to these emergency controls only affect the following vehicle, except in the case of a rear-end collision with their own vehicle. It is not considered to be a factor that impairs the benefits of encouraging one's proper reinstatement coping behavior. In other words, if the use of autonomous driving is simply entrusted to the driver's good sense from the perspective of HCD, there is a risk that social order cannot be maintained.

Therefore, in order to maintain social order and promote appropriate user behavior while introducing HCD control, for example, it is requested while obtaining the benefits listed in the above-mentioned items (1) to (14). It is necessary to have a mechanism to motivate the return without delay. However, moral motivation is only an idea, and it is appropriate to educate drivers who use autonomous driving with the expectation of moral behavior without ignoring return requests or responding to delays. The effectiveness of the return is not always accompanied.

In order to elicit a behavior change for the driver to take an appropriate and prompt response to the system return request, he / she bears the risk of neglecting the return request instead of gaining benefits, and the balance is that the return request is driving. It needs to act directly on the behavioral psychology of the person. In other words, there needs to be a mechanism in which effective risk aspect input works for the driver in some way. This is because the behavior is determined by the balance between the merits and benefits and the risks involved.

For example, Japanese Patent Application Laid-Open No. 2019-026247 and Japanese Patent Application Laid-Open No. 2016-153960 are known as publicly known examples that microscopically promote a return action in response to a return request. Japanese Patent Application Laid-Open No. 2019-026247 discloses a technique of blowing cold air to a driver in order to maintain the driver's awakening. Further, Japanese Patent Application Laid-Open No. 2016-153960 discloses a technique for giving awakening notification to a driver step by step by using an alarm. These publicly known examples were not mechanisms for promoting macroscopic behavioral changes in the use and usage of the driver's automatic driving function.

For example, in order to obtain the benefit of arriving early, in balance with the disadvantage of paying the usage fee when using the highway, do not use the highway when the usage fee is high, but only when the usage fee is low. Examples such as using are thoughtful balances. In addition, if a sports broadcast is watched as NDRA during automatic driving and the watch is continued without interruption immediately after receiving a transfer request, a penalty can be given to the driver as a demerit to the driver. In this case, as a penalty, it is conceivable to give a certain period of viewing authority of the broadcast or prohibition of viewing on the same day, prohibition of automatic driving reuse for a certain period such as forced stop of the vehicle evacuation space, and the disadvantage for the user. It will be different for each person.

<3-2-2-4. About the driver's working memory and thinking when driving>
Here, in order not to cause the collapse of social infrastructure by controlling HCD, whatever the advantages and disadvantages for individual users, it will eventually result in human behavior, and the return limit point from the time when the transfer request is made. By then, it is required to take over to smooth and high-quality manual driving without lowering the basic cruising speed.

Here, the quality of the return (takeover) action will be explained. Since there are individual differences in driver behavior for each driver, the system learns the normal takeover behavior of the target driver, and based on the driver behavior learned from that behavior, the driver concerned. Estimate the time required for the return of. The quality of the return action is that the driver promptly executes the return action in response to the return request from the system and completes the return action in time, or gives a notification that the return is completed in time. Behavior quality such as delaying the start of return and taking actions that are not seen in normal learned return behavior due to slow movement, etc., without the driver taking the return behavior expected for normal return after learning. The overall behavioral evaluation indexed from the evaluation is shown.

In other words, it is necessary to grasp the control necessary for this user's return from the HCD viewpoint from the next viewpoint.

It cannot be expected that the user will make an action decision after understanding the detailed situation of the invisible ODD, which is determined by the automatic driving system realized by the advanced and precise design such as the performance of the equipment and the vehicle. Therefore, the autonomous driving system needs a mechanism to present to the user as a descriptive and embodied risk that allows the user to intuitively grasp the benefits and risks as a sense.

The human brain makes risk judgments from finite information, finds coping behaviors that reduce risk within a limited time, and takes action. As a human behavioral psychology, whether or not a person can take the necessary coping action at the necessary timing depends on the necessity of the person taking the coping action and how the necessity is learned from past experience. , It depends on experience and background, and it varies from person to person. On the other hand, as autonomous driving becomes more sophisticated, it is expected that driving and steering will be able to continuously handle more diverse situations.

The driver's skepticism about the system is gradually diminishing, as the need for the driver to intervene in the return from automatic driving to manual driving is diminished, at least sensuously, in order for the autonomous driving system to cope with various situations. In the meantime, when a necessary situation arises, the driver will gradually stop paying attention to the front, checking the side and rear, and checking the observation of the vehicle in front of him in preparation for driving and steering.

Therefore, even if the system requests a sudden return to manual driving, the driver shifts his thinking to attention and interests other than driving once he leaves the driver's independent steering work loop. .. As a result, the system notifies the driver of a request to return to manual operation, and even if the driver catches the notification, the driver needs to provide the missing information in order to start from grasping the interrupted situation. It will take a long time before it becomes possible to grasp the uptake status and take action to avoid an actual takeover accident. In addition, when the driver, as NDRA, starts an action / action that is physically separated from the driver's seat, it will take more time to move including the recovery of consciousness.

When a person drives a car manually, he / she can safely deal with various events that occur in the course and many of the events that occur each time, without any accidents, and refrain from staying in progress. I'm doing the work. However, behind the seemingly casual and safe driving work, in reality, it is necessary to unknowingly confirm a lot of information in advance and make a judgment each time in order to predict the future of the impact of the operation from that information. We are devising ways to prevent delays in action decisions necessary for accident prevention by exploring various information in advance to obtain a certain degree of security.

For example, even if one action of stepping on the brake pedal is taken, the information that the driver grasps in advance when stepping on the brake pedal is
・ How to apply the brakes of the vehicle according to the degree of depression of the brake pedal,
・ Cargo of own vehicle,
・ Judgment information on whether the distance required for braking required by the occupant load capacity is long,
・ Understand the risk of road slip (wetness, snow cover, etc.) and decelerate in advance to reach the relevant section.
・ Prediction from the behavior of the vehicle in front,
・ Judgment of the danger of sudden deceleration due to the existence of the following vehicle and its model,
・ Fog generation at the destination,
・ Whether there is a risk that judgment will be delayed due to factors that obstruct visibility such as backlight.
Etc., a lot of information will be combined and the final braking brake will be applied for the final danger prevention.

In other words, once the driver deviates significantly from the driving steering loop in automatic driving, the related prior storage information (working memory) required for driving steering control has not been acquired, that is, the situation is not grasped, and the automatic driving is manually performed. The transfer to operation will be started. Even if a driver is mechanically and suddenly requested to take over, the driver may not always be able to instantly obtain prior judgment information.

Therefore, the driver may fall into a panic state if he / she is required to make a decision on how to deal with the behavior while he / she is not fully aware of the situation. In this case, the driver may be required to take action in the panic state. In other words, in order to enable HCD control that takes into account the process of human judgment, as a system, the time required to restore the state of the driver's thoughts, posture, etc., and a system that enables continuous driving with automatic driving. It can be said that there is a need for a mechanism to request the return of the driver who has secured the option of securing the remaining grace section that allows continuous driving by automatic driving at all times, in balance with grasping the situation of the road environment.

At this time, it is actually extremely difficult to estimate the thinking state of a person. For example, just by observing the driver from the outside, the driver's line of sight looks forward and the thought seems to be paying attention, but it is not actually related to driving. It may happen that you are thinking about. In such a case, the driver's thinking (working memory) may be assigned to an event completely different from driving, and the working memory may be in a situation where the information necessary for determining the driving behavior is insufficient.

The system needs to estimate the time (grace time) required for the driver to return to normal driving before the end of the section where the automatic driving function can be safely used. In the case of existing manual operation, for example, carelessness ahead by the driver may lead to oversight of danger and may directly lead to an accident. Therefore, in order not to inevitably fall into a situation where the driver neglects to pay attention to the front, basically, even if he / she is temporarily involved in work other than driving, he / she continuously collects information necessary for driving. There is no interruption.

Therefore, during the existing manual driving, the driver constantly pays attention to the eyes intermittently, and it is easy to index the withdrawal of attention such as drowsiness for driving based on the observation data by observing the decrease in their behavior. .. On the other hand, if the automatic driving function of automatic driving level 1 or higher is used, the driver does not need to perform some work required for steering. Therefore, as the automatic driving function becomes more advanced to automatic driving level 1 or higher, the driver Gradually reduces the need to intervene in driving. As a result, the amount of information gathering and judgment behavior by the driver based on safe driving and steering judgment is gradually reduced, and additional information that is insufficient to grasp the situation and take judgment behavior after receiving the notification of returning to complete manual driving is required. It will be necessary and time consuming.

There are two types of triggers for a person to take action: a trigger based on a thoughtful procedure and a trigger by a stimulus that is reflected in the action in order to avoid danger even if the thoughtful procedure is missing. Here, since the latter behavior is a reflexive avoidance behavior and is a risk avoidance behavior based on limited information performed in an information contingency, thought feedback appropriate for the behavior works appropriately and effectively. Often the behavior is untouched.

In other words, when the normal manual operation control is continuously performed, the driver suddenly operates the steering, suddenly depresses the brake pedal, or suddenly presses the brake pedal in order to control while constantly monitoring and confirming the front of the vehicle. , It is not usually the case to overdo them more than necessary. On the other hand, if your vehicle is trying to get out of the lane due to inattentiveness or carelessness, or if you are distracted only by the risk without noticing the brakes of the vehicle in front and not being able to grasp the situation, it is more than necessary. Excessive steering operation or sudden braking may cause unexpected disasters such as vehicle rollover, rear-end collision with the following vehicle, and vehicle spin.

The cause of this lack of appropriate behavioral control is considered to be the lack of information that enables the prediction of secondary damage to the working memory required to control the amount of behavior, and the excess of information that needs to be dealt with. Excessive information leads to a panic of thinking and makes it difficult to perform appropriate coping feedback actions such as controlling the degree of coping, resulting in an operation such as excessive steering for avoidance. Furthermore, human information collection also has a function to filter out unnecessary information by excluding such unnecessary information when continuously receiving unnecessary information for behavioral judgment or the like.

Therefore, even if the information is related to the transfer, if the information that is not necessary for judgment in normal use is continuously and mechanically provided to the driver unilaterally from the system, the information will be operated. It will occupy the working memory of the person. Such unwanted information is unknowingly perceived as noise by the brain and loses its weight of importance and is filtered, as this is an obstacle to obtaining other potentially important information. Will be done. In this way, among the external world information physically acquired by humans, noise, which is called the "cocktail party effect" in the field of psychology, is a good example of filtering processing before entering the thinking of the brain. Among them, it is known that the conversation of a specific person can be heard well.

Based on this mechanism by which the human brain selects and uses information, how is the driver as continuous information that is frequently unfolded and updated as the vehicle travels, and that the events that appear on these new routes are urged? It is necessary to allocate it to the working memory, which determines how the driver decides how to judge the related information that is important for taking over based on the information given by the system, and to connect it to the judgment. ..

Here, the brain region of human working memory cannot be directly observed and visualized, and the fact that it acts directly on working memory is not explicitly realized by today's technology. It's extremely difficult. There are many different priorities for the driver's thinking activities depending on the situation of the driver at that time, and this working memory is not something that the system can directly leave information on.

Based on these characteristics of human beings, the system has a priority factor for actions for drivers in order to take over the autonomous driving with a margin before the end of the section where autonomous driving is available. It is necessary to provide the unique information to be used and to train the driver to measure the influence of the individual information on the driver. That is, in the case of simple information provision to the driver, the provided information may be equivalent to noise for the driver. Therefore, if the information provided makes sense in predicting results and there is a risk of detriment to the driver through learning, that information is important, high priority information in working memory. ..

<3-2-2-5. About the "contract" between the system and the driver>
Here, the exchange and confirmation operation of the notification between the first system and the driver, in which the driver receives the information notified by the system and takes responsibility for the receipt, is referred to as the "contract" between the system and the driver. I consider it. From the initial contract, the execution of the takeover work based on the initial contract and the "degree of achievement of the contract" are analyzed based on the observable information as the quality of the return transition (return operation). The quality of the analyzed return motion is the "credit information" for the execution of the driver's contract. This credit information will be used as a threshold for determining the availability of non-defective products with higher quality than the contract when switching to automatic driving and reusing in the next itinerary or the previous itinerary segment, and will be used by the driver each time. Reinforcement learning progresses as an intuitive sensation in the driver by feeding back the coping event, influence, and visual sensation.

In other words, this series of "contracts" is performed by responding to the notification, not just the passive information that the system has unilaterally notified to the driver. As a party, the driver sees the response to the notification as an obligation to return to his manual driving for the contract. The driver will be able to control the HCD that has voluntarily participated in the use of the autonomous driving system through a series of repetitive operations corresponding to the return obligation.

The method of presenting the individual information described in the embodiments of the present disclosure merely describes some representative means of the various means that can be used to achieve this HCD. It is not limited to the examples described. In particular, how drivers can remember the terms of a "contract", remember their obligations over time, and fulfill them with high priority and without delay at the required time will vary from person to person. There is a difference and there is no need to limit it.

HCD is not a simple control that provides the driver with simple specific information. HCD is a design from a broader perspective that incorporates consideration for fulfilling the functions associated with the intended use of the system, which is grasped from the perspective of human cognitive judgment behavior. More specifically, HCD needs to build a system that incorporates the mechanisms necessary for the development and development of desirable cognitive behavior.

Human behavioral psychology does not spontaneously and unconditionally promote desirable behavioral development. In other words, a person has the merits of the desires, etc. that he or she seeks in accordance with the norms and rules that are required of an individual who is a member of a family, community, or society, and the disadvantages of punishment, etc. that the society has established as rules and norms. A person's unique behavioral psychology develops and he / she comes to deal with things in a balance with the risk that he / she directly suffers regardless of social norms. From this point of view, the psychological impact of human behavior brought about by the stage of driving support, which is one of the automatic driving functions, is a demerit or risk of a sense of risk for a decrease in attention due to direct driving steering mistakes or fatigue. It is a reduction, and there is an unnecessary improvement in security.

However, the use of the driving support system in the first place improves comfort, and even if the driver who uses the vehicle with his / her original attention should lose his / her attention or overlook important information. The ultimate goal is to enable accident prevention and avoidance, and to reduce the risk of accident aggravation in the worst case.

Therefore, in the embodiment of the present disclosure, in order to avoid excessive dependence on the driving support system by the driver, not only comfort is pursued for all steering in which the system assists and intervenes, but also for obvious excessive dependence. Even if the self-avoidance process by the automatic driving system is executed, such as the introduction of unpleasant control for the driver and the forced stop of the support function, the alternative risk that is a penalty for the driver is given. As a result, even if it is possible to prevent the driver's excessive dependence on the autonomous driving system from leading to a direct accident, it will not be a complete risk aversion, thereby creating a cycle of modest usage psychology of the driver. I'm building.

Here, if the automatic driving function is added to the system beyond the driving support, the concept of using the system will change drastically. In particular, since there is a period in which the driver is not required to be involved in driving control at all at the automatic driving level 4, a situation occurs in which the driving steering is functionally risk-free for the driver. ..

In MCD's view, it would be fine if all the conditions for driving at autonomous driving level 4 are met, but as already mentioned, the use of excessive dependence brought about by autonomous driving has various negative effects such as traffic congestion. As a social mechanism, the chaotic use of autonomous driving level 4 is not a desirable situation. There are modest rules for using the system that must be observed, and a typical rule is to use autonomous driving only within the range of use where conditions are met.

Based on the idea of HCD, the use in accordance with the orderly social norms means the use of the automatic driving level 4 in which the driver is involved in NDRA only in the section where the automatic driving system allows the use in the automatic driving level 4. However, the driver can quickly learn the behavior stipulated by social norms when the necessity arises due to the situation where the end is predicted or the situation change, and the strengthening learning of the habit progresses by daily use. It needs to have a mechanism.

In other words, if the driver's behavior change to a prompt and suitable return behavior quality after the notification of the return request does not proceed, he / she will enjoy the merit of using the automatic driving level 4 automatic driving as a "benefit". I can't. Then, the initial information of the "risk" necessary for human judgment thinking is the information that is unknowingly temporarily stored in the memory by the driver's approval of the "contract" presented to the driver by the system. Changes that may occur after the start of one section of autonomous driving level 4 use and the interaction between the system and the driver by the reconfirmation HMI become an "incidental contract" for reviewing the conditions for changes that occur over time. In addition, every time the driver resumes driving in the section of automatic driving level 4, he / she comes into contact with the information of the end point once, and under the agreement, that is, in contact with the necessity of return and the general return timing and end request information, automatic driving is performed. The driving itinerary within the ODD section determined by the system as level 4 is started.

Conceptually, the "contract" referred to in this disclosure may be an interaction between the system and the driver regardless of the actual exchange via physical documents or the like. Through the interaction, the driver's memory is informed of the necessity of recovery, the risk of its influence, and the severity of the result in case of violation, so that it becomes memory information that is hard to forget according to the importance of coping.

The driver cannot unconditionally use the automatic driving of the automatic driving level 4 permitted by the system, but can use the automatic driving by including the obligation to return from the automatic driving to the manual driving as an incidental condition. The quality of the driver's compliance with the incidental conditions is the credit evaluation when the driver uses the automatic driving later. For example, if the driver's use of the automatic driving function violates the permissible use, the execution permission merit such as NDRA, which is the merit of using the automatic driving, cannot be obtained at all. Furthermore, restrictions on the use of vehicles may be a disadvantage. Due to these disadvantages, the driver becomes more cognitively sensitive to the risk prediction notice information in the middle of the itinerary, and as reinforcement learning progresses, the driver becomes more sensitive to the notice information that contributes to maximization without losing the merit. It will be like.

That is, the automatic operation control by the HCD according to the embodiment of the present disclosure incorporates into the system a mechanism by the conventional MCD that simply issues an alarm when the transfer point is imminent and forcibly restores the driver's consciousness. However, it is completely different from the concept that the system periodically forces a return request because the driver's consciousness does not leave the driving steering loop in the first place.

It is considered that this conventional MCD control concept is only intuitively annoying to the driver. Therefore, some drivers may weaken the function of the alarm in order to eliminate the annoyance, and it becomes a habit to become insensitive to the alarm and immerse themselves in NDRA without worrying too much about the alarm. It can lead to situations. This leads to a situation in which the driver ignores the alarm issued by the system, or if the alarm is, for example, a repeated monotonous buzzer sound, the driver's auditory filtering effect makes it unimportant.

In issuing a return request to the driver, the system described in detail above uses the risk information that affects the driver's near future as appropriate change information to the driver in a multidimensional and variable manner, that is, uniformly. The driver actively reconfirms the "incidental conditions" in response to this presentation. As a result, the risk information is distributed to different memories such as the auditory language center, the visual language center, and the visual field in the driver's working memory, and the memory stimulus for the driver's return is not monotonous.

As a result, even if there is a release of thought that is separated from the driving steering task, such as mind wandering, forgetting of the obligation to return will be suppressed. At the time of starting the automatic driving in the automatic driving level 4 section, the factor information that is attached to the "contract" and requires the return by the return obligation is visually presented. Further, even in the intermediate process of automatic driving in the automatic driving level 4 section, new updated information is presented according to the updated information according to the characteristics of the driver's ease of forgetting. Since the driver can re-evaluate the risk by presenting this information, it is possible to reactivate the important memory of the driver's working memory.

Another important point based on HCD is that the contribution of the system's notification to the driver and the stimulus by the alarm to the cognition necessary for the judgment of driving behavior is simply the strength of the notification (loudness). It does not act as (etc.), but acts according to the difference in sensitivity to stimuli that lead to the driver's individual risk.

In other words, in HCD, the strength of physical stimulus is not important, but in the brain, information that is important for judgment in the near future is processed with higher sensitivity, and less important information is postponed. Take advantage of the fact that there is a mechanism to make it. In the embodiment, the system machine-learns how to present the information that has grown up to the specific information group that is possessed as the individual characteristic of the person by artificial intelligence or the like, and makes an early judgment using the information that has an influence. Design the HMI to encourage.

For example, instead of limiting all information to visual information and narrowing down the method of presenting information, information is presented to the driver by comprehensively stimulating using multiple different information. Specifically, it is conceivable to give a stimulus to the driver by letting the driver hear a specific sound that is auditory and then giving a visual notification following the sound. Then, the system gives the driver credit points as a good driver when the driver's response to this stimulus is quickly and accurately performed. In addition, the credit points added to the good driver are not limited to simple additional points stored in the mechanical storage medium (memory, hard disk drive, etc.), but intuitive visual feedback to the driver on the spot via the HMI. Do it together. As a result, the driver's psychological psychology is that the scheduled timing based on the driver's "contract" and the obligation to perform an accurate early return under the circumstances are intuitively linked, and the driver's own response is optimized. Reinforcement learning occurs.

In neurons, which are the optic nerves that trigger decisions, microscopically, synapses are stimulated by a large number of factors, and the memory that temporarily holds information that requires attention corresponds to this alert state of waiting for firing. .. Increase sensitivity to related information that has risks in the near future, and deal with important matters of memory so that you can take prompt action when you receive the necessary stimulus with the information necessary for judgment. The state of waiting is the situation where related information is incorporated into the so-called action judgment working memory. And if this stimulus path is many and diverse, it will be an anchor effect to keep the priority high even if the floating of thought occurs temporarily due to mind wandering etc., and it will be a risk factor presented in parallel at the same time. The need can be strongly maintained by the visual and auditory information of.

In other words, the act of the driver confirming the "contract" or "incidental contract" presented to the driver by the system is a reconfirmation of the information by the driver. Microscopically, the operation of reconfirming this information can be considered to play a role in activating the synaptic potential in the state before the ignition of the judgment. Since the judgment optic nerve is placed in this preparatory standby state, the driver's perceptual sensitivity to trivial information when approaching the end of the section permitted as ODD such as automatic driving level 4 is increased. As a result, even if the information obtained is not perfect, the driver will be placed in a situation where the awareness of necessity has increased, and if the memory is insufficient, the driver will voluntarily try to supplement the information in order to suppress the increase in risk. As a result, we aim to complete the "incidental obligation" based on the initial "contract". When the driver feels uneasy, for example, it is reflected in the action of visually reconfirming the status screen related to the contract.

<3-2-2-6. About operation of automatic operation level 4 ＞
Next, the concept of operation of the automatic operation level 4 according to the embodiment of the present disclosure will be described.

As the vehicle approaches the end point of the available section of autonomous driving level 4, the driver gives up early on the continuous use of autonomous driving of autonomous driving level 4 and takes prompt and appropriate measures (return action). The question is whether to take it. In order for the driver to interrupt the NDRA, which is considered to be a merit when using the automatic driving by the automatic driving level 4, and shift to the returning action to the manual driving, the working memory that controls the judgment should be used for the returning action. Information related to the migration needs to be remembered.

The trigger for storing the related information in the working memory is a "contract" between the system and the driver when the automatic operation level 4 is started to be used. The driver once recognizes the liability obligation at the time of "contract" at the start of use. However, if the point at which the actual use section ends is some time ahead, it will be necessary to re-clear the memory in order to fulfill the return obligation before reaching the limit point for taking the action to return to manual driving. , The presence or absence of trigger information and the degree of risk importance greatly affect the success or failure.

From an ergonomic point of view, a stimulus associated with some reason leads to an overwhelmingly accurate judgment, compared to a mere return behavior that has no reason other than itself. Therefore, it is useful to present the succession factors with different factors below for the successful return.

Assuming the use of autonomous driving up to autonomous driving level 4, if the driver starts using autonomous driving level 4 in a road extension section that has been maintained and managed, the action to immediately return to manual driving in that road section will be Basically not required. In low-speed automatic driving such as low-speed automatic driving (Low Speed Automated Driving), when the coping limit of automatic driving is exceeded, it is possible to secure time for coping by decelerating or stopping the vehicle.

On the other hand, on general roads where many general passenger cars come and go, it is necessary to take automatic measures while riding on the flow without disturbing the flow of the relevant road section. At this time, when it is expected that automatic handling will be difficult, the system will either take over to manual driving or allow the smooth handing over to the driver to be completed at the cruising speed that is safe for autonomous driving. Or, if that expectation is low, whether to take an evacuation run to the shoulder, service area, evacuation parking pool space, stop or general road where low-speed driving is possible, etc., while the evacuation option is left. You need to make a choice.

In a situation where a general vehicle is driving at automatic driving level 4 on general roads, highways, highways, etc., what forces the continuous driving of automatic driving to be interrupted depends on the characteristics of the vehicle, the road, the environment, and driving. There are various situations such as the coping situation of the person. Here, when it becomes difficult to continue driving due to automatic driving, the system does not always have a 100% probability of taking over to manual driving, and if an emergency stop or deceleration is executed on the spot, the vehicle will not be so much. As already mentioned, even if there is no negative impact, there is a high possibility that the impact on the following vehicle, that is, a large social impact, will occur.

Therefore, what is needed is a remaining driving section in which even if it is difficult for the driver to take action to return to manual driving, there is a retreat option that allows the vehicle to minimize the obstruction to the following vehicle, etc. In the meantime, it is the start of coping control so that the system can determine the confirmation of the success of the takeover.

The following is an example of an event in which continuous use of automatic operation level 4 may be difficult.

Depending on individual vehicles, individual risk coping sensations, abilities, etc., driving at autonomous driving level 4 needs to be compatible with road use, which is actually an orderly social infrastructure. Here, it is extremely difficult at present for the system to judge the appropriate use for an orderly road environment in the same way as the human thinking ability. Alternatively, leaving the functions up to that point to the system is also a problem related to the roots of human beings, and from a moral point of view, it is conceivable that the ultimate autonomous driving will not be performed. This is also the denial of a society in which humans are used by machines.

As an example, in a situation where it is possible to pass for the first time due to a concession on a narrow street, a person interrupts or intervenes in the steering of automatic driving, selects a priority, breaks the entangled state, and passes. As a simple example, when passing through a narrow bridge with a single lane, by communicating with the oncoming vehicle in advance and alternately giving up the passage, the progress of some vehicles that oppose each other was completely hindered. There are cases where it is prevented from being left behind and each section is passed.

Therefore, in reality, it is advisable to artificially incorporate a control mechanism for maintaining social order just before that section. At this time, for example, on a highway, even if driving at automatic driving level 4, it does not affect other vehicles, so under what circumstances the planned itinerary driving is abandoned and the driver manually drives. It is preferable to proceed to the selection judgment process such as taking over the vehicle, supporting remote driving, and evacuating in advance without affecting the vehicle. In this case, an appropriate judgment is required.

Examples of factors that abandon the continuous use of automatic driving level 4 automatic driving are shown below as each item. These items can be a factor in this, either alone or in combination of multiple items.

The first factor is a factor that abandons continuous driving of automatic driving level 4 due to the availability of road environment information obtained in advance of the destination of the relevant road in use, and as an example, the following items. Can be mentioned.

(20-1) High freshness from LDM and lack of ward information of updated LDM due to a failure of a vehicle traveling in a section where information is collected regularly (20-2) High freshness update Communication band for constantly updating LDM Information has not been acquired due to primary congestion (20-3) Restrictions on the use of local map information based on the subscription contract due to expiration of the contract (20-4) There is a leading vehicle with the driving assistance of the pairing leader. Unable to acquire continuous data due to failure of the leading vehicle (20-5) Insufficient shadow roving data of road information from general vehicles due to a decrease in density due to the time zone of vehicles traveling in the section (20-6) Malfunction of communication equipment of own vehicle , Infrastructure communication failure (20-7) Communication Insufficient update information required for continuous automatic driving due to cyber attack, or change due to fabrication

As the second factor, there is a possibility that the continuous driving of the automatic driving level 4 may be abandoned according to the information notified from the pre-update road environment information of the destination of the corresponding road in use. Examples of factors in this case include the following items.

(21-1) Overcapacity for coping by remote driving support controllers (21-2) Lack of remote driving support operators (21-3) People and large animals entering vehicle-only roads, loading platform for vehicles traveling ahead Situation where animals from the area are wandering on a dedicated road, reception of information on scattered emergency falling objects from the section leading vehicle (21-4) Reception of information that makes it difficult to predict abnormalities such as earthquakes, cliff collapses, and tsunamis (21-4) 21-5) Receiving warning information for following vehicles obtained from a voluntary danger report from the leading vehicle in the section (21-6) Partial and unexpected freezing of the road surface such as wet bridges and shaded mountains (21-6) -7) Traffic restrictions due to unexpected road construction and post-accident processing (21-8) Passing through sections by traffic control through human communication (traffic control such as accident handling)
(21-9) When the vehicle continues to drive automatically, there will be no evacuable sections due to narrow road sections, alternating single passage bridges, tunnels, etc., so entry into the prohibited sections according to the driver's condition (21-9) 21-10) Crossing the railroad crossing section

As a third factor, there is a possibility that continuous driving of automatic driving level 4 may be abandoned due to the performance limit of the sensing device mounted on the own vehicle and the performance fluctuation with the passage of time. Examples of this case include the following items. In this case, the continuous driving is abandoned according to the start / start time of the automatic driving according to the automatic driving level 4 and the change of the situation during the running.

(22-1) Deterioration of detection performance of millimeter-wave radar, LiDAR, camera, etc. due to snow removal material and dirt rolled up by the traveling vehicle in front (22-2) Windshield of insects and flying objects while using high-speed driving Temporary deterioration and limitation of sensing camera performance due to collision (22-3) Noise generation due to temperature rise during use, partial deterioration of detection performance (22-4) Pebbles rolled up by the windshield, adjacent lanes, etc. Local damage to windows due to flying objects, etc. (22-5) Failure of headlights, etc. in vehicles running at night, reduction of visibility detection limit by camera (22-6) Windshield due to erroneous operation of indoor air conditioning, etc. Cloudy, temporary deterioration of detection performance of indoor-installed sensing camera (22-7) Request for recovery of unknown cause based on self-diagnosis result by vehicle system

As a fourth factor, there is a possibility that the continuous driving of the automatic driving level 4 may be abandoned due to the change in the condition of the load affecting the running of the own vehicle during the continuous driving of the automatic driving and the change of other vehicle dynamics. .. Examples of this case include the following items.

(23-1) Load collapse during normal driving (23-2) Tire air bleeding, tire burst (23-3) Riding on scattered objects, driving performance fluctuations due to road abnormalities, abnormal noise in the vehicle (23) -4) Load collapse due to sudden braking to prevent collision, large movement of passengers in the vehicle, and weighted balance loss due to it (23-5) Brake abnormality detection by self-diagnosis of the running vehicle (23-) 6) Engine overheating, malfunction of control equipment

As a fifth factor, there is a possibility that continuous driving of automatic driving level 4 may be abandoned due to an abnormality of the driver. Examples of this case include the following items.

(24-1) Unexpected nap of the driver, leaving the seat, unpredictable return when necessary due to undetectable state (24-2) Sudden seizure of the driver (asthma, convulsions or numbness of legs, etc., Allergic reaction due to sudden invasion of pollen etc., heart attack, sudden headache, stroke, cerebral infarction, etc.)
(24-3) Detection of abnormal behavior due to drug dependence (24-4) Interference with driver status monitoring during implementation (24-5) Necessary response processing required by the system for continuous use of autonomous driving Ignored or left unattended by the driver (24-6) Driver's break status, activity / break / custom history information from the previous day, etc.

There may be a combination of several conditions and contributions, such as each item of the first factor to the fifth factor described above, but if the vehicle continues to drive at automatic driving level 4 with those combinations, the vehicle will decelerate. Situations where emergency stops, inefficiencies or major disruptions to social transportation infrastructure should be avoided.

Of the above, at least the first to fourth factors are the degree of impact and penalties for violations when the driver does not perform the return action according to different conditions according to the risk judgment. Receive feedback in advance by HMI, which can present sensory expressions such as applications, for example, expressions using visual stimuli. The driver temporarily remains as a visual sensory stimulus because, for example, the visual stimulus is taken into the working memory at least once by this pre-feedback by the HMI. By giving the driver a stimulus related to this visual and sensory stimulus, it is possible to refresh the memory and maintain the memory of the need for the return behavior.

On the other hand, if the driver himself has an abnormality, which is the fifth factor, it is extremely difficult to encourage early recovery behavior by this HCD. However, even in this case, the system presents the driver with a function to abandon the automatic driving at an early stage, and voluntarily requests the driver to give up based on the information of the evacuation possible point by this function. Or you can use it to make a rescue request. This use is a precautionary measure rather than the system leaving the driver unattended without presenting any information to the driver, advancing to the takeover limit point, and activating MRM only after reaching that point. Control with a margin is possible.

In addition, the examples shown in each of the above items are typical examples of the treatment required for important highways of social infrastructure, which may cause problems when the own vehicle stops in the driving zone of the road. Become. If the road is very unlikely to cause traffic obstruction even if the vehicle suddenly or completely stops due to MRM, such as a road with extremely low traffic volume or a road with a wide road width even if it is not an arterial road, the above-mentioned It is not limited to each item of.

In other words, when the driver continues to drive at automatic driving level 4, if the section can be executed by MRM for emergency stop or evacuation without hindering social activities, the system will take measures for the driver's return. It is possible to have the vehicle continue the planned run as scheduled, without taking into account the availability or regardless of the driver's condition. That is, in this situation, even if the system encounters a difficult situation to deal with at the automatic driving level 4 and the driver cannot return properly, even if the vehicle is urgently stopped or evacuated to any place, the society This is because it does not obstruct the passage of roads as an infrastructure.

In this way, the situation where the use of autonomous driving of automatic driving level 4 is appropriate is not uniformly determined, it is not a fixed environment, and the ability to respond to the driver's return request, road conditions, usage environment, vehicle condition, etc. It changes actively due to various factors.

<3-2-2-7. Effect of adopting HCD>
Next, as described above, the effect when HCD is adopted in place of the existing MCD will be described.

First, the essence of the control of the HCD according to the present disclosure is that the available section of the driver's automatic driving is variable, and the determination of the available section is the awake state at the time when the driver is observed. In addition to the observable evaluation value, it depends on the acquired credit information of the driver. Further, in the HCD according to the present disclosure, the information detected by the system and the result thereof are presented to the driver as near-future risk information using at least a visual representation in the intermediate process in determining the available section.

Unlike simple return request notifications that are uniquely presented to the system, information is presented as a risk of the driver's own selection behavior, so that thoughtful selection work that considers the balance between advantages and disadvantages is performed. .. This captures information about the importance of the driver's working memory and keeps it fresher depending on the importance of the takeover. In addition, the behavioral evaluation for this thoughtful selection work is further reflected in the future usage conditions, and it is determined whether the current usage permission can obtain the merit of using the automatic driving based on the past evaluation results. Therefore, in HCD, the driver can acquire a sense of responsibility for use and a sense of whether or not to enjoy the merits, which is different from the aircraft instruction by MCD, through repeated use.

Furthermore, through repeated use, the driver learns how to engage in a way that suits him / herself, and if the behavior impedes social activities, the system imposes a penalty and suppresses the use of the driver. Can avoid detrimental behaviors that are an obstacle, and can suppress usage patterns with poor social acceptance.

Secondly, because autonomous driving can significantly reduce the intervening load on human driving and steering work, in today's social situation where accidents caused by human factors are said to be about 94%, instead of humans. By driving the vehicle in the automatic driving mode, it is expected that the occurrence of social accidents will be greatly reduced.

However, the introduction of autonomous driving, which is widely considered today, is based on the premise that the driver can appropriately return to manual driving in response to the request of the system and the driver can promptly respond to the request. We are trying to introduce it to society. However, this premise may not hold in a situation where the improvement of the performance of the automatic driving system may lead to the driver's excessive dependence on the automatic driving system.

In this disclosure, the availability of the automatic driving function is basically provided according to whether the appropriate usage applicability of the driver is achieved. In addition, HMI that promotes appropriate use is incorporated into the mechanism. This prevents the driver from being overly dependent on the system, and realizes a control technology for the driver to subjectively take an appropriate return behavior.

That is, the present disclosure changes the method of usage control for automatic driving of a vehicle from an MCD that unilaterally issues an instruction from a conventional device to an HCD that performs usage control according to human behavior characteristics, and realizes this. Regarding vehicle control with HMI introduced. In particular, when the driver uses the automatic driving function of the system, he concludes a "contract" with the system called "confirmation act" that the driver executes without neglecting the request when the "ceremonial" automatic driving is completed. Then, the validity of the contract will be reconfirmed as appropriate during the course of the use of autonomous driving.

Such an HCD system is not realized by simply incorporating one function into the system, but reinforcement learning is carried out by repeating the complex use necessary for various drivers to develop usage habits. There is a need. An embodiment is an HMI that works with the driver to advance their reinforcement learning and maintains an appropriate early return as a sense of use for a long period of time. The combination of execution means for feeding back to the driver is not limited to the examples described in this specification.

The allowable automatic driving level of conventional automatic driving was determined mechanically by the system according to the road environment conditions that the equipment can handle, and it was assumed that the driver would uniquely switch the usage selection at his / her own discretion. On the other hand, the present disclosure provides support for the development of a habit in which the driver appropriately starts the return procedure without delay in response to the request of the system by repeated use, and the HMI necessary at that time. As a result, even if the introduction of autonomous driving becomes widespread in society, the system will have an emergency deceleration or stop on the road due to an emergency MRM due to a delay in the driver's return work, etc. It has the effect of widely suppressing the occurrence of controls that lead to obstruction and preventing the obstruction of social activities.

Thirdly, in order to use automatic driving that frequently takes over to manual driving, it is necessary to understand the characteristics of the driver's working memory in order to continue driving safely.

Even if the necessary coping events are perceived and recognized and information is once taken into the working memory, the consciousness of mind wandering progresses with respect to other events due to the person's thoughts at that time and the imbalance of the autonomic nerves. Even if it is an important matter, its importance diminishes from memory, and the response may be delayed, leading to critical results.

The amount of information assigned to a person's thoughts is finite, and at the same time, complete consideration cannot be assigned to all matters. Therefore, if information is learned by multiple different systems, information is input to the information of different systems by different means such as visual and linguistic, and it becomes possible to describe the results of each, it is necessary even if mind wandering occurs. It will be easier to get your thoughts back to what you are dealing with. In other words, when the HMI, which presents information that leads to the prediction of the outcome of a certain effect, as a sense of risk, rather than presenting information in a rutted manner such as a simple symbol, the need for handing over is clarified. It becomes effective for.

When HCD is adopted for control, even if the information once stored in the working memory of such thinking during work, a transition of thinking occurs in other things when it happens, and even the information acquired by thinking that it is originally important changes the time. As a result, it may be forgotten. Therefore, it is necessary to give feedback to the driver to revive his / her memory according to the degree of forgetfulness inherent in the driver's current situation. Therefore, in this disclosure, "easiness to be forgotten", which is an important matter as a daily characteristic of the driver, and "how freshly the driver can remember and retain the driving-related precautions at the time of driving use". Evaluate and present memory refresh information suitable for the driver.

In addition, as one of the embodiments of the present disclosure, in the case of using remote support, it is possible to confirm and predict the point where the support cannot be received in advance. By providing sufficient information such as road shoulder bandwidth and SA (Service Area) to the driver via HMI, if support is not received, it will not interfere with traffic to others. Since standby selection becomes easy and a mechanism can be realized that can be established even with a limited number of remote support operators, it is possible to provide practical and efficient remote support. In addition, the infrastructure required for remote monitoring can be operated efficiently.

Fourth, it is possible to improve attention by the following measures for secondary tasks other than driving performed by the driver, that is, when NDRA mainly uses visual information of an electronic terminal. That is, in a terminal device having a monitor screen, short-term information related to automatic operation may be presented in the display image area as the original NDRA, and visual information related to the necessity of taking over may be displayed.

For example, the visual information may be displayed in an extremely short period of time aiming at the so-called subliminal effect, which the viewer does not consciously notice. Further, apart from the presentation of information that remains completely out of consciousness, it is possible to present information that is clearly conscious by the driver, which is longer than when the subliminal effect is aimed at.

In the case of information that does not necessarily pass through verbalized understanding, such as the subliminal effect, it is more effective if the driver has a visual and intuitive sense of risk as a risk when the transfer request is ignored. For example, in the case of a violation, the visual depiction of being watched by a white bye on the side of the road rather than the textual information of the penal provisions, and the depiction of receiving a tracking stop order for violation crackdown rather than the static information of the alternation image, Examples include depictions of situations where confirmation of violations is required, and depictions of risks incurred when MRM cannot be continuously implemented.

A person usually grasps visual information consciously and makes a linguistic interpretation of the captured visual information. On the other hand, it is academically suggested that humans have a mechanism of information transmission that can act on the brain in a way that does not involve any linguistic interpretation and even leaves no consciousness.

These short-term stimuli, for example, due to the subliminal effect, are expected to have the effect of refreshing and reactivating important information that has disappeared from the working memory. This effect is different from the effect of the system acting to awaken the decrease in consciousness due to fatigue and drowsiness, and has the effect of restoring the stored information for the working memory necessary for the decision to take over in consciousness.

When a person remembers important information, he / she may understand the necessity of correspondence somewhere, but forget it and cannot remember it at the required timing, and may remember it later.

This is because important information that needs to be dealt with is not preferentially stored in the working memory as an important matter and does not lead to recall from the memory. In order for the memory information to work effectively for judgment, it is necessary to increase the amount of stimulation of the memory. Similar to blindsight, the subliminal effect affects behavioral judgment even if it is not consciously regarded as visual information. However, since the main purpose is not to display, the subliminal effect is an example of the ultimate short-time HMI, and a longer display that affects consciousness may be performed, and the display may be displayed until the driver cancels the display. By continuing it, you may work more strongly as a risk description.

In other words, when using automatic operation, the system will only have a "contract" between the system and the driver to respond to a request to take over manual operation due to changes in circumstances. Allow the use of continuous automatic operation. The driver does not forget the "return obligation fulfillment" that accompanies the "contract" while the driver utilizes the automatic driving function according to the "contract" and utilizes the benefits of NDRA during automatic driving. Some kind of stimulus that reminds us is effective in doing so. When using an electronic terminal, it is effective to display a display that gives this stimulus on the usage screen of the electronic terminal. In this case, there is an effect of "notifying" the driver of the "return obligation fulfillment" accompanying the "contract" without disturbing the viewing of the screen related to NDRA using the benefits.

<3-2-3. Specific example of HCD according to the embodiment>
Next, the HCD according to the embodiment will be described more specifically. In the following, unless otherwise specified, the automatic operation shall be the automatic operation according to the automatic operation level 4 defined in SAE.

<3-2-3-1. Operation example of automatic operation to which HCD according to the embodiment is applied>
Using the flowcharts of FIGS. 7A to 7C, an operation example of automatic operation to which the HCD according to the embodiment is applied will be described more specifically. In addition, in FIGS. 7A to 7C, the reference numerals "A" to "F" indicate that the processing shifts to the corresponding reference numerals in the flowchart of another figure in FIGS. 7A to 7C. Further, in the flowcharts of FIGS. 7A to 7C, the block generally showing the "document" indicates that the driver is provided with information by a visual stimulus or the like.

FIG. 7A is an example flowchart showing the flow from the itinerary setting to the transition to the automatic operation mode according to the embodiment. The itinerary referred to here indicates a travel plan of the vehicle, and includes information indicating a starting point and a destination of traveling and information indicating a traveling route. In addition, starting the itinerary means starting the running according to the itinerary.

In step S100, the itinerary including the destination of travel is set by the user (driver) of the vehicle. The set itinerary is input to the automatic driving control unit 10112 (see FIG. 1). Based on the input itinerary, the automatic driving control unit 10112 acquires various information necessary for traveling according to the itinerary such as LDM. For example, in step S101, the automatic driving control unit 10112 acquires information such as the LDM, the driver's return to manual driving characteristics, the local weather included in the itinerary, and the luggage loaded on the vehicle. Among these, as the characteristic of returning to manual operation of the driver, for example, the characteristic based on the evaluation made for the driver in the past for the operation of returning to manual operation can be applied.

In the next step S102, the automatic operation control unit 10112 presents a bird's-eye view of the entire itinerary to the driver. Although a specific example will be described later, the automatic operation control unit 10112 may, for example, map information indicating the entire travel route of the itinerary based on LDM, or information indicating a section of the travel route that can be traveled at the automatic operation level 4. Generate display information that visualizes such things. The automatic operation control unit 10112 supplies the generated display information to the output unit 10106 via the output control unit 10105, and causes, for example, a display device connected to the output unit 10106 to display an image according to the display information.

This display is a navigation display that displays the itinerary settings recommended by the system, that is, the automatic operation control unit 10112. The bird's-eye view display here does not need to be a bird's-eye view at a three-dimensional scale that combines the scale based on the physical distance, and if the driver can recognize the intervention point, the time-converted display is used. It may be present, it may be a stereoscopic display, or it may be in another display form.

In the next step S103, the automatic operation control unit 10112 asks the driver whether or not he / she agrees with the itinerary setting recommended by the navigation display presented in step S102. For example, the automatic driving control unit 10112 determines whether or not there is an agreement according to the operation of the driver on the input unit 10101. Not limited to this, the automatic driving control unit 10112 can detect the movement of the driver by using a camera for capturing the inside of the vehicle and determine whether or not the agreement is reached according to the detected movement. The determination may be made according to the utterance of.

When the automatic operation control unit 10112 determines in step S103 that the driver does not agree with the recommended setting (step S103, "No"), the process shifts to step S104. In step S104, the automatic operation control unit 10112 adds another recommended travel route and presents the driver with an option to select this other travel route. Then, the automatic driving control unit 10112 returns the process to step S102, and presents the driver with a bird's-eye view of the entire itinerary by the other traveling route.

On the other hand, when the automatic operation control unit 10112 determines in step S103 that the driver agrees with the recommended setting (step S103, "Yes"), the process shifts to step S105. At this time, when the driver accepts and agrees on the itinerary proposed by the system, the driver grasps the concept of the entire itinerary, and that fact is stored as storage information # 1 in the driver's working memory (WM10). ..

In step S105, the driver starts driving the vehicle and the itinerary starts. The automatic driving control unit 10112 updates the bird's-eye view of the itinerary according to the traveling of the vehicle at the start of the itinerary, and presents the updated bird's-eye view to the driver (step S106). At this time, the automatic driving control unit 10112 controls the automatic driving for each section in the itinerary, and calculates the automatic driving mode for each ODD corresponding to each section in chronological order.

At this time, the driver can grasp the current state of the itinerary by confirming the updated bird's-eye view presented by the automatic driving control unit 10112, and is obliged to return to the manual driving for the latest selection. Can be grasped. The grasped information is stored in the driver's working memory as storage information # 2 (WM11).

In the next step S107, the automatic operation control unit 10112 determines whether or not the ODD section that allows automatic operation is approaching. When the automatic operation control unit 10112 determines that the section is not approaching (step S107, "No"), the process shifts to step S108, continuous monitoring of the situation change is performed, and the monitoring result. Various risk information is updated based on the above, and the process is returned to step S106.

On the other hand, when the automatic operation control unit 10112 determines that the section is approaching (step S107, "Yes"), the process shifts to step S109. In step S109, the automatic driving control unit 10112 presents a contract for dealing with the ODD section to the driver, and determines whether or not the driver has agreed to this contract. This contract includes, for example, conditions for allowing automatic operation in the ODD section. The automatic driving control unit 10112 makes this determination based on, for example, the presence or absence of an operation or action indicating an agreement on the presented contract by the driver. If the automatic operation control unit 10112 determines that no agreement has been obtained (step S109, "No"), the process returns to step S106.

When the automatic operation control unit 10112 determines that the agreement for the contract has been obtained in step S109 (step S109, "Yes"), the automatic operation control unit 10112 permits the use of automatic operation in the ODD and shifts the process to step S110. In step S110, when the own vehicle enters the ODD section that allows automatic driving, the automatic driving control unit 10112 shifts the driving mode from the manual driving mode to the automatic driving mode.

The driver is obliged to return to the manual operation for the selection in the step S109 by shifting the operation mode to the automatic operation mode in the step S110. By grasping the contents of the agreement (contract contents) with the system, the driver has agreed with the system regarding risk handling at the end of the ODD section. In addition, a breach of fulfillment of the obligation to return is accompanied by a penalty for the driver. Information # 3-1 and # 3-2, ... Showing each condition included in the agreed contract are stored as storage information # 3 in the driver's working memory (WM12). Further, this stored information # 3 is stored as ancillary contract when using automatic driving, and is stored as information related to dealing with a new unplanned incident occurring in the itinerary (WM13).

When the operation mode is selected and transitioned to the automatic operation mode in step S110, the process proceeds to the processing of the flowchart shown in FIG. 7B according to the reference numeral "A".

FIG. 7B is an example flowchart showing the flow of processing in the automatic operation mode according to the embodiment. The process shifts from step S110 of FIG. 7A to step S120 of FIG. 7B, and in step S120, the automatic operation control unit 10112 carries out continuous monitoring of the situation change, and various risk information based on the monitoring result. The update is performed and the process is shifted to step S121.

In step S121, the automatic driving control unit 10112 determines whether or not an event requiring the intervention of the driver has occurred based on the result of the situation monitoring in step S120. When the automatic operation control unit 10112 determines in step S121 that the event has not occurred (step S121, "No"), the process shifts to step S122.

In step S122, the automatic driving control unit 10112 determines whether or not the end point of the ODD section that allows automatic driving is approaching. When the automatic operation control unit 10112 determines that the end points of the section are not approaching (step S122, "No"), the process returns to step S120. On the other hand, when the automatic operation control unit 10112 determines that the end points of the section are approaching (step S122, “Yes”), the process shifts to step S123.

Note that the loop processing according to steps S120 to S122 indicates processing within the ODD section in which the automatic operation according to the automatic operation level 4 can be stably used.

In step S123, the automatic operation control unit 10112 notifies the driver that the transfer point for transferring the operation from the automatic operation to the manual operation is approaching. In the next step S124, the automatic driving control unit 10112 monitors the driver's behavior related to the transition from the automatic driving mode to the manual driving mode, that is, the quality of the driver's takeover operation from the automatic driving to the manual driving. Evaluation points are added or deducted to the driver according to the quality. The monitoring of the quality of the takeover operation and the calculation of the evaluation addition / subtraction points for the quality will be described later.

In the next step S125, the automatic driving control unit 10112 determines whether or not the entire itinerary set in step S100 of FIG. 7A has been completed. When the automatic operation control unit 10112 determines that the process has ended (step S125, “Yes”), the automatic operation control unit 10112 ends a series of processes according to the flowcharts of FIGS. 7A to 7C. On the other hand, when the automatic operation control unit 10112 determines in step S125 that the entire itinerary has not been completed (step S125, "No"), the process shifts to step S106 in the flowchart of FIG. 7A according to the reference numeral "B". ..

When the automatic operation control unit 10112 determines that an event requiring the intervention of the driver has occurred in the above-mentioned step S121 (step S121, “Yes”), the processing is performed according to the reference numeral “D” in the figure. The process proceeds to step S130 in the flowchart of 7C.

FIG. 7C is an example flowchart showing the correspondence to the event generated during the automatic operation by the automatic operation level 4 according to the embodiment. In step S130, the automatic driving control unit 10112 notifies the driver of the occurrence of a new event. In the next step S131, the automatic operation control unit 10112 determines the urgency of the new event. The automatic driving control unit 10112 determines the urgency according to, for example, the distance between the point where the own vehicle is traveling and the point where the new event occurs. This is substantially synonymous with determining the urgency according to the margin of time for the own vehicle to reach the new event occurrence point.

When the distance to the new event occurrence point is less than or equal to the predetermined time and the time margin is small, the automatic driving control unit 10112 determines that the urgency of the new event is high (step S131, “high”), and performs processing. The process proceeds to step S160. In step S160, MRM by the system is started, and deceleration of the own vehicle, movement to a place where the vehicle can be evacuated such as a road shoulder, and the like are forcibly executed. When MRM is started in step S160, a series of processes according to the flowcharts of FIGS. 7A to 7C are temporarily terminated.

Further, the process of shifting to step S160, which is determined to be highly urgent in step S131, is subject to the deduction target (1) described later, in which the driver's evaluation is a light deduction.

In step S131 described above, the automatic operation control unit 10112 has a distance to a new event occurrence point within a predetermined range (the distance is longer than when it is determined that the urgency is high, and the distance is longer than when it is determined that the urgency is low. (Short), and if there is some time to spare, it is determined that the urgency is medium (step S131, "medium"), and the process is shifted to step S132.

In step S132, the automatic driving control unit 10112 confirms the own vehicle and the surrounding conditions (presence or absence of a following vehicle, etc.) after securing time for deceleration in advance, and the surroundings when the traveling speed of the own vehicle decreases. Predict the impact on. In addition, the automatic operation control unit 10112 confirms the situation of the driver and observes whether or not the driver can return to the emergency manual operation. Based on this observation result, the automatic operation control unit 10112 predicts the delay of the return action of the driver from the automatic operation to the manual operation.

In the next step S133, the automatic operation control unit 10112 determines whether or not the grace time from the automatic operation to the return action to the manual operation can be extended based on the prediction result in the step S132. When the automatic operation control unit 10112 determines that the grace time can be extended (step S133, “Yes”), the process shifts to step S140. On the other hand, when the automatic operation control unit 10112 determines that the extension of the grace time is not possible (step S133, "No"), the process shifts to step S134.

Detailed control will be omitted, but in order to forcibly respond to a small time margin when a new event occurs on the itinerary that requires new manual operation due to changes in the situation during driving. Prolonging the arrival time due to sudden deceleration of the system increases the risk of secondary damage such as collision of following vehicles and induction of congestion, and is not always safe.

Therefore, it is necessary to judge the coping strategy, and review the driving plan to see if there is any impact on the road infrastructure if the vehicle decelerates in the relevant road section. The usefulness of the determination process for reviewing the driving plan will be described with reference to a specific example.

As an example, while traveling on the travel route indicated by the itinerary at the maximum allowable speed of the road section, it is difficult to continue driving at the automatic driving level 4 at a speed within the performance limit that the automatic driving system can handle. Consider the case where the system decides. In this case, if the road section is a quiet double-line road in which vehicles cruising at the same speed are not crowded around the own vehicle and the road is a straight road, the vehicle is slowly decelerated. However, deceleration can be the best choice decision, as it will not have a significant impact on road traffic.

As another example, it was found by regular monitoring in advance that the driver had no prospect of returning to manual driving due to poor physical condition, etc. Furthermore, there was a lot of traffic on the road section while driving, and there were many curves with poor visibility. If the following road section is close to the running road section, it may be safer to decelerate in advance on the straight road section.

In step S134, the automatic operation control unit 10112 urgently starts braking the MRM. For example, the automatic operation control unit 10112 issues an alarm notification in advance to notify the surroundings of the own vehicle of the start of the MRM before the start of the MRM. Further, the automatic operation control unit 10112 instructs the driver to prepare a posture (position) corresponding to the MRM. After the processing of step S134, the automatic operation control unit 10112 shifts the processing to step S160 and starts MRM by the system.

Further, the process of shifting from step S134 to step S160 is subject to deduction target (2), which will be described later, to deduct points from the driver's evaluation.

In step S131 described above, the automatic driving control unit 10112 determines that the urgency is low when the distance to the new event occurrence point is equal to or greater than a predetermined value and there is sufficient time to spare (step S131, “low”). , The process shifts to step S140.

In step S140, the automatic operation control unit 10112 adds the new event to the bird's-eye view of the ODD section and updates the bird's-eye view. In the next step S141, the automatic operation control unit 10112 notifies the driver of the new event, and observes the driver's response to this notification.

In the next step S142, the automatic operation control unit 10112 decides whether or not the driver accepts the added new event based on the driver's response observed in step S141, that is, agrees on the contract in the ODD section. Determine if it has been updated. When the driver determines that the agreement has been renewed (step S142, "Yes"), the automatic driving control unit 10112 shifts the process to step S143.

In step S143, the automatic driving control unit 10112 adds insensitive points as the driver's evaluation in accordance with the recognition of the driver's excellent notification. Then, the automatic operation control unit 10112 shifts the process to step S122 in the flowchart of FIG. 7B according to the reference numeral “E”.

This transition from step S143 to step S122 is recognized by the driver as the addition of a takeover event because the driver has made a cognitive response to the notification at his / her own will. This becomes information acting on the working memory, and appropriate processing by the driver is expected. This also applies to the case of shifting from step S149 to step S122, which will be described later.

Note that the transition of processing from step S142 to step S143 means that the driver has agreed to the contract presented by the system and consciously allowed the schedule of automatic operation. Therefore, the information regarding this contract is stored in the driver's working memory as storage information # 4 (WM14).

On the other hand, when the automatic operation control unit 10112 determines in step S142 that the driver has not renewed the agreement (step S142, "No"), the process shifts to step S144. In step S144, the automatic operation control unit 10112 observes whether or not the driver can accept the notification from the system based on the state of the driver. In the next step S145, the automatic operation control unit 10112 determines the validity of the forced return notification for urging the driver to forcibly return to the manual operation based on the observation result of the step S144. Further, the automatic operation control unit 10112 calculates the evacuation point where the influence at the time of return is minimized.

In the next step S146, the automatic operation control unit 10112 determines whether or not there is a grace time from the present time until the timing when the manual operation should be restored. When the automatic operation control unit 10112 determines that there is a grace time (step S146, “Yes”), the process returns to step S144. For example, when the driver is executing NDRA such as a nap, which has a large withdrawal from driving, soft notifications such as a display and a warning sound may not be recognized by the driver. Therefore, the automatic operation control unit 10112 repeats the processes of steps S144 to S146 until the grace time is exhausted.

If the automatic operation control unit 10112 determines in step S146 that there is no grace time from the present time until the timing for returning to manual operation (step S146, "No"), the process shifts to step S147. In step S147, the automatic operation control unit 10112 adds a return point generation based on the determination result of validity regarding the forced return notification in step S145, and informs the driver of the return point step by step. For example, the automatic driving control unit 10112 issues preliminary alarms and notifications to the driver in stages.

When notifying the driver of an unscheduled new event, if the driver is asleep at the time of the new event, the driver will have a new takeover point, the need for takeover, and a memory that is a prerequisite for urgency. Will not have at all. Therefore, in order to prevent the driver from panicking, a certain early notification and warning are given to give the driver a grace period for thinking and grasping the situation, unlike the scheduled transfer.

This is because, as described above, the return operation earlier than originally expected in the new event does not store the return point information in the driver's return necessity memory, and prompts the driver's work memory to return to manual operation. This is because the information has not been memorized yet. The limit of this judgment is the point where the RRR (Request Recovery Ratio) is high when MRM is activated, and the margin time α or more that the driver can return without causing the operation obstruction of the main road is secured. For example, when the driver is taking a nap, it is the time until he / she returns to the nap. Here, if the quality of recovery from the driver's nap is poor, the RRR is high, and the vehicle is approaching a section where there is a risk of traffic obstruction, the automatic driving control unit 10112 will move the MRM in front of it. Take proactive measures.

Note that RRR indicates the desired probability that the transfer is desired to be completed at the transfer limit point when the driver is requested to return to the manual operation.

RRR will be explained in more detail. Ideally, it is desirable for the [1/1] driver to successfully complete the takeover at the takeover limit. In indicating the success rate, RRR is defined as [1/1].

However, in reality, there are rare cases where the transfer is not successful. For example, in a certain road section, when the driver of 5 out of 10000 vehicles allows a level where the driver does not succeed, the RRR required for the road section is expressed as [1-0.405 / 1]. It will be the ratio.

This RRR includes various dynamic information defined as LDM as physical information on the road, and when a vehicle is stopped on the road section by invoking MRM for each lane of the road section, a following vehicle or the like is used. It is an index showing the success target value of takeover defined for each road section so that there is no collision accident with the own vehicle or the induction of congestion and the own vehicle does not have to stop suddenly in the middle of the road with a single lane. be. It is desirable that the RRR is used as a determinant that dynamically changes in response to changes in the temporal situation in association with the LDM.

As a specific example, in Japan, RRR is used in road sections that do not have shoulders such as shoulders, such as the Metropolitan Expressway, and in situations where the slope is already filled with first-come-first-served vehicles. , It is desirable to set RRR to [1] in those sections. On the other hand, when there is a space to evacuate to the evacuation site, or when the driver's return is unavoidable in front of the exit of the expressway where evacuation is possible on a general road, the influence on other following vehicles is minimized, but the MRM Since it is possible to select evacuation by steering to these evacuation centers, getting off to a general road and stopping, etc., it is conceivable to set the return request rate to, for example, about 0.95. Further, in a road section where the traffic volume is extremely light, RRR may be [0] as long as the influence of the emergency stop is related only to the own vehicle in the road section.

Basically, it is desirable that RRR is information that is constantly updated and provided to vehicles that use autonomous driving as part of LDM in order to suppress the obstruction of traffic by MRM to social infrastructure.

When the process of step S147 is completed, the automatic operation control unit 10112 shifts the process to step S148. In step S148, the automatic driving control unit 10112 determines whether or not the preliminary warning and notification according to step S147 have been recognized by the driver.

In step S148, the automatic operation control unit 10112 detects the driver's predetermined response to the alarm / notification (operation for the input unit 10101, specific action, etc.), and determines that the alarm / notification has been recognized by the driver. If so (step S148, “Yes”), the process is transferred to step S149. In this case, since the driver responded to the call notification at an early stage, it is possible to shift to the normal takeover process.

In step S149, the automatic operation control unit 10112 adds or subtracts insensitive points as a driver's evaluation according to the superiority or inferiority of the driver's response. Then, the automatic operation control unit 10112 shifts the process to step S122 in the flowchart of FIG. 7B according to the reference numeral “E”.

Note that the shift of the process from step S148 to step S149 means that the driver can shift to the normal takeover process by responding to the call notification at an early stage. Therefore, the information indicating that the preliminary warning and notification have been recognized is stored in the driver's working memory as storage information # 5 (WM15).

Here, the storage information # 5 stored in the working memory by the WM15 and the storage information # 4 stored in the working memory by the above-mentioned WM14 are shown by the reference numeral “C” in FIG. 7A. It is applied to the processing by WM13.

When the automatic driving control unit 10112 determines in step S148 that the alarm or notification is not recognized by the driver (step S148, "No"), the process shifts to step S150. In step S150, the automatic driving control unit 10112 determines whether or not there is a margin for waiting for recognition of the driver's return. When the automatic operation control unit 10112 determines that there is a margin for the standby (step S150, “Yes”), the automatic operation control unit 10112 returns the process to step S132 in the figure according to the reference numeral “F”.

On the other hand, when the automatic operation control unit 10112 determines in step S150 that there is no such margin (step S150, "No"), the process shifts to step S160. The transition from step S150 to step S160 is a process in which the return of the driver is timed out and the soft MRM is executed. In this case, the points will be deducted from the driver's evaluation according to the delay and negligence of the driver's return, which will be described later (3).

Here, the meaning of the processes of steps S144 to S148 described above will be described.

When the automatic driving of the automatic driving level 4 becomes available, in the section corresponding to the ODD section, the driver can be engaged in the NDRA which is more separated from the driving steering loop, for example, to take a nap or to the loading platform. You can move.

In particular, in the case of a large departure such as a nap, the ODD traveling section at the automatic driving level 4 includes, for example, a straight road and a plurality of continuous curves after traveling the straight road for a dozen minutes. Consider the case where a minor incident occurs in a straight section of the section. An example of such an incident is an insect strike where an insect hits the windshield. When an insect strike is encountered, the windshield may become dirty, which may interfere with the confirmation of forward visibility.

Here, even if the front glass is dirty due to the Insect Strike, it is safe to drive in the automatic driving level 4 on the straight road section. However, in the continuous curve section after the straight section, the road is greatly ridged, and when the front glass is dirty, there is a possibility that the state is not suitable for automatic operation of the automatic operation level 4. Therefore, it is considered that the ODD is reviewed in response to this change in the situation, and the automatic operation of the automatic operation level 4 becomes difficult. In this case, in consideration of safety, the system temporarily notifies the driver of the transfer from the automatic operation to the manual operation at a timing earlier than the normal transfer timing. Then, the system needs to observe the status of the confirmation response by the driver to the notification and take measures as described below.

That is, similar to the concept of the automatic driving level 3, all vehicles having a design function of traveling at the automatic driving level 4 can always drive at the automatic driving level 4 in the section that can be passed at the automatic driving level 4. Rather, it can be said that it is a section that can be driven at automatic driving level 4 when the conditions are met. Whether or not the driver can respond appropriately according to the conditions requires the system to provide the driver with the information necessary for making a decision with sufficient time, and the driver to take the initiative in dealing with the situation. be.

At that time, if the system has sufficient time to reach the point where a new return is required due to a change in conditions, the system forces the NDRA to be interrupted and prompts the driver to return to manual operation. What is requested is a useless return request from the viewpoint of the user who is the driver.

Actually, the driver is in a state of moving to the loading platform, taking a nap, etc., so that the interruption of NDRA is troublesome and there is no necessity for urgent return. For the driver, it is nothing more than a risk-free, unnecessary and tedious task. Therefore, repeated unnecessary requests only increase the useless feeling of the driver, and the above-mentioned filtering effect on the notification is promoted, and its importance is gradually neglected.

In order to prevent unnecessary early confirmation and prevent excessive risk, a determination is made in step S146 to prepare for the return procedure at a certain margin and at the timing of issuing a notification or an alarm. Then, depending on whether the notification or the alarm is recognized by the driver, the determination of the countermeasure when the driver's response is delayed, such as whether to start the process equivalent to the normal return procedure, is performed in step 148. ..

The processes of steps S144 to S148 described above are for realizing such control.

The standard of the grace time in step S146 is parameterized for general passenger vehicles, vehicles loaded with heavy hazardous materials, large shared vehicles, etc., such as a safety coefficient according to the characteristics of the vehicle and a target value of RRR obtained in the road section. It may be operated as a reference value.

The information presented to the driver by the system via the information display unit 120 or the like is taken into the driver's working memory as risk judgment information by the driver, and is taken out from the working memory according to the awareness of the importance of taking over for operation. It will encourage people to judge their actions.

<3-2-3-2. Evaluation of driver's return behavior>
Here, the evaluation of the driver's return behavior according to the embodiment will be described more specifically. First, the deduction targets (1) to (3) for deducting the evaluation value of the driver when the process shifts to the step S160 for starting MRM will be described with reference to Table 2.

In the example of Table 2, in the point deduction target (1), points are deducted [-1] for a single occurrence, and points are deducted [-2] for repeated occurrences within the same itinerary. In the point deduction target (2), points are deducted [-4] for a single occurrence, and points are deducted [-4] for repeated occurrences within the same itinerary. Further, in the point deduction target (3), a deduction [-5] is applied to the occurrence of a single unit, and a deduction [-5] is applied to the occurrence of a single item repeatedly within the same itinerary.

The point deduction target (1) is the deduction when shifting from step S131 to step S160, and the degree of deduction is the deduction in an imminent situation. In this case, it is a response to an event that occurred immediately before the own vehicle on the road on which the own vehicle travels without prior notice, and does not attribute to the direct responsibility of the driver. However, if the start of MRM is predicted by the situation judgment when using automatic operation, a deduction is applied so that the use of automatic operation depending on the system is not repeated (third degree of deduction). Further, in the point deduction target (1), for example, as a deduction with a temporary conditional flag, the deduction can be canceled if it is not reapplied for a certain period of time.

The point deduction target (2) is a deduction when shifting from step S134 to step S160, and as a degree, it is a deduction in a situation where there is a little time to spare. In this case, even if the system slows down the traveling speed of the vehicle and extends the arrival time to the point where it is essential to take over, the recovery to manual operation becomes insufficient due to the driver's cause such as negligence, and MRM. Is an example of starting. In this case, since the responsibility for starting MRM lies with the driver, a heavier deduction than the above-mentioned deduction target (1) is imposed (second degree deduction).

The point deduction target (3) is the deduction when shifting from step S150 to step S160, and as a degree, it is a deduction in a situation where there is originally sufficient time to spare. In this case, it should be used with plenty of time, and it is a transfer that can be completed if it returns early. As a system, MRM is executed by a method that is soft and has less influence on the surroundings. However, in order to prevent the driver from neglecting the takeover behavior and to encourage behavioral changes such as taking prompt action, a heavier deduction than the above-mentioned deduction target (2) is imposed (first degree). Deduction).

Next, an example of evaluation for the driver at the time of a normal transfer request from the system to the driver (also referred to as Request to Intervene or Transition Demand) will be described using Table 3. The evaluation exemplified in this Table 3 is performed, for example, in step S124 of FIG. 7B, but the evaluation is not limited to this, and the evaluation according to this Table 3 can be performed at other timings of takeover or at other timings.

In Table 3, the first 1st to 4th rows are examples of adding points to the driver's evaluation, the 5th row is an example of not adding points and deducting points to the driver's evaluation, and the 6th and subsequent rows are , This is an example of deducting points from the driver's evaluation.

According to Table 3, as an example of adding points, if the driver selects a break / rest at an early stage, or if a detour is selected and the transfer at the transfer point is abandoned in advance, the driver When requesting assistance by leading vehicle, remote control or remote operation at an early stage, and when taking over is started by the return warning sound or the driver's voluntary confirmation of the situation (autonomous generation of return feeling), respectively. Points are added [+0.2] regardless of whether it occurs once or repeatedly within the itinerary. In addition, when there is recognition detection by the driver in advance notification of the return request, points are added [+0.1] regardless of whether it occurs once or repeatedly in the itinerary.

When the driver starts the return operation in response to the return notification, neither points are added nor points are deducted as it is a normal return operation.

On the other hand, as an example of deducting points, when the driver starts returning with a return alarm, the driver disregards the situation, deducting points [-0.2] for a single occurrence, and 2 for repeated occurrences within the same itinerary. The points are doubled. This deduction has the meaning of preventing the driver from neglecting the situation and postponing high-priority processing. If there is no recognition detection by the driver in advance notification of the return request (leaving the notification not in memory), points will be added as a malicious measure regardless of whether it occurs once or repeatedly in the itinerary [-0.5] It is supposed to be. When the driver starts returning due to a compulsory return request, the deduction is [-1.0] for a single occurrence and double the deduction for repeated occurrences within the same itinerary as a lack of sense of risk.

In addition, if the system barely achieves the takeover by decelerating in advance and generating a time grace on the main road, a deduction [-2.0] will be given for a single occurrence, and 1.5 times that for repeated occurrences within the same itinerary. It is a deduction of points. Similarly, on a highway, if the driver cannot handle the takeover, that is, if the takeover fails and the system executes MRM, a deduction [-4.0] will be given for a single occurrence, and 1.5 times that for repeated occurrences within the same itinerary. It is a deduction of points. This deduction has the meaning of suppressing the driver's conviction violation.

In addition, on low-speed non-main roads, the driver cannot take over when the system decelerates in advance to generate a time grace and achieve the takeover, and on low-impact roads (such as roads with extremely low traffic volume). That is, if the takeover fails and the system executes MRM, the points are deducted [-0.5] regardless of whether they occur once or repeatedly within the same itinerary.

Furthermore, regarding the use of NDRA, if the driver starts NDRA without confirming the application status of ODD, a deduction [-2.0] will be given for a single occurrence, and a double deduction for repeated occurrences within the same itinerary. However, when NDRA is used outside the ODD, this is a violation, and the deduction is [-3.0] for a single occurrence and double the deduction for repeated occurrences within the same itinerary.

The system (automatic driving control unit 10112) accumulates the addition / subtraction points shown in Table 3 for the same driver and uses it as the evaluation value of the driver. The system accumulates the addition and subtraction points of the driver, for example, for all the itineraries set and executed by the driver in the system, or for the itineraries carried out within a predetermined period. In this way, by imposing a pay-as-you-go penalty according to the driver's history, even though the takeover request is issued from the system, it neglects to deal with it. This evaluation result is reflected in the control to prevent malicious use such as repetition.

The system can give a penalty to the driver's use of autonomous driving when the driver's evaluation value is low (for example, the evaluation value is a negative value).

As an example of a penalty for drivers, there are restrictions on the use of autonomous driving. Regarding restrictions on the use of this automatic driving, for example, processing to delay the scheduled arrival time to the destination, forced stop-by to the service area, running lock for a certain period of time, restriction on the upper limit speed of running, use of the automatic driving mode Restrictions (same day, applicable week or month), restrictions on the section of automatic operation mode, etc. can be considered. These usage restrictions give the driver an intuitive sense of loss (risk), and have the effect of encouraging early return to manual driving and appropriate measures.

Another example of a penalty for a driver is the restriction on the use of secondary tasks (NDRA) that the driver engages in while using autonomous driving. This usage restriction gives the driver an intuitive sense of loss (risk), and can obtain the effect of prompting the driver to return to manual operation early and take appropriate measures.

Regarding the usage restriction of this secondary task, for example, the usage restriction of the terminal device used by the driver for the secondary task can be considered. As the usage restriction for the terminal device, it is conceivable to fill the screen displayed on the terminal device and to erode using an arbitrary image for the screen. According to these, for example, the driver can be made aware of the risk by a stepwise advance notice that appeals to the driver's intuition. Further, it is conceivable to replace the screen in use of the terminal device with the takeover information window (replacement of the sub screen and the parent screen). According to this, for example, the driver can be alerted to the takeover of driving.

Furthermore, it is conceivable that the screen of the terminal device is forcibly frozen, and the work performed by the driver using the terminal device is invalidated retroactively. These can give the driver a feeling that the work up to that point is wasted when the NDRA is forcibly performed by the forced interruption of the operation, and can further call attention to the takeover of the operation.

For control of these terminal devices, for example, application software for using the system according to the embodiment (presentation of a bird's-eye view of the itinerary, advance notification of the end of the ODD section to the driver, etc.) is installed in the terminal device. It can be realized as a function of the application software.

Note that the values of points added and deducted described using Tables 2 and 3 are examples, and are not limited to the above examples. Further, each example of adding points and deducting points is also an example, and is not limited to the above-mentioned example.

<3-2-3-3. About the bird's-eye view display of the itinerary applicable to the embodiment>
Next, the bird's-eye view display of the itinerary applicable to the embodiment will be described more specifically.

FIG. 8 is a schematic diagram schematically showing an example of a bird's-eye view display of the itinerary applicable to the embodiment. In FIG. 8, the bird's-eye view display 50 includes a short-distance display unit 51a, a medium-distance display unit 51b, and a long-distance display unit 51c. In FIG. 8, the traveling direction of the own vehicle is shown from the lower end side to the upper end side. In FIG. 8, the lower end portion is the current position of the own vehicle, but this is not limited to this example. Further, the icon 52 indicating the own vehicle is for facilitating the image of the itinerary, and the display can be omitted.

In FIG. 8, the short-distance display unit 51a displays a section from the current position of the own vehicle to a predetermined first distance. The first distance is, for example, a distance of about 15 minutes from the own vehicle in terms of traveling time. In the example of FIG. 8, in the short-distance display unit 51a, the vertical position on the screen and the actual distance can be in a linear relationship.

The medium-distance display unit 51b has a shape in which the width is narrowed according to the height on the screen so as to converge at the point at infinity VP from the upper end having the width W ₁ of the short-distance display unit 51a. In the medium-distance display unit 51b, the vertical position on the screen and the actual distance are regarded as a non-linear relationship, and the change in the actual distance with respect to the position on the screen is increased as, for example, upward on the screen. Can be done.

Here, when the position in the vertical direction is the arrival time along the time in the traveling direction in FIG. 8, the reciprocal of the distance h _diff from the point at infinity VP of the display can be displayed in proportion to the traveling time. By displaying the medium-distance display unit 51b with a sense of perspective in this way, it is possible to efficiently present the display of the arrival time on a narrow screen. By accurately displaying the degree of influence of each takeover point or the like through the display form of the bird's-eye view display 50, the driver can intuitively grasp the time at each arrival point.

On the other hand, the long-distance display unit 51c is extended from the position of the width W ₂ in front of the point at infinity VP while maintaining the width W ₂ . Similar to the short-distance display unit 51a described above, the long-distance display unit 51c can have a linear relationship between the vertical position on the screen and the actual distance.

Further, it is assumed that all the sections shown in FIG. 8 are sections that can be automatically operated at the automatic operation level 4. Here, it is assumed that there is a section in the section in which it is preferable for the driver to return to manual driving due to reasons such as a narrow road width or a railroad crossing. In such a section, it is considered that the RRR (return request probability) for the driver is high, and if the driver does not properly return to manual driving, it may cause social harmful effects such as influence on the following vehicle. There is.

Therefore, in the bird's-eye view display 50, information indicating the section where the RRR becomes high is displayed. For example, a caution display 53 that narrows the road width is displayed for such a section. With this caution display 53, it is possible to call attention to the driver. In addition, a transfer start recommended point can be provided to the driver by a predetermined distance in front of the section, and the section 56 recommended for transfer can be highlighted and displayed.

In the embodiment, in order to make it easier for the driver to return from automatic operation to manual operation, a section display indicating a recommended operation mode for each section is performed with respect to the bird's-eye view display 50 shown in FIG. To add. The bird's-eye view display 50 to which this section display is added will be described with reference to FIGS. 9A to 9C.

FIG. 9A is a schematic diagram showing an example of a bird's-eye view display 50a in which each section is color-coded according to the embodiment. In FIG. 9A, the bird's-eye view display 50a shows the automatic operation possible section 53a, the return posture maintenance section 53b, and the operation return essential section 53c by color coding.

The section 53a where automatic driving is possible indicates a section where automatic driving is possible according to the automatic driving level 4, and is displayed in green, for example, to give an image of safety and security. The return posture maintenance section 53b is a section immediately before returning from the automatic driving to the manual driving, and indicates a section in which the driver is desired to maintain the posture for returning to the manual driving. The return posture maintenance section 53b is displayed in yellow, for example, to call attention to the driver. The operation return required section 53c indicates a section in which manual driving by the driver is required, and is displayed in red, for example, indicating caution.

The above-mentioned color coding of green, yellow and red is an example, and is not limited to this color combination. Further, as long as each section can be clearly distinguished, a single color may be used without color coding.

In this way, by changing the display method according to the section and the distance from the own vehicle, the driver can easily grasp the timing when the automatic driving should be returned to the manual driving.

FIG. 9B is a schematic diagram showing an example of the bird's-eye view display 50b configured in an annular shape according to the embodiment. In the example of FIG. 9B, the top of the head in the display of the ring is the position of the own vehicle, and the display is such that the distance from the own vehicle increases clockwise (rightward) from that position as the starting point. Also, as the distance from the vehicle increases, the width of the display is narrowed to emphasize the sense of distance.

The bird's-eye view display 50b configured in an annular shape in this way is suitable for display in a narrow area such as a display screen of a wearable device such as a wristwatch type.

FIG. 9C is a schematic diagram showing an example of the bird's-eye view display 50c including road information according to the embodiment. The bird's-eye view display 50c shown in FIG. 9C is an example in which road information such as an icon 54a corresponding to a traffic sign and an icon 54b indicating a facility is added to the bird's-eye view display 50a shown in FIG. 9A. The icon 54a indicates, for example, a location and content that the driver should pay attention to in an automatically driven vehicle. In this example, the icon 54a is a display imitating a traffic sign actually installed on the road. The icon 54b indicates a facility required for the vehicle to travel, and is displayed corresponding to a point such as a gas station, a parking area, or a service area.

Further, in FIG. 9C, sections such as a congested section where the time for passing greatly fluctuates are shown as section displays 55a and 55b.

In this way, by using the bird's-eye view display 50c with road information added, the risk information at each approach time is taken into the visual field as the driver progresses to the memory of the above-mentioned consciousness, which is of high importance, that is, At high-risk points, it is a stimulus that acts on the driver's work memory when making behavioral decisions. Therefore, the driver can predict the timing to return from the automatic operation to the manual operation earlier, and the return to the manual operation is smoother than the case where only the monotonous course display is uniformly presented. It will be possible to execute.

The bird's-eye view display 50c and the bird's-eye view display 50a shown in FIG. 9A can be displayed on the screen of the terminal device used by the driver when using the automatic driving system according to the embodiment, for example. For example, the display of the bird's-eye view display 50a or the bird's-eye view display 50c is controlled by operating the application program related to the information processing program according to the embodiment mounted on the terminal device on the CPU 10010. At this time, it is conceivable that the bird's-eye view display 50a or the bird's-eye view display 50c is displayed, for example, on the right end or the left end of the screen in a state where the width direction is compressed. Not limited to this, the bird's-eye view display 50a or the bird's-eye view display 50c may be displayed over two sides sharing the vertices of the screen, or may be displayed over three sides of the screen or around the screen.

<3-2-4. HCD control configuration example according to the embodiment>
Next, an example of the control configuration of the HCD according to the embodiment will be described more specifically. FIG. 10 is a functional block diagram of an example for explaining the function of control by HCD in the automatic operation control unit 10112 according to the embodiment. In FIG. 10, among the functions of the automatic operation control unit 10112, the function for realizing the control by the HCD is focused on, and the other functions are omitted as appropriate.

In FIG. 10, the automatic driving control unit 10112 includes an HMI 100, a driver return delay evaluation unit 101, a travel path advance predictability acquisition range estimation unit 102, a remote support control / steering support support availability monitoring unit 103, and a driver. It includes a behavior change achievement level estimation unit 104, a vehicle road performance information provision unit 105, an ODD application estimation unit 106, an automatic driving use permission integrated control unit 107, and a driver behavior quality evaluation unit 108. Each of these parts is configured and realized as, for example, a module on the RAM 10012 which is the main storage device by operating the information processing program according to the embodiment on the CPU 10010.

The HMI 100 realizes an interface for the driver, and is connected to, for example, an information display unit 120, a terminal device 121, a vehicle interior light source 122, an acoustic device 123, and an actuator 124.

The information display unit 120 performs a predetermined display according to a command from the HMI 100. The terminal device 121 may be a terminal device brought into the vehicle by the driver, or may be a terminal device installed in the vehicle in advance. The HMI 100 can communicate bidirectionally with the terminal device 121. The terminal device 121 can accept the user operation, and supplies the control signal corresponding to the accepted user operation to the HMI 100. Further, the terminal device 121 displays a predetermined screen on the display device of the terminal device 121 in accordance with the command from the HMI 100. The vehicle interior installation light source 122 is a light source installed in the vehicle interior, and the lighting / extinguishing and the amount of light are controlled by the HMI 100.

The audio device 123 includes a speaker and a buzzer, and a drive circuit for driving the speaker and buzzer. The sound device 123 emits a sound according to the control of the HMI 100. Further, the microphone can be included in the acoustic device 123. The sound device 123 converts an analog sound signal based on the sound picked up by the microphone into a digital sound signal and supplies it to the HMI 100.

The actuator 124 drives a predetermined part in the vehicle according to the control of the HMI 100. For example, the actuator 124 applies vibration such as haptic vibration to the steering. Further, another actuator 124 can control the reclining of the seat on which the driver sits according to the command of the HMI 100.

The HMI 100 is based on information from the travel path advance predictability acquisition range estimation unit 102, the remote support control / steering support support availability monitoring unit 103, and the ODD application estimation unit 106, which will be described later, and these information display units 120, terminal device 121, and vehicle. It controls the operation of the indoor light source 122, the acoustic device 123, the actuator 124, and the like. As a result, the following visual and auditory notifications can be given to the driver.

-Advance notification by guidance sound At this time, the guidance sound gives an excessive stimulus that can be easily recognized by a person, such as an in-flight chime sound (for example, a "pawn" sound when a seatbelt wearing sign). It is preferable to use no sound. The notification by the guidance sound can be applied to, for example, the advance notification of the return operation from the automatic operation to the manual operation. For example, the system may use this guidance sound notification when presenting an agreement to a "contract" to the driver.

-Notification for requesting return to manual operation At the time of the request, the HMI 100 can perform an auditory notification by the sound emitted by the acoustic device 123 and a visual notification by the display of the information display unit 120. .. Further, the HMI 100 may drive the actuator 124 to give a haptic vibration to the steering to give a tactile notification at the time of the request. Further, the HMI 100 can instruct the driver to point and call the front of the road.

-Warning and Warning The HMI 100 can give a warning or warning to the driver audibly, visually, or tactilely. For example, the HMI 100 can control the acoustic device 123 to emit a warning sound and give an audible warning. In this case, it is conceivable to use a louder stimulating sound as the warning sound as compared with the above-mentioned induction sound. Further, the HMI 100 can control the information display unit 120 and the light source installed in the vehicle interior to give a visual warning by blinking a red light, emitting a warning in the vehicle interior, or the like. Further, the HMI 100 can control the actuator 124 to strongly vibrate the seat in which the driver sits to give a tactile warning.

-Penalty The HMI 100 can control the driver to perform a penalty operation. For example, the HMI 100 can perform controls that may be offensive to the driver, such as visuals, operational restrictions, mild pain to the driver, blowing cold air, and moving the driver's seat forward. .. Further, the HMI 100 performs pseudo control such as occurrence of rolling of the vehicle, acceleration / deceleration that causes discomfort, pseudo deviation of the lane, etc., and directly or directly urges the driver to return early. You can give a penalty that affects later rather than the field. Furthermore, the HMI100 presents fine information, compulsory entry into service areas as penalties and presentation of restraint time at that time, notification of unavailability due to penalties for autonomous driving, and warning presentation for next or repeated usage restrictions. It is possible to give a penalty according to the driver's knowledge information.

The driver return delay evaluation unit 101 evaluates the delay of the driver's return from automatic driving to manual driving, for example, a life log data information server 130, a wearable device log data 131, and a face / upper body / eyeball. The camera 132, the biometric information index acquisition unit 134, the vehicle interior localizer 135, and the response evaluation input unit 136 are connected. Further, the driver return delay evaluation unit 101 acquires information indicating the return characteristic of the individual driver to manual operation from the remote remote server dictionary 137.

The wearable device log data 131 is log data acquired from the wearable device when the driver wears the wearable device. The wearable device log data 131 includes, for example, a driver's behavior history and biometric information.

The face / upper body / eyeball camera 132 is a camera provided in the vehicle interior so as to image the upper body including the driver's head. The face / upper body / eyeball camera 132 is provided in the vehicle interior so as to be able to capture the driver's facial expression, fine movement of the eyeball, and the behavior of the upper body. The face / upper body / eyeball camera 132 is not limited to this, and may include a plurality of cameras that image the face, the eyeball, and the upper body, respectively. The body posture / head camera 133 is provided in the vehicle interior and is a camera that captures the body posture including the driver's head. By analyzing the images captured by the body posture / head camera 133 in chronological order, the body posture of the driver and the position and orientation of the head can be tracked.

In the embodiment, the camera is described so that the face / upper body / eyeball camera 132 and the body posture / head camera 133 are separated for convenience from the degree of freedom of installation. A device in which these cameras are integrated may be used without limitation.

The biometric information index acquisition unit 134 acquires the biometric information of the driver based on the outputs of various sensors provided in the vehicle, for example. As the acquired biological information, respiration, pulse, exhalation, body temperature distribution, electrooculogram, and the like can be considered. Not limited to this, the biometric information index acquisition unit 134 can also acquire a part of the biometric information of the driver from the wearable device worn by the driver.

The vehicle interior localizer 135 is a localizer provided in the vehicle interior. The response evaluation input unit 136 inputs a response by the driver to a request, a warning, or the like presented to the driver by the HMI 100.

The information acquired by the wearable device log data 131, the face / upper body / eyeball camera 132, the biological information index acquisition unit 134, the vehicle interior localizer 135, and the response evaluation input unit 136 can be used as a driver's life log. It is stored in the life log data information server 130.

FIG. 11 is a functional block diagram of an example for explaining the function of the driver return delay evaluation unit 101 according to the embodiment. In FIG. 11, the driver return delay evaluation unit 101 includes a driver behavior response evaluation unit 1010, a correlation characteristic learning unit 1011, a condition-based return distribution individual characteristic / situation recognition decrease characteristic dictionary 1012, and a situation recognition decrease transition prediction unit. Includes 1013 and.

Conditional return distribution individual characteristic / situational awareness deterioration characteristic dictionary 1012 is a dictionary regarding the observable evaluation value of the individual driver and the characteristic of situational awareness deterioration. The correlation characteristic learning unit 1011 includes information indicating the return characteristics of the individual driver to manual operation acquired from the remote remote server dictionary 137, and evaluation values and situations acquired from the condition-specific return distribution individual characteristics / situation recognition deterioration characteristic dictionary 1012. Based on the recognition deterioration characteristic, the correlation characteristic between the observation evaluation value of the individual driver and the return delay time distribution is learned.

In the embodiment, the remote remote server dictionary 137 is arranged in the remote remote server outside the vehicle, but this is not limited to this example. That is, the reason why the remote remote server dictionary 137 is installed on an external server is that the use of the driver's characteristics that are not necessarily tied to the unique vehicle due to the spread of business vehicles and shared cars is one example of use. The remote remote server dictionary 137 may be installed in the vehicle to be used.

The driver behavior response evaluation unit 1010 acquires each information about the driver from the HMI 100. For example, the driver behavior response evaluation unit 1010 acquires advance preliminary information from the HMI 100 by using a life log. Further, the driver behavior response evaluation unit 1010 acquires, for example, the following information regarding the driver from the HMI 100 based on the images of the face and body acquired by various sensors (cameras).
-Facial expression and body posture recognition information.
・ Information about the eyes. In this example, an evaluation is obtained for the local behavior of the eye such as PERCLOS (percentage of closed eyes per unit time) and saccade (rapid eye movement).
・ Posture and transition of posture. In this example, the quality of the return behavior is evaluated based on the posture and the transition of the posture.
・ Position and posture of leaving the seat in the passenger compartment.
-Biological information.

The driver behavior response evaluation unit 1010 evaluates the driver behavior response based on each information acquired from the HMI 100 and the correlation characteristic acquired from the correlation characteristic learning unit 1011. The evaluation result is passed to the situational awareness decline transition prediction unit 1013. Based on this evaluation result, the situational awareness decline transition prediction unit 1013 predicts the transition regarding the decline in situational awareness by the driver.

If the life log data information server 130 is available, a part of the life log data acquired from the life log data information server 130 can be input to the driver behavior response evaluation unit 1010. The driver behavior response evaluation unit 1010 uses the driver's life log data to estimate the driver's arousal level, etc., such as lack of sleep time in advance, overwork accumulation, apnea syndrome, and alcohol residue of drinking. The accuracy is improved, and the judgment accuracy of the driver's situation awareness ability is improved, which enables safer control against sudden occurrence of drowsiness such as microsleep.

As described above, the driver return delay evaluation unit 101 includes the life log data information server 130, the wearable device log data 131, the face / upper body / eyeball camera 132, the biometric information index acquisition unit 134, and the response evaluation input unit. It has a function as an acquisition unit that acquires the state of the driver based on the information acquired from 136 and the remote remote server dictionary 137.

The explanation returns to FIG. 10, and the travel path pre-predictability acquisition range estimation unit 102 acquires the high freshness update LDM 140, and estimates the acquisition range of the advance predictability for the travel path based on the acquired high freshness update LDM 140. That is, the travel path pre-predictability acquisition range estimation unit 102 acquires a range in which the event can be predicted in advance on the travel path based on the high freshness update LDM 140.

FIG. 12 is a schematic diagram for explaining the high freshness update LDM140 applicable to the embodiment. In each region, regional distributed arrangements LDM1400a, 1400b, ..., 1400n are arranged. These regional distributed layouts LDM1400a, 1400b, ..., 1400n correspond to each region, such as a stationary sensor 1401, a dedicated probe car information 1402, a general vehicle ADAS information 1403, a weather information 1404, and an emergency notification information 1405 (fall of dangerous goods). Etc.), etc., and will be updated from time to time.

The regional distributed layout LDMs 1400a to 1400n are transmitted by 5G (fifth generation communication), 4G (fourth generation communication), or another communication method. The transmitted regional distributed arrangement LDMs 1400a to 1400n are received, for example, by the automatic operation control unit 10112, and are aggregated to form the high freshness update LDM140. Further, the emergency notification information 1405 is delivered by the broadcast 1406 and is received by the automatic operation control unit 10112 directly to the automatic operation control unit 10112 or included in the above-mentioned 5G or 4G communication. The automatic operation control unit 10112 updates the high freshness update LDM 140 based on the received emergency call information 1405.

The explanation returns to FIG. 10, and the travel path pre-predictability acquisition range estimation unit 102 describes the range in which the event can be predicted in advance by the acquired high freshness update LDM 140 on the travel path based on the information and the situation exemplified below. presume.

Here, a large amount of investment in environmental infrastructure and dedicated probe cars is possible in the center of urban transportation. On the other hand, in areas where the investment effect is small and in the usage time zone, there are situations where it depends mainly on sporadic data collected in shadow mode from ADAS information 1403 of general vehicles, and advance for the route where the high freshness update LDM140 can be provided. Predictability is live information that actively changes with time according to the deployment of regional distributed LDMs, which will be described later, and the allowable communication band of the communication network.

-Preliminary information before the itinerary for the section provided by the high freshness update LDM140.
-High freshness update Risk information that the update frequency of LDM140 decreases. The update frequency of the high freshness update LDM140 varies with the passage of time, for example, due to the passing presence density of a probing vehicle (for example, a dedicated probe car).
・ Information omission due to lack of regional wireless communication band.
・ Insufficient passage of probing vehicles to supplement information against deterioration of predictability due to bad weather.
・ Reduced predictability due to temporary lack of information that occurs in information acquired from leading vehicles / vehicle groups that replaces and complements itinerary sections that lack information that are not provided by updated LDM due to infrastructure development.

The remote support control / steering support availability monitoring unit 103 monitors the availability of remote support control and the availability of steering support based on the information acquired from the remote support control I / F 150. The monitoring by the remote support control / steering support availability monitoring unit 103 is expected to be used as an option such as regional traffic, platooning support, and limited connection support.

FIG. 13 is a schematic diagram for explaining the acquisition of information by the remote support control I / F 150, which is applicable to the embodiment. The remote control commanders 1500a, ..., 1500n-1 collect information from the standby steering operator 1501 and the standby steering operator 1501n + 1, respectively, and the dedicated lead guidance contract vehicle 1502. Further, in this example, the remote control commander 1500a also collects information from the leading guidance vehicles 1503 m and 1503 m + 1. The remote control commanders 1500a, ..., 1500n-1, respectively, transmit the collected information to the remote support control I / F 150 using communication by, for example, 5G. The communication method here is not limited to 5G and may be 4G. Further, in the example of the figure, the guided leading vehicles 1503m and 1503m + 1 directly transmit the collected information to the remote support control I / F 150.

The explanation returns to FIG. 10, and the remote support control / steering support support availability monitoring unit 103 performs the following processing based on the information acquired from the remote support control I / F 150.
-Control support using remote control support services when it is difficult for the driver to return from automatic driving to manual driving. This includes control on behalf of the driver by the system, such as early evacuation behavior and operator assignment control.
-Maneuvering control by a remote operator for vehicle steering control. This is a control command when an external remote control contract is concluded and operator assignment is possible.
-Monitoring the execution of remote support and monitoring of fallback (degenerate running) when a problem occurs.

The driver behavior change achievement level estimation unit 104 estimates the achievement level of the change in the driver's behavior change with respect to the system limit. FIG. 14 is a functional block diagram of an example for explaining the function of the driver behavior change achievement level estimation unit 104 according to the embodiment. The driver behavior change achievement level estimation unit 104 includes an excellent return steering behavior evaluation point addition unit 1040 and a penalty behavior cumulative addition recording unit 1041.

The excellent return steering behavior evaluation point addition unit 1040 adds the evaluation value for the excellent driving operation when returning to the manual operation. The penalty action cumulative addition recording unit 1041 deducts the evaluation value according to the violation of the return request to the manual operation from the system and the negligence of dealing with the return request. Further, the penalty action cumulative addition recording unit 1041 cumulatively adds the evaluation value to the action in which the penalty occurs. The driver behavior change achievement level estimation unit 104 can acquire the evaluation value from, for example, the driver return delay evaluation unit 101.

The explanation returns to FIG. 10, and the own vehicle travel road performance information providing unit 105 provides the own vehicle travel (passing) road performance information to the LDM regional cloud (for example, the regional distributed arrangement LDM1400a to 1400n). Specifically, the own vehicle travel road performance information providing unit 105 provides the following information and the like.

・ Changes / differences to the map information acquired in advance, notification of abnormal information, and risk information to the following vehicle even after entering the section.
・ When an abnormality / danger risk (falling object, accident, disaster, etc.) is detected while driving, the detected information is automatically or manually transmitted from the own vehicle as emergency information.
-Characteristic event notification (event notification by driver / vehicle user), suspicious risk information.
-Providing information by manual notification by the driver instead of automatic notification. In this case, the temporarily recorded road environment information from the time several minutes before the report is notified.
-Uploading a probing request from the server. For example, we receive an emergency call for the risk of falling objects of unknown details from the vehicle in front and confirm the details. Therefore, a detailed scanning request is issued to the LDM cloud server that manages the section, and the information obtained by the detailed scan by the environmental grasp enhancement scan, which is equivalent to the environmental scan during normal driving or has a faster refresh rate, is uploaded. ..

The own vehicle travel path performance information providing unit 105 can optionally provide information to the following vehicle, the waiting vehicle, etc. based on the information possessed by the own vehicle when it is difficult to provide the LDM.

For example, if the traffic volume passing through a section is light and it is difficult to provide a constantly updated LDM with the acquired upload information, or if the infrastructure is insufficiently equipped with an LDM-equipped cloud, the infrastructure will automatically update the freshness based on the LDM 140. Automatic driving by driving level 4 cannot be expected.

In such a case, the driving is switched from automatic driving to manual driving, and pairing is performed for the supported vehicle (for example, the following vehicle or the standby vehicle) that requires assistance. Then, the environment acquisition data when the own vehicle is driving at the automatic driving level 2 or lower is provided to the supported vehicle as data necessary for driving at the automatic driving level 4, and further, the supported vehicle is specified. LDM is provided by pairing with the vehicle, and information is provided when the vehicle behind is following and traveling in the state of automatic driving level 4.

For example, the own vehicle track record information providing unit 105 can provide this information in cooperation with the above-mentioned remote support control / steering support support availability monitoring unit 103. For example, when the paired partner is a leading support support vehicle, information can be provided as road guide information to the following and following vehicles of the partner. The operation that combines these leading vehicles and remote support together with the high freshness update LDM140 will be further found to be useful especially when using platoon transportation including unmanned traveling vehicles, and the driver will operate the actual vehicle. It may be applied to use without boarding.

The ODD application estimation unit 106 determines whether or not the vehicle can travel at each automatic driving level (ODD section). The ODD application estimation unit 106 makes this determination based on the following information.

-History evaluation information such as driver's excellent credit evaluation, return violation, deductions, penalties, etc.
-Evaluation information on the degree of understanding and proficiency of the driver's need to return based on HCD.
-Information indicating the acquisition status of the high freshness update LDM140.
-Information on the return request probability (RRR) based on LDM such as the high freshness update LDM140, and information indicating the points that can be selected by the evacuation options.
-Information indicating the limits of the application of autonomous driving based on the diagnosis results of vehicle-loaded equipment.
-Information showing vehicle dynamics (loading passengers, luggage, and risk of collapse).

Further, the ODD application estimation unit 106 has an ODD section to which non-monitoring automatic operation at automatic operation level 4 can be applied according to LDM such as high freshness update LDM140 and other update status, and automatic operation equivalent to automatic operation level 3. Estimate the available ODD intervals. In addition, the ODD application estimation unit 106 reviews and updates the applicable ODD section according to the risk information newly acquired during the travel itinerary, the adhesion of dirt on the equipment, the change in the state of the driver, and the like. At this time, the ODD application estimation unit 106 notifies the driver of the information update through the HMI 100, and evaluates the degree of understanding of the situation change based on the driver's response to the notification.

The automatic driving use permission integrated control unit 107 controls the automatic driving use permission in an integrated manner. For example, the automatic driving use permission integrated control unit 107 controls the automatic driving permission status for each traveling section in an integrated manner. Further, the automatic driving use permission integrated control unit 107 controls the execution of the MRM. Further, the autonomous driving use permission integrated control unit 107 controls to give penalties and penalties to the driver, such as forcibly suspending the use of the violation act while using the automatic driving. Examples of violations include delays in the driver's response to requests from the system to return to manual driving, and repeated and continuous use of automatic driving according to automatic driving level 3.

The driver behavior quality evaluation unit 108 evaluates the quality of behavior (behavior quality) of the driver during automatic driving.

The driver behavior quality evaluation unit 108 evaluates the quality of the driver's behavior based on, for example, the stability of the driver's steering. The driver behavior quality evaluation unit 108 evaluates each item related to driving such as steering operation, accelerator and brake operation, and winker operation by the driver, for example. In addition, the driver behavior quality evaluation unit 108 evaluates the designated operation such as pointing and calling by the driver and the behavior in response to the transfer request from the system to the manual driving. Further, the posture return evaluation may be performed when returning to the driving steering posture from the NDRA task in which the posture is broken.

One of the information that is difficult to directly observe by the system when realizing HCD control is grasping the situation awareness (Situation Awareness), which is the information in the driver's brain. Therefore, in the HCD according to the embodiment, attention is paid to the steering behavior in a state where the situational awareness is lowered. For example, in a state where situational awareness is insufficient, that is, steering behavior in a state where situational awareness is deteriorated, intelligent feedback becomes insufficient and steering due to excessive reflection increases. In other words, paying attention to the fact that steering that would normally be performed smoothly becomes excessive steering in which feedback is not performed correctly when situational awareness is reduced, steering during automatic driving is replaced with manual driving during normal times. It is used as an evaluation index for the driver's situational awareness as compared with steering by.

<3-3. About automatic operation level 4 applicable to the embodiment>
Here, the automatic operation level 4 applicable to the embodiment will be described.

<3-3-1. Basic structure>
First, the basic structure of the automatic operation level 4 will be described. FIG. 15 is a schematic diagram for explaining the basic structure of the automatic operation level 4 applicable to the embodiment.

In FIG. 15, each of the charts (a) to (g) has a horizontal axis as a position. The chart (a) shows an example of the relationship between the return time ΔT _drd and the position (or the arrival elapsed time calculated from the vehicle speed), and the chart (b) shows an example of the relationship between the grace time ΔT _{2 lim} and the position. .. The return time ΔT _drd and the grace time ΔT _{2 lim} will be described later.

Chart (c) shows an example of a constantly updated (high freshness update) LDM data section. In this example, in the runnable section in the automatic operation of the automatic operation level 4 (described as Level 4 in the figure), for example, an LDM data section in which the freshness is deteriorated without being updated is included. As shown in the chart (b) as section 64, this freshness-reduced LDM data section is a section in which LDM maintenance is insufficiently announced in advance, and a section in which manual operation is temporarily indispensable. The section 63 in the chart (b) is a section in which the provision of the high freshness update LDM140 cannot be maintained due to a decrease in passing vehicles and a shortage of communication bands due to excessive use of surrounding public communications. Manual operation is also essential in this section.

Chart (d) shows an example of RRR and return success rate. The charts (e), (f), and (g) show examples of the presence / absence of a leading vehicle, the availability of a waiting area, and the availability of a control operator, respectively.

Although not shown, for example, when there is no support from the information shown in the charts (e), (f) and (g) in the section 65b, and a takeover event that the driver cannot deal with occurs when entering the section 65b. Can occur. In this case, if the vehicle stops in the section 65b due to the MRM function, the road including the section 65b is blocked, or at the tunnel exit with poor visibility, the own vehicle suddenly stops and there is a risk of a rear-end collision with the following vehicle. There is a high risk of causing a violation situation (details will be described later).

The system communicates with the LDM on the cloud network via the regional infrastructure communication network to request new information, and the cloud network provides the constantly updated high-precision status of the LDM for the planned travel section. Alternatively, the leading vehicle provides highly fresh individual LDM information acquired in each situation such as V2V (Vehicle to Vehicle).

Based on this information provided, the system has a grace period that indicates the grace period during which it is presumed that the vehicle can drive safely, depending on the equipment status confirmed to be used in the latest self-diagnosis status of the vehicle. ΔT _2lim (Time to reach limit of MRM) and the return time required for the driver to return to manual driving, which is passively or actively detected by the system ΔT _drd (Time delay to resume driving =) Notification to driving) and ask.

As shown in the chart (a) of FIG. 15, the return time ΔT _drd is only the distance corresponding to the return time ΔT _drd at the position P61 from the position P61, for example, assuming that the current position of the own vehicle is the position P61. It indicates that it will return to manual operation at the advanced position. On the other hand, as shown in the chart (b), the grace time ΔT _2lim corresponds to the grace time ΔT _2lim from this position P62a to the position P62a when the position P62a is a point where the return to the manual operation is indispensable. It shows the grace period for returning to manual operation at the position traced back from the position P62a by the distance.

The return time ΔT _drd and the grace time ΔT 2 lim change according to the road environment during traveling and the state of the driver, as shown in the positions P62a and _P62b of the chart (b), respectively.

The system compares these grace time ΔT _{2 lim} with the return time ΔT _drd , and determines whether or not the grace time ΔT _{2 lim} and the return time ΔT _drd satisfy the relationship of the following equation (1).
ΔT _2lim >> ΔT _drd … (1)

If the grace time ΔT _{2 lim} and the return time ΔT _drd satisfy the relationship of equation (1), the driver can use the vehicle for automatic driving at automatic driving level 4 while the vehicle is running. , The chances of encountering a situation that requires immediate action are low. Therefore, the risk is limited, and even if the driver is not in time to return to manual driving, it will fall back unless the MRM sharply increases other traffic risks.

Here, the system determines the risk of causing traffic obstruction on the relevant road based on the information obtained from LDM or the like when the own vehicle makes an emergency stop or the like on the MRM in the traveling section. According to this judgment result, if there is a possibility, the system searches for a detour that can be evacuated, determines whether pairing with a leading guide vehicle, and a remote driving support controller before entering the section. -Execution operator-Determine the margin of required communication lines. The system provides risk selection information to users of detour or avoidance selection up to the limit of provision of automatic driving according to the presence or absence of provision of workarounds associated with MRM execution according to the result of this determination. Whether the operation is to prompt the decision or whether the system gives priority to the save selection in advance is a settable selection switchable item, and the process is completed according to the decision.

In other words, whether or not the vehicle traveling in the section can use the automatic driving by the automatic driving level 4 is up to the detour selection, or the continuation limit point where the paired leading guide vehicle or the remote remote operator can remotely steer. , The system becomes one of the influences on the following vehicle, that is, the limit point of control that can be driven without causing a great social influence. By presenting these as information prompting the driver to make a risk coping action decision by the information display unit 120, the information is taken into the driver's working memory, so that the driver approaches a point requiring coping. Sometimes, situational awareness can be performed at an early stage.

The driver does not perform the return action requested by the system at these marginal points, and if the contract is violated, the driver is given a more familiar intuitive penalty to replace the secondary trigger accident in response to the violation. Imposing on the target. In other words, it imposes penalties such as speed limit during continuous driving, forced pit stop at parking, stink, etc., which act directly as a demerit, not a stochastic possibility that the driver is not aware of. As a result, the system can suppress the violation use of the automatic driving, or encourage the driver to change the behavior in which the violation is not positively performed when the automatic driving is used.

Here, as explained using the chart (b), the grace time ΔT _2lim is information that changes with the running of the own vehicle, and it may not be possible to obtain a sufficiently long prediction as originally planned. It can happen. The data of the high freshness update LDM140 provided from the infrastructure may change over time even if the data of the entire itinerary section is received at the start of the itinerary, and acquiring the high freshness update LDM140 each time is the communication band. There is a great risk of causing pressure.

Therefore, the information acquired in advance by the system of the own vehicle is the confirmation information of the section where the automatic driving by the automatic driving level 4 cannot be used, and the prediction information of the grace time ΔT _{2 lim} in each section scheduled as a service. Here, the grace time ΔT _2lim is acquired as more accurate and immediately updated information before actually approaching each section. The acquisition of such information may be acquired by directly requesting it from the regional management server, acquired by V2V from the leading guide vehicle, or acquired from the broadcasted information.

The RRR and return success rate of the chart (d) will be explained. The section where the value of RRR (Request Recovery Ratio) is 100% is a section where there is an extremely high possibility that a sudden deceleration will be required for the following vehicle if the vehicle is stopped or a rapid deceleration is performed in that section. In this section, in order to ensure safety, we request the completion of the transfer in advance.

Examples of sections with high RRR are some special limited sections, such as one-way bridges, where traffic may stop at least halfway or completely prevent two-way traffic, and capitals without vehicle shelters. Examples include special roads such as expressways, sections where general vehicles take time to grasp the situation, such as tunnel exits, roundabouts, and intersections. On the other hand, the RRR can be set to 0% in a section where the traffic volume is extremely small and even if the vehicle is stopped on the road, the possibility of obstructing the view or traveling of the following vehicle is extremely small.

In the example of the figure, in the section 65a, the RRR is set to a value lower than 100%, whereas in the section 65b, the RRR is set to 100%. This indicates that the section 65b is a section in which the stopping or sudden deceleration of the vehicle is extremely likely to have a great influence on the running of the following vehicle. On the other hand, the section 65a in which the RRR is set to a value lower than 100% indicates that the influence of the vehicle stop or the sudden stop on the following vehicle is smaller than that in the section 65b.

The presence or absence of a leading vehicle in the chart (e) indicates the presence or absence of mutual assistance volunteer support for a dedicated standby vehicle or a general vehicle that guides and supports autonomous driving in sections where it is difficult to pass by own vehicle equipment and LDM. ing. When the waiting area on the chart (f) is vacant, for example, when the chart (e) indicates that there is a leading vehicle, the leading vehicle or the own vehicle arrives when the vehicle is assisted in a difficult-to-pass section. Indicates whether there is a waiting place to wait until. The vacancy status of the controller in the chart (g) indicates the vacancy of the controller (whether or not it can be handled) and whether or not the actual pilot operator can be supplied. Affects the return request rate to.

These complex controls may not always have merit when used by general healthy people. On the other hand, when using it as a public service such as for groups with limited manual driving ability (elderly people, children, etc.), which is an advantage of automatic driving, it is difficult to secure the driver required for mobiles due to labor shortage. Above all, it is useful for providing a service network over a wide area of society.

If pairing that can follow the leading vehicle and pairing of remote driving support steering at cruising speed can be guaranteed, it will be possible to drive with automatic driving level 4. On the other hand, when the driver deals alone, the transfer to manual operation is completed or MRM is performed in advance before approaching the next section where RRR is 100% (indicated as section 66 in the figure). You will be asked if you want to stop.

<3-3-2. About ODD at automatic operation level 4>
Next, the ODD at the automatic operation level 4 according to the embodiment will be described. FIG. 16 is a schematic diagram for explaining ODD at the automatic operation level 4 according to the embodiment.

In FIG. 16, the direction in which the own vehicle travels is shown from the left to the right in the figure. The upper figure shows an example of a section provided as static information on the road 70, which can be driven by automatic driving at automatic driving level 4 (Level 4). The lower part of FIG. 16 shows a case where the lane width is limited due to, for example, road construction, the automatic driving of the automatic driving level 4 is difficult, and the section 71 having a limited lane width occurs in the travelable section. An example of is shown schematically. In FIG. 16, the section 71 is between the point R and the point S, and in this section 71, the driver needs to drive by manual operation. From the point S at the end of the section 71 to the section 72 having a predetermined length, the manual operation can be shifted to the automatic operation.

Here, the ideal use of automatic driving level 4 will be explained. If the equipment of the vehicle has a certain level of performance or higher, automatic driving of automatic driving level 4 can always be used in the physical road section from the conditions confirmed before the start of the itinerary. On the other hand, since there is a possibility that the automatic driving of automatic driving level 4 may not be allowed while driving after deciding the itinerary and starting driving, continuous status monitoring should be performed in case of an abnormal situation. From the user's point of view, the existence and significance of autonomous driving will diminish.

Therefore, it is conceivable to introduce a control that minimizes the use of MRM by limiting the use to conditions where the negative social impact is below a certain level while allowing emergency measures called MRM.

It is also conceivable to avoid activating MRM. In this case, it acts accurately and intuitively, that is, with an appropriate priority for the working memory, to provide information for the driver to take voluntary precautions and grasp necessary information before reaching the MRM. Do so. Here, the information provided to the driver is, for example, vehicle dynamics characteristic displacement, self-diagnosis and status presentation of vehicle-mounted equipment, information presentation regarding predictability of the preceding road (including temporary deterioration of sensing performance, etc.), evacuation.・ It is conceivable to provide information on the availability of evacuation (temporary acceptable amount due to accommodation fluctuations) in advance.

Furthermore, even if the driver is involved in NDRA, there is a need for a mechanism to foster a sense of usage priority so that predictable priority coping and its usage pattern can respond to voluntary coping behavior.

Next, a more realistic use of autonomous driving level 4 will be explained. The following conditions are required for the vehicle to autonomously set the ODD as the automatic driving level 4 on the spot in the section where the vehicle can run at the automatic driving level 4 and to drive automatically without the intervention of the driver. Must be met.

First of all, it is necessary for the system to be able to obtain in advance the high freshness update LDM140 of the destination of the assumed itinerary. Then, the acquired high freshness update LDM140 contains information indicating the return success rate (RRR) for each road section of the traveling course, and the system is manually operated by the transfer limit point assumed by the driver. The estimated delay time that can be restored is calculated as the delay time from the notification that achieves this RRR to the restoration.

Furthermore, we will present an avoidance option when the driver does not return before the time considering this delay time. Then, in actual driving, when information that is expected to return to the manual driving of the driver is taken in by some update information from the traveling course destination, the driver actually returns without negligence according to the information. You need to take action.

Here, whether or not the driver takes the expected return behavior in response to the notification from the system is an area determined by the behavioral psychology of the person that the system cannot directly detect.

However, people do not always take voluntary coping actions in all directions at the same time. In other words, it is difficult for a person to expect this voluntary coping unless the ethics achieved based on the social behavior code of conduct are fostered.

Here, first of all, on the premise that this voluntary coping has been developed, it is considered that the person takes action coping from the notification, and the benefit of performing the person's secondary task, the benefit of the main purpose of movement, and the return to driving The disadvantages of not returning when the request is received, the disadvantages of neglecting to acquire the prior information necessary for returning, etc. are projected in the future as a result of the selection action, and the result can be intuitively described. Make a selection decision on coping behavior.

Further, as a result, when actually coping, information important for determining the preselected coping behavior is preferentially stored in the working memory, that is, the working memory that declines in time. ..

And, as a person's behavioral psychology, the delay time from the notification of the necessity of returning to manual driving to the completion of the actual return is the driver's method of providing these advance information and the importance of the notified content. Accurate cognitive status by, elapsed time when the importance in memory from notification of the need for new takeover to notification of implementation declines, presence or absence of concentration matters outside the driver, and the driver's It is determined largely depending on the individual ability difference of memory retention in the working memory of important matters.

Here, the lower figure of FIG. 16 will be described. For example, when new takeover information is generated by the high freshness update LDM140 (step S70), the takeover information is acquired by the system via the regional LDM cloud network as shown in step S71. In addition, the system may receive an abnormality notification signal from a leading vehicle or the like (step S72). Based on the acquired takeover information or abnormality notification signal, the system notifies the driver of the information on the points (points) involved in the takeover and the importance of dealing with the takeover in step S73 (provisional contract). Presentation).

The driver decides the importance according to this notification (step S74), and agrees and responds to the provisional contract. The system detects the driver's response (step S75). As a result, a provisional contract is concluded. Further, in the driver, the information regarding the takeover is stored in the working memory by the agreement response to this provisional contract (step S76).

For example, if the driver does not perform the takeover operation within a predetermined time from the first notification, the system notifies the driver of reconfirmation and determines whether or not the driver is aware of the notification. (Step S77). Depending on the presence or absence of the recognition, the correspondence branches, for example, as shown at point P.

Here, the timing of the driver's return to manual operation differs depending on the driver's recognition of the importance of taking over. For example, if the driver does not detect the driver's response to the provisional contract in step S75, the driver is requested to return to manual driving at the point (position) _Q1 .

The system gives a reconfirmation notice (step S77), and depending on whether or not the driver recognizes this notice, the point Q ₂ ahead of the point Q ₁ and closer to the section 71, or a section further ahead. Issue a return request at point _Q3 near 71.

When a person causes a thinking activity (brain activity) that requires conscious judgment, the knowledge information that is the basis of the thought is temporarily taken into the working memory by the brain unconsciously in order of importance. The information captured in this working memory gradually diminishes from the working memory as its importance diminishes.

During that time, for example, when the driver is absorbed in NDRA, a request to take over from automatic driving to manual driving is issued from the system. The driver is concerned about the need for action if the action is not urgent or does not involve a sense of risk in the near future that acts on the intuition that is detrimental to not performing the takeover overlooking the notification. The feeling, that is, the stored contents of the working memory, fades. In addition, peripheral monitoring information and preconditions (vehicle dynamics characteristics) that the vehicle is running, which are necessary for the transfer, are diminished from this working memory. For example, when the peripheral monitoring information is judged to be important, the acquired information necessary for the judgment is held in the working memory.

For example, while the vehicle is traveling along a route that is allowed to be driven by automatic driving of automatic driving level 4, the road is based on the high freshness update LDM140 and the preceding information of the route destination obtained by V2V communication from the leading vehicle. It is assumed that an event fluctuation that approaches a section where manual operation is essential occurs at the end of the section over time. Furthermore, if there is a narrow road section in front of it that is difficult to evacuate, for example, considering the maintenance of social order, it is required that the transfer be successful even before that. And the delay from notification to successful takeover largely depends on how awake each driver is and remembers the prior information needed to make a decision.

The driver is aware of the need, captures advance information with a degree of tension, recognizes the information in the initial notification and responds (that is, the system detects cognition in the form of response), and its importance is the driver's. If it remains in the working memory as important for memory, the time from notification to recovery can be shortened, and notification just before the limit point is sufficient.

On the other hand, if the driver does not recognize the notification correctly and the system cannot detect the recognition of the driver's notification, the driver considers the need for takeover not sufficiently in the working memory and is faster. A return request is issued at the timing (point _Q1 at the bottom of Fig. 16).

However, unlike the mechanical mechanism of the system, working memory conceptually captures the function of the brain that controls human thinking and judgment, and if the upper limit that can be memorized by each person is different, the health condition and age. For example, some people immediately forget their priorities even if they are important information.

The driver receives such information due to the change sufficiently before the relevant point R (for example, at the corresponding point _Q1 ), and it takes several tens of minutes to reach the corresponding point R after receiving this notification. Imagine a long time. From an ergonomic point of view, when controlled by HCD, one basically stores information in working memory based on the importance of the information. At this time, if the information is not obtained as a direct sense of the importance of the near future, the priority of the information such as NDRA, which is highly important at that time, will be lowered.

In that case, the system shows the driver the urgency of the takeover request and the penalty for failing to fulfill the takeover request, and detects the response by the driver. By responding with an understanding that is not a reflexive response, the driver's working memory enables information injection in a reliable form. As a means to see the response after understanding, the observation state evaluation of the intentional gesture of the driver's awakening cognitive state by pointing and calling shown in Japanese Patent Application Laid-Open No. 2019-021229 and International Publication No. 19/017215 is applied. It is possible. Since pointing and calling has cognitive feedback in its gestures, it can play an extremely large number of roles of confirmatory cognition. Not limited to this, a simpler cognitive response means such as the driver answering a question presented by the system may be used.

Here, the driver who received the return notification at an early stage is expected to lose the importance of the necessity of return due to insufficient response to the advance notification. From the delay history learning from the past notification to the return by the driver, the required time is calculated based on the return success rate to a certain manual operation at the required transfer completion limit point. Then, while the time from the notification of the return request to the completion of the return can be long in this case, the quality of the return action that quickly returns from the notification is managed and indexed. Therefore, in the case of low-quality return behavior, points are deducted and a penalty is incurred. Therefore, the driver's psychology is expected to be an early return behavior.

The driver's advance notice and accurate action judgment for the notification are determined by whether the prior information presented by the information display unit 120 or the like is correctly and appropriately performed according to the risk and is incorporated into the judgment memory. It will be possible for the first time.

Here, use cases # 1 to # 5 in the lower part of FIG. 16 will be described.

・ Use case # 1
In use case # 1, the permitted route of the automatic operation level 4 updated by the quasi-static LDM is automatically controlled by the automatic operation level 4 without active monitoring control by constantly receiving the latest data such as the high freshness update LDM140. This is an example when a driving plan is made assuming that the route is a viable route for driving. In this case, depending on the state of the driver, the driver will be taken over for a new situation that requires the intervention of the driver, which has not been acquired from the quasi-static LDM and the update information has not been obtained. The return to manual operation has not been completed within the permissible limit period. This is an emergency response by MRM, and in some cases, it may obstruct the passage of the following vehicle or induce a rear-end collision risk.

In this use case # 1, for example, when the information is updated by the contract-based reception right such as subscription, the contract expires, the reception is inadequate due to the usage condition limitation in the importance charge, and the remote support concierge service is canceled. , Etc. may occur in various situations.

・ Use case # 2
In use case # 2, the driving environment information of the road ahead is appropriately received in advance from the high freshness update LDM140 or the like and notified to the driver, and the information is accurately recognized by the driver and the response is detected. This is an example when it is not done. In this case, it is uncertain whether the importance of the necessary intervention and the timing of occurrence are stored in the driver's working memory, and there is a risk that the transfer will not be completed safely in time, so it is early for the driver. Will be notified to (Point _Q1 in Fig. 16). Therefore, the time that the driver can be involved in work other than driving (NDRA, etc.) is squeezed. Among them, if the work of taking over from the return notification is not performed quickly and the return quality is low, the penalty evaluation will be deducted, which is a disadvantage of future use.

・ Use case # 3
Unlike the above-mentioned usage case # 2, the usage case # 3 is an example in which the driver correctly recognizes the recognition at the notification stage. Here, in the use case # 3, the automatic operation by the automatic operation level 4 is continued for a long time until the notification of these new events is received at an early stage and the actual corresponding point is reached (point Q ₂ ). In this case, it is uncertain whether or not the importance of taking over and the timing of its occurrence are stored in the driver's working memory at the time of receiving the notification. In a situation where a certain amount of time has passed since the first notification by step S73, the memory may be fading. In this case, the system issues a reconfirmation notice (step S77) to the driver and sees the driver's response, so that the remaining state of the driver's memory is less than that of use case # 4, which will be described later. It turns out that it is, and an early notification is given.

At this time, since the driver has once made a cognitive response to the situation change in step S74, there is no residual memory, and the time to reach the situation awareness (Situation Awareness) is shorter than that of the above-mentioned use case # 2. Therefore, the notification is made at an intermediate time between the above-mentioned use case # 2 and the later-described use case 43.

・ Use case # 4
Use case # 4 is an example in which the driver receives a notification, understands the importance, receives a response in the reconfirmation notification (step S77) to the driver, and the memory is held in the working memory. .. In this case, even if there is time to reach the relevant point, the driver can check the situation due to the approach to the relevant point (for example, for the front or the notification screen) as appropriate before reaching the relevant point. By performing pointing and calling), the driver's working memory memory is refreshed, and risk awareness increases as the driver approaches. As a result, the system can accurately detect the driver's condition and behavior for reconfirmation, even immediately before the notified transfer point (point _Q3 ), based on the undeclined residual information in the working memory. Accurate takeover return is possible, and as a result, high-quality return action can be realized, and the evaluation is an excellent point.

・ Use case # 5
The use case # 5 is the same as the above-mentioned use case # 4 until the recognition when receiving the next transfer necessary information during the automatic operation use for the working memory. On the other hand, in use case # 5, so-called mind wandering causes the driver's consciousness to move away from other thoughts due to the retention of new information in the working memory and the transition of time, and the driver's consciousness breaks out of the loop. is seperated. In this case, the timing of reconfirmation for returning at the required timing differs greatly from person to person.

In use case # 5, the system utilizes observable evaluation indicators such as the awake state and health status of individual drivers, including autonomic imbalance, and provides feedback to the driver continuously with merits and penalties. It is carried out in a form that works intuitively. The system repeatedly presents to the driver appropriate leading risks, information on options for avoiding the risks, and near-future drawable information on the degree of risk impact if the risks are not avoided. As a result, the driver is psychologically strengthened and learned about the early return to manual driving and the habit of performing follow-up observation necessary for the return, and the psychology of grasping the situation before reaching the transfer point is more reliably worked. Can be formed in memory.

In case # 5, the driver unconsciously wears it by self-learning from the presentation of information to the driver, the occasional health condition of the individual driver, and the repeated use behavior of the system. Based on the evaluation of behavioral characteristics, this is a conceptual example of variable notification of the range in which the system allows participation in NDRA as ODD according to the superiority or inferiority of the driver, and it changes greatly depending on the behavioral change of the driver. It is a schematic representation of what is to be obtained.

As described above, based on the travelable section at automatic driving level 4 as the same physical environment, accessibility of information, risk information included in the information, weighting and avoidance selection for importance, timing information In addition, the usage case will differ depending on the response (response) status to the information.

In autonomous driving based on HCD, risk information that serves as a standard for human behavior judgment is added to the timely update information provided by the system, and a penalty is imposed depending on the response at the time of use, or a merit is obtained by a good response. That, the driver will experience it repeatedly over the long term. If the driver enjoys the merits of autonomous driving and becomes over-dependent, the risk that can be intuitively described for the use of the state can be realized by introducing the HCD according to the embodiment of the present disclosure. This allows drivers to rest assured that they will be involved in the return from aggressive autonomous driving to manual driving in order to take advantage of the appropriate and comfortable NDRA, and the system will constantly provide risk information. While confirming information during automatic driving, the benefits of using automatic driving will be utilized. The HMI that promotes the driver's independent action, which is created by the balance between the driver's merits of using NDRA and the penalties, is the control by the HCD, instead of such a compulsory confirmation request to the driver.

<3-3-3. Operation example of automatic operation level 4 according to the embodiment>
Next, an operation example of the automatic operation level 4 according to the embodiment will be described. 17A and 17B are flowcharts for explaining an operation example of the automatic operation level 4 according to the embodiment. Note that in FIGS. 17A and 17B, the reference numeral "G" indicates that the process shifts to the corresponding reference numerals in FIGS. 17A and 17B.

In FIG. 17A, in step S200, the automatic driving control unit 10112 acquires and holds various information such as the LDM initial data 80, the driver personal return characteristic dictionary 81, the RRR 82, and the vehicle dynamics characteristic 83. Further, the automatic operation control unit 10112 also acquires updated LDM information (# 1, # 2, ...), Acquires updated diagnostic information, and acquires other updated information (N).

In the next step S201, the automatic operation control unit 10112 identifies the initial ODD and approves the setting for the automatic operation based on the information acquired in the step S200. In the next step S202, the automatic driving control unit 10112 presents the itinerary to the driver and requests the driver to select a route or the like. In addition, the automatic driving control unit 10112 requests the driver to select a route for whether or not NDRA is possible. In the next step S203, the driver starts driving the own vehicle.

In the next step S204, the automatic driving control unit 10112 acquires each information described in step S200, and constantly updates the information accompanying the running after the start of the itinerary. Further, the automatic driving control unit 10112 visually displays the driving inhibition information at each automatic driving level including the arrival time (see FIGS. 8, 9A to 9C).

In the next step S205, the automatic driving control unit 10112 determines whether or not the vehicle has entered the ODD section where the automatic driving by the automatic driving level 4 is approved. When the automatic driving control unit 10112 determines that the own vehicle has not entered the ODD section (step S205, "No"), the process returns to step S204. On the other hand, when the automatic driving control unit 10112 determines in step S205 that the own vehicle has entered the ODD section (step S205, "Yes"), the process shifts to step S206.

In the flowcharts shown in FIGS. 17A and 17B, the same process is repeated from step S204 (not shown) when once exiting the section where automatic driving is possible and entering the section where it is determined that a new ODD is available.

In step S206, the automatic driving control unit 10112 determines whether or not the driver has requested to switch the automatic driving mode. When the automatic operation control unit 10112 determines that there is no such switching request (step S206, "No"), the process returns to step S204. On the other hand, when the automatic operation control unit 10112 determines that the switching request is met (step S206, "Yes"), the automatic operation control unit 10112 shifts the process to step S207.

In step S207, the automatic driving control unit 10112 determines the probability of the driver's ability to return to manual driving. Here, the automatic driving control unit 10112 has an advantage due to automatic driving (NDRA, etc.) when the driver is a good automatic driving user driver who positively performs a returning operation based on, for example, the driver's individual return characteristic dictionary 81. ) Is allowed to be used. On the other hand, the automatic driving control unit 10112 prohibits or limits the use of the automatic driving function when the driver is prone to drug dependence or sleep disorder even if the deduction or penalty is small. With the determination process in step S207, many drivers are self-learning to avoid violations in order to obtain the benefits of NDRA during autonomous driving.

The usage permitted for the driving route is not always the environment in which the use of automatic driving according to the automatic driving level 4 can be provided even if the use of automatic driving is permitted. As described above, if the condition that the driver has the ability to return is guaranteed, there is also a condition that the use of automatic driving and advanced support is permitted up to the automatic driving level 3. In that case, it can be said that the determination process in step S207 is the determination of the use of automatic driving at the driver's automatic driving level 3 (Level 3). The operation and the like of the automatic operation level 3 according to the embodiment will be described later.

When the automatic driving control unit 10112 determines that the driver's ability to return to manual driving is promising (step S207, "OK"), the process shifts to the flowchart of FIG. 17B according to the reference numeral "G" in the figure. Let me. On the other hand, when the automatic operation control unit 10112 determines that the return capability is unlikely (step S207, "NG"), the automatic operation control unit 10112 shifts the process to step S208.

In addition, in the usage form of automatic driving, when using in combination with remote driving support, leading guidance vehicle support, etc., the judgment to confirm the driver's ability to return to manual driving is not essential, and another judgment processing is performed. Therefore, it is not included in the process shown in the example of the present embodiment.

In step S208, the automatic driving control unit 10112 presents the driver with a notice of disapproval of the use of automatic driving with a reason. Reasons presented to the driver in this case may be, for example, driver fatigue, drowsiness, a driver's penalty history in which the cumulative addition value of overdependence violations in the past history is equal to or higher than a predetermined value. Drivers who want to get the benefits of autonomous driving can learn to improve their behavior by presenting a disapproval notice from the system with a reason, such as improvement learning and a limited number of excellent boost permission requests (described later). There is expected.

After the processing in step S208, the processing is returned to step S204. Here, while driving, there are cases where the vehicle is driven on a road section suitable for the conditions for using the automatic driving function, or the initial conditions are replaced with the conditions where the use is permitted by improving the driver's awareness of coping. .. Therefore, the system loops the processes of steps S204 to S208 and continuously monitors the state of the driver and the like.

The explanation moves to the flowchart of FIG. 17B, and according to the reference numeral "G", that is, when the driver selects to drive in the automatic driving of the automatic driving level 4, in step S220, the automatic driving control unit 10112 is the latest after the start of the itinerary. Update the information. After entering the ODD section where the automatic driving of the automatic driving level 4 is permitted, as long as the conditions do not change, the driving can be continued as it is by using the automatic driving of the automatic driving level 4. In the flowchart of FIG. 17B, step S220 shows a state in which monitoring is performed in the steady state, that is, a loop process for updating the latest information accompanying traveling.

When some change in the situation along the route is detected in step S220 or the end point of the ODD is approached, the automatic operation control unit 10112 shifts the process to step S221.

In step S221, the automatic driving control unit 10112 determines whether or not information related to the latest route, which is indispensable for continuous automatic driving driving, is updated. When the automatic operation control unit 10112 determines that the information has not been updated (step S221, “No”), the automatic operation control unit 10112 shifts the process to step S226.

In step S226, the automatic operation control unit 10112 starts executing a safe takeover sequence as scheduled due to the approaching end of the automatic operation section (NDRA use section). Then, the takeover sequence is executed.

Here, the automatic driving control unit 10112 evaluates a good driver, for example, a driver who is faithful to the return request by the system, does not constantly neglect the return request, and confirms the status change during the itinerary. Add points to the value. Further, the automatic driving control unit 10112 permits the excellent driver to select the next automatic driving mode without going through complicated confirmation approval procedures such as multiple authentication. The excellent driver can also preferentially receive the usage guidance of the automatic driving of the automatic driving level 4. In this way, a good driver can receive various merits.

In order to realize HCD for the driver to take the best confirmation action, the driver has the "memory" necessary for making these confirmation judgments and judgments that cause accurate behaviors from the system. It is the process of capturing and is the "quality" of the information that the system provides to the HMI. For example, it works on the information such as when, what, what should be done and what kind of effect is caused, that is, the working memory, which is exemplified in FIG. 9C.

The driver's return behavior in the safe takeover sequence in step S226 is evaluated for its quality, acquired as return behavior data, and stored. This return behavior data affects the evaluation points for the driver.

On the other hand, when the automatic operation control unit 10112 determines in step S221 that the information is updated (step S221, "Yes"), the process shifts to step S222a, step S222b, and step S223.

In step S221, when it is determined that the information is updated (step S221, "Yes"), the allowable ODD condition at the time of entering the section of ODD is a sudden deterioration of the weather during traveling, a malfunction of the vehicle, or the like. , It is assumed that measures will be taken when an unexpected event such as a load collapse occurs. When the occurrence of such an unexpected event is found, it is necessary to take measures according to the grace period required to deal with the event. The flowchart of FIG. 17B shows an example of a series of processes related to this measure applicable to the embodiment. The quality of the driver's event coping behavior at the time of an abnormality is also learned by the driver through the driver's appropriate risk judgment behavior, and is an important factor for producing an appropriate behavior change for the driver.

In step S222a, the automatic driving control unit 10112 monitors the driver's condition including the response judgment to the notification recognition, obtains an estimated value of the delay time required for returning to the manual driving based on the monitoring result, and obtains an estimated value of the existing estimated value. To update. Further, in step S222b, the automatic driving control unit 10112 updates the RRR information for the road, and from this update, calculates the review of the usage allowance limit of the MRM.

In step S223, the automatic operation control unit 10112 displays the review ODD calculation based on the information determined to be updated in step S221, the updated and acquired information in steps S222a and S222b, the self-diagnosis information, and the like. Further, the automatic driving control unit 10112 confirms the driver's response to this display.

In the next step S224, the automatic driving control unit 10112 calculates the predicted arrival time T _L4ODDEND until the end of the ODD section related to the automatic driving level 4 and the predicted time T _MDR of the return delay to the manual driving. Then, in the next step S225, the automatic operation control unit 10112 determines whether or not the calculated predicted arrival time T _L4ODDEND and the predicted time T _MDR satisfy the relationship of [ _TL4ODDEND > T _MDR + α]. The value α is the margin time to the point required to start taking over.

When the automatic operation control unit 10112 determines in step S225 that the relationship of [ _TL4ODDEND > _TMDR + α] is not satisfied (step S225, “No”), the process shifts to step S226.

On the other hand, when the automatic operation control unit 10112 determines in step S225 that the relationship of [ _TL4ODDEND > _TMDR + α] is satisfied (step S225, “Yes”), the process shifts to the next step S227.

In step S227, the automatic driving control unit 10112 determines whether or not the driver has largely left the driving based on the monitoring result of the driver's condition. When the automatic operation control unit 10112 determines that the driver has not largely departed from the operation (step S227, “No”), the process returns to step S220. In this case, the driver has not deviated significantly from the driving and can expect a response to the notification from the system.

On the other hand, when the automatic operation control unit 10112 determines in step S227 that the driver has largely left the operation (step S227, "Yes"), the process shifts to step S228.

In step S228, the automatic operation control unit 10112 determines whether or not there is a section beyond which a high return success rate is required. When the automatic operation control unit 10112 determines that the section does not exist (step S228, “No”), the process returns to step S220. In this case, it means that even if the own vehicle is suddenly stopped by, for example, MRM, the situation in which the influence on the surrounding vehicles and the like is extremely small continues for the period of the margin α in the future.

On the other hand, when the automatic operation control unit 10112 determines that the section exists (step S228, "Yes"), the automatic operation control unit 10112 shifts the process to step S229.

In step S229, the automatic operation control unit 10112 determines whether or not there is still a workaround such as a runaway slope or a waiting pool area on the way to the takeover start point. When the automatic operation control unit 10112 determines that there is a workaround (step S229, “Yes”), the process returns to step S220.

On the other hand, when the automatic operation control unit 10112 determines that there is no workaround (step S229, "No"), the process shifts to step S230. In this case, if the own vehicle travels as it is, there is a high possibility that the evacuation means will be cut off and the MRM will be activated, and there is a risk of inducing traffic obstruction to the following vehicle or a rear-end collision.

In step S230, the automatic driving control unit 10112 notifies the driver of the approach of the essential point for returning from the automatic driving to the manual driving, and detects the response by the driver to approve this notification. In the next step S231, the automatic operation control unit 10112 determines whether or not there is a remaining time until the takeover. When the automatic operation control unit 10112 determines that there is no remaining time (step S231, “No”), the automatic operation control unit 10112 shifts the processing to the sequence of MRM execution.

On the other hand, when the automatic operation control unit 10112 determines that the remaining time is present (step S231, "Yes"), the process shifts to step S232 and attempts to return to manual operation within the allowable time by the driver. do. In this case, a success rate higher than RRR cannot be expected. In the next step S233, the automatic operation control unit 10112 determines whether or not the takeover attempted in step S232 succeeded before the limit. When the automatic driving control unit 10112 determines that the transfer is successful (step S233, "Yes"), it is assumed that the use of one section of the entered automatic driving use is completed. On the other hand, when the automatic operation control unit 10112 determines that the takeover has failed (step S233, "No"), the automatic operation control unit 10112 shifts the processing to the MRM execution sequence.

The driver's return behavior at the time of the transition of the MRM execution sequence from step S231 or step S233 and the transition from step S233 to the end of use of one section is acquired and saved as return behavior data. .. This return behavior data affects the evaluation points for the driver.

In the above description, each determination process in FIG. 17B, for example, the determination process of steps S225 to S229 is shown to be performed as a sequential process in chronological order, but this is not limited to this example. For example, the processes of steps S225 to S229 are executed in parallel in chronological order to determine whether or not the driver can return, and at least one of the determinations of steps S225 to S229 cannot be expected to return. In the case of the determination result, the process may be directly transferred to the MRM execution sequence. You may fly.

<3-4. Application example of HCD for automatic operation level 3>
Next, an example of applying the HCD to the automatic operation level 3 according to the embodiment will be described.

Automatic driving level 3 is defined as a mode in which the driver can always respond to abnormal situations. Therefore, in order for the vehicle to drive safely without disturbing the social order at the automatic driving level 3, the driver can always quickly return to the manual driving while using the vehicle at the automatic driving level 3. It is necessary to pay attention to the condition of the driving road environment in advance and to prepare the posture and posture to return to manual driving. In other words, if the driver cannot expect these situations, it is no longer appropriate to use the vehicle at autonomous driving level 3. That is, in this case, considering the state of the driver, it is hard to say that the vehicle can be driven at the automatic driving level 3.

That is, at least the ODD of the automatic operation level 3 handled in the definition of the automatic operation level 3 according to the embodiment is an operation design area that can be used after these conditions are satisfied. In this case, the ODD of automatic driving level 3 corresponds to the automatic driving level 3 up to the range where the driver can be expected to deal with the situation after continuing to drive under the current state of the driver. It becomes the limit of the area.

In other words, the available ODDs of autonomous driving level 3 will leave the driver's attention and the continuous surrounding environment necessary for driving when the driver is withdrawn from long-term driving steering work that does not intervene in the driving. This will lead to a decline in the ability to collect information, and it will be difficult to take action to deal with an urgent transfer from automatic driving to manual driving.

The driver continuously collects the information necessary to perceive, recognize and judge the situation when he / she is responsible for driving. The reason is that action judgment requires predictability in the near future associated with the selected action action, and in order to ensure the predictability in a more reliable state, the driver should use continuous manual driving. Continue to collect a lot of information. The information collected here includes not only the behavior of the vehicle in front, but also the cargo loaded by the vehicle, the road conditions further ahead of the vehicle in front, the presence or absence of traffic congestion, and the manual warning of the section by road signs. It contains a lot of information that cannot be obtained instantaneously after notification of a request to return to driving.

The information that is originally necessary for these judgments and is unknowingly acquired in the case of manual driving is gradually diminished or not updated in the working memory from the stage when the surrounding monitoring attention is interrupted by using the automatic driving function of automatic driving level 3.

Therefore, in the embodiment, the automatic driving level 3 is determined according to how long the driver's unique thinking characteristics and states are continuously grasping the surrounding environment and the own vehicle status close to the caution state of manual driving. The ODD is set by limiting the driving in. The ODD design of the vehicle is determined by taking into consideration the driver's current state and future prediction estimation in addition to the vehicle's environmental recognition performance, acquisition of prior information on the driving route, and the self-diagnosis result of the own vehicle.

From the viewpoint of HCD, it is difficult for the driver to continuously grasp the peripheral information so that the driver can immediately take over the driving steering without intervening in the actual driving steering (manual driving). In autonomous driving, it is thought that thinking will be assigned to other than driving during many hours of driving, and from the working memory, the surrounding environment assumption that is essential for safe steering, which is necessary for handing over to short-term manual driving, is assumed. There is an increased risk that the memory of the information, the characteristics (changes, etc.) of the own vehicle, and even the awareness that the driver is driving and steering will fade away.

Therefore, the ODD that defines this automatic driving level 3 based on the HCD according to the embodiment is different from the existing ODD defined as a design based on the performance limit of the system, the maintenance status of the road, the prior information of the road, and the like. Become. That is, a section that can be expected to be dealt with even if the driver receives a request to take over from automatic driving to manual driving is determined from the driver's awakening state and the history of continuous grasp of the surrounding situation. Within this section, the range defined as a design from the system performance limit, road maintenance status, road advance information, etc., is the automatic driving level 3 that is allowed by the driver's current situation grasping and coping ability. It is a travelable area.

Here, the extension of the section of automatic driving level 3 will be explained. Depending on the driver's own condition, the driver is allowed to use the automatic driving level 3 for a short period of time, which can be expected to return in that state, and after grasping the situation, an extension application is made to the system, and the automatic driving level 3 is interrupted. It is assumed that it will be used in a similar manner.

FIG. 18A is a diagram schematically showing how a driver traveling on a road 70 with his / her own vehicle extends a section of automatic driving level 3. In this example, the driver repeatedly executes short-term automatic driving by automatic driving level 3 by an extension application in a section where automatic driving of automatic driving level 3 (Level 3) can be used conditionally. It is shown. That is, the driver performs automatic driving according to the short-term automatic driving level 3, applies for an extension at the end of the period, and further performs automatic driving according to the short-term automatic driving level 3. In the example in the figure, the driver repeats this action.

Here, since the system allows the extension by a simple extension request by the driver, for example, by operating a button, the driver needs to be careful about the actual continuous safety confirmation and take over. Prerequisite information necessary for grasping the situation ahead is not taken into the working memory. Therefore, there is a possibility that the system may allow the extension while the prediction information is faded in the driver's working memory.

For example, the system detects the behavior of confirmation such as pointing and calling by the driver to the front of the road together with the extension request by button operation or the like. As a result, a "re-contract" agreement is reached between the system and the driver regarding the situation grasp and the responsibility as a result, and the driver is re-incorporated into the working memory with a sense of responsibility, that is, a memory of the need for return. It will be possible to make it.

When permitting the use of autonomous driving at level 3 of autonomous driving, the issues are whether or not the driver is in an appropriate return system, posture, and awake state during the use, and the driver is continuously. It depends on whether or not he has fulfilled his duty of due care. From an ergonomic point of view, there are no penalties for neglecting those original obligations, and if the driver is not careful just because he / she may be at risk, these obligations are not necessarily obeyed. do not have.

It is already socially known that the use of autonomous driving with these violations will occur, and it should not be overlooked if this is left unattended. In other words, when these state transitions are recorded and a feedback loop is formed as a matter not to be overlooked as a legal penalty, the driver is separated from the interval of fulfilling the duty of care to avoid the risk of the accident itself. You will be sensuously affected by the "virtual pain" that is recorded and saved in the driver's consciousness as a violation state that is subject to more intuitive penalties.

The reason why there are few speed violations on roads where speed violations are monitored and cracked down is that the psychology of behavioral judgment that projects the driver's crackdown as a recent risk works, and the same penalties have been introduced. However, violations have the same effect in terms of behavioral psychology as they increase preventive psychology because there is "more familiar and realistic" monitoring such as crackdown.

It is also possible to record the violation state in the driver's behavior as a silent record, that is, to record in the shadow mode without making the driver aware. However, even in this case as well, it is desirable to intentionally perform sensational recording such as visual, auditory, tactile, and olfactory from an ergonomic point of view. In other words, by repeatedly receiving the perceived violation state, the information in the driver's working memory when approaching the violation state becomes a state in which the priority of the action judgment for avoiding the risk is higher. This is because the understanding of the state of the brain develops. Depending on whether this risk is visible to the driver as a sensation or not, it has a different effect on the driver's behavioral psychology.

The perceptual feedback that influences these judgments acts as an HMI that indicates the risks of the near future based on HCD, and its psychological effect is on the driver's intuitive future, which cannot be obtained only by introducing stricter penalties such as legislation. It is reflected as an influence on the movement behavior. When there are multiple means of HMI feedback, information on risks in the brain is a variety of stimuli, such as risk information acting directly on the visual cortex, even when it is difficult to convey information in words. Through this, it is possible to evolve into a mechanism that avoids danger by lowering the probability of overlooking danger. All drivers have the mechanism of the brain and body to avoid risks in the near future without overlooking them, although there are individual differences.

FIG. 18B is an example flowchart showing the processing in the conditional automatic operation level 3 available section shown in FIG. 18A according to the embodiment. When the own vehicle enters the section where the automatic driving is used according to the automatic driving level 3 (Level 3) in step S300, the automatic driving control unit 10112 learns the return characteristics of the driver of the own vehicle based on the past information. With reference to the characteristic dictionary 81, in step S301, the driver is constantly monitored, and the driver's arousal level and attention level are indexed.

The driver's individual return characteristic dictionary 81 can include, for example, a driver's body model and a head model described later, and each part of the body, the head, when the driver returns from automatic driving to manual driving. It can contain information indicating an action including the time required for the eyes, etc.

The indexing in step S301 is, for example, an automatic driving level that is permissible from the driver's arousal state, fatigue accumulation state, and psychological burden level detected based on the driver's past recoverable attention duration. The maximum time for using the automatic driving according to 3 is predicted, and this maximum time is used as an index of the driver's arousal level and attention level.

In the next step S302, the automatic driving control unit 10112 determines whether or not the driver has applied for the start of automatic driving according to the automatic driving level 3. When the automatic operation control unit 10112 determines that the application has not been made (step S302, "No"), the process returns to step S301. On the other hand, when the automatic operation control unit 10112 determines that the start of use has been applied for by the driver (step S302, "Yes"), the automatic operation control unit 10112 shifts the process to step S303.

In step S303, the automatic operation control unit 10112 turns on the timer and starts measuring the time. In the next step S304, the automatic driving control unit 10112 monitors the driver, detects the wakefulness state and the state of reduced attention due to the continuous use of the automatic driving by the driver's automatic driving level 3, and also detects the driver's state. Monitor the standby status for returning to manual operation.

In the next step S305, the automatic driving control unit 10112 determines whether the measurement start time in step S303 exceeds a predetermined time and the use of the automatic driving level 3 has timed out, or the driver's wakefulness has decreased. To judge. When the automatic driving control unit 10112 determines that the use of the automatic driving level 3 has not timed out and the driver's wakefulness has not deteriorated (step S305, "No"), the process is set to step S304. return.

On the other hand, when the automatic driving control unit 10112 determines that the use of the automatic driving level 3 has timed out or the driver's wakefulness has deteriorated (step S305, "Yes"), the process shifts to step S306. ..

The processing from step S306 to step S310, which will be described later, is the processing in the conditional automatic operation level 3 available section shown in FIG. 18A.

In step S306, the automatic driving control unit 10112 monitors the driver and detects the notification of the driver's request to return to manual driving and the behavior of the driver. Here, the automatic driving control unit 10112 displays a warning to the driver and voluntarily early to manual driving when a decrease in arousal is detected even within the period in which the driver is presumed to be able to maintain awakening. Present a display prompting you to return.

In the next step S307, the automatic driving control unit 10112 determines whether or not a normal return to manual driving by the driver has been detected.

When the automatic operation control unit 10112 determines that a normal return is detected (step S307, "Yes"), the process shifts to step S308. In step S308, the automatic driving control unit 10112 gives the driver excellent function utilization points, assuming that the driver has used the function of the automatic driving level 3 excellently. Further, the automatic driving control unit 10112 updates the characteristic learning dictionary (for example, the driver individual return characteristic dictionary 81) of the pre-detection evaluation index.

When the automatic operation control unit 10112 determines that a normal return is not detected in step S307 (step S307, "No"), the process shifts to step S309. In step S309, the automatic driving control unit 10112 records a penalty for the driver for violating the function of the automatic driving level 3. Further, the automatic driving control unit 10112 updates the characteristic learning dictionary (for example, the driver individual return characteristic dictionary 81) of the pre-detection evaluation index.

Further, in the transition from the process of step S307 to the process of step S309, the automatic driving control unit 10112 causes the driver's awakening state, fatigue accumulation state, psychological burden degree, etc. predicted from the driver's past recoverable attention duration. The maximum time allowed based on the automatic driving level 3 automatic driving is displayed and presented to the driver.

After the processing of step S308 or step S309 is completed, the use of automatic operation by the automatic operation level 3 (Level 3) in the individual times is terminated (step S310).

On the other hand, when the automatic driving control unit 10112 detects an extension application from the driver in step S307 to extend the usage time of the automatic driving of the automatic driving level 3 (step S307, "extension application"), the usage time. Is extended by a predetermined time, and the process is returned to step S306. The extension application is made, for example, by the driver operating a predetermined operator provided in the input unit 10101. At this time, the automatic driving control unit 10112 detects that the driver has made a forward confirmation such as pointing and calling for the operation, and then approves the application for extension of the section to be used individually.

<3-5. About the determinants of ODD>
Next, the determinants of ODD applicable to the embodiment will be described with reference to the flowcharts of FIGS. 19A and 19B. Here, as an example, when actually using the automatic driving level 3 on a specific road where the automatic driving by the automatic driving level 3 is expected to be used as the environmental infrastructure, the ODD of the usage permission is based on the usage conditions. The usage determination of the above and the determinants of the control will be described. FIG. 19A is an example flowchart showing the flow of automatic operation processing applicable to the embodiment, paying attention to ODD.

When the driver sets the itinerary including the destination of the driving and the vehicle starts driving, the automatic driving control unit 10112 uses the automatic driving according to the automatic driving level 3 on the target road in step S400. The possible condition is determined by the determination process described later, and the availability section (ODD) of the automatic operation according to the automatic operation level 3 is set. That is, the automatic driving control unit 10112 includes road environment data 85 indicating the road environment included in the itinerary such as LDM as the vehicle travels, own vehicle information 86 regarding the vehicle driven by the driver, and further, the driver. Get the status of whether or not to return. Of these, at least the status of whether or not the driver can return is always acquired while the driver is using the automatic driving according to the automatic driving level 3. In step S400, the ODD is set from the current state based on the self-diagnosis result for perception, recognition, judgment, and control of the performance of the on-board equipment of the vehicle.

In the next step S401, based on the latest self-diagnosis result of the vehicle, whether or not the automatic driving control unit 10112 can maintain the driving within the ODD section even if the vehicle continues on the course, that is, ODD. As a result, it is determined whether or not the conditions under which the continuous use of the automatic operation of the automatic operation level 3 is permitted are continuously maintained. When the automatic operation control unit 10112 determines that the traveling can be maintained within the ODD section (step S401, "Yes"), the process returns to step S400.

Here, as long as the conditions for continuous use of automatic operation of automatic operation level 3 or possible conditions are met, the automatic operation permitted by automatic operation level 3 can be continued to be used as long as the state continues. On the other hand, the usage actually permitted to the automatic driving level 3 is limited to a specific time range, and when a driver's consciousness decline or the like is observed in the determination of step S401, the system automatically drives the driver. The use permission judgment for level 3 automatic driving will be cancelled.

On the other hand, if the automatic driving control unit 10112 determines in step S401 that the vehicle cannot maintain running in the ODD section when the vehicle continues on the course (step S401, "No"), the process shifts to step S402. Let me.

As a more specific example, the automatic driving control unit 10112 will proceed outside the ODD section that can handle automatic driving if it continues on the course, or due to changes in the weather or a decrease in the driver's awakening state. When the use of autonomous driving of automatic driving level 3 is not allowed, that is, there is a situation change that deviates from the ODD defined as the automatic driving of automatic driving level 3 is available, and the vehicle continues on the course, the ODD It is determined that the traveling cannot be maintained in the section (step S401, "No"), and the process is shifted to step S402.

In step S402, the automatic driving control unit 10112 predicts the remaining time until the vehicle reaches the ODD end point based on the latest updated information such as LDM. The automatic operation control unit 10112 can notify the driver of the remaining time until the arrival time at the predicted transfer completion point.

In the next step S403, the automatic driving control unit 10112 explicitly indicates to the driver that the vehicle concerned is requested to return to the outside of the ODD, that is, from the automatic driving state to the manual driving state. At the same time, the automatic driving control unit 10112 causes sudden deceleration or sudden excessive steering of the driver in the surrounding vehicles toward the outside of the vehicle when the transfer to manual driving is not performed normally and smoothly. Disseminate information to notify the possibility of risk factors.

Further, the automatic operation control unit 10112 records the state. This record serves to help prevent over-reliance on the driver's autonomous driving system. Repeated violations of autonomous driving by drivers and inappropriate use that deviates from ODD will be regulated depending on the local approval system. Even if the user (driver) of autonomous driving does not have a direct accident due to excessive dependence on autonomous driving, the penalties retroactively based on the recorded information act directly on the driver as a psychological risk. , It becomes a factor of direct behavior change for improvement in the driver.

Whether or not the driver actually performs the action of returning to manual driving is not included in the flowchart of FIG. 19A because it is not the sequence that the system itself bears. If the driver dislikes falling into a violation state and wants to avoid being fined, it is considered that he / she will voluntarily complete the use of autonomous driving in the middle of the process according to the flowchart. Therefore, when the takeover request notification is actually given to the driver in step S405 described later, the driver performs a process of terminating the use of the automatic driving, and a series of processes of the flowchart of FIG. 19A is completed (illustrated). do not do).

The process is transferred to step S404. In step S404, the automatic operation control unit 10112 determines whether or not the margin (for example, time) required for taking over the manual operation is equal to or less than the limit by the end of the ODD section. When the automatic operation control unit 10112 determines that there is a margin until the takeover and it is not necessary to deal with the takeover at the present time (step S404, "No"), the process returns to step S400.

On the other hand, when the automatic operation control unit 10112 determines that the margin is equal to or less than the limit (step S404, "Yes"), the automatic operation control unit 10112 shifts the process to step S405. In this case, the vehicle has already entered the section where the handling of the transfer to manual driving should be started.

In step S405, the automatic driving control unit 10112 presents a notification requesting the driver to take over to manual driving, and confirms the driver's response response to the notification. In addition, the automatic driving control unit 10112 presents a warning regarding use outside the ODD where automatic driving is permitted inside and outside the vehicle, and records the progress state.

In the next step S406, the automatic driving control unit 10112 determines whether or not the completion of the appropriate return action to the manual driving by the driver has been confirmed. When the automatic operation control unit 10112 determines that the completion of the return action is confirmed (step S406, “Yes”), the process is returned to step S400.

On the other hand, when the automatic operation control unit 10112 determines that the completion of the return action is not confirmed (step S406, "No"), the process shifts to step S407. In this case, the driver's return behavior violates the allowable ODD. In step S407, the automatic operation control unit 10112 executes the MRM and records the progress and the result of the execution of the MRM. Then, in step S408, the control of the vehicle is transferred to the MRM as an emergency response.

If you are a driver who is normally using automatic driving at automatic driving level 3, you will usually be able to continuously pay attention to the front so that you can quickly return to the system in response to a request to return the system, and you will be aware of the situation necessary for prompt recovery ( Since Situation Awareness) is maintained, it is expected that human behavior will promptly return to manual driving so as not to receive violations or penalties.

Here, if the support system for autonomous driving becomes more sophisticated, the factors that affect the sense of risk in behavioral psychology other than direct penalties will diminish, and the sense of risk will no longer be felt. As a result, not only the automatic driving of automatic driving level 3 but also some drivers who are alert, neglect to return to manual driving despite the violation state, and the automatic driving function does not guarantee the handling. There is a risk that usage cases will progress. Even in such a use case, it is desirable that the stepwise MRM execution is performed during a period in which safety can be guaranteed, instead of sudden braking by the MRM. The flow from step S404 to MRM in step 408 described above corresponds to this fallback process when the return of the driver to manual operation cannot be confirmed by the system.

FIG. 19B is an example flowchart showing in more detail an example of the ODD setting process applicable to the embodiment according to step S400 in the flowchart of FIG. 19A described above.

The automatic driving control unit 10112 acquires the road environment static data 90 such as LDM, and if possible, the high freshness update LDM 140, and in step S420, for example, the conditions are set for each section included in the road environment static data 90. If it is arranged, it is determined whether or not the section can be used for automatic driving. When the automatic operation control unit 10112 determines that each section does not include a section in which automatic operation can be used even if the conditions are met (step S420, "No"), the process shifts to step S429.

In step S429, the automatic driving control unit 10112 indicates to the driver in the next step S430 that the automatic driving cannot be used, assuming that there is no section in which the automatic driving can be used. Not limited to this, the automatic driving control unit 10112 may display a symbol of the limited support function, a required predicted time until a section where automatic driving can be used, and the like.

After the process of step S430, a series of processes according to the flowchart of FIG. 19B is completed, and the process is transferred to step S401 of FIG. 19A.

When the automatic operation control unit 10112 determines in step S420 that there is a section in which automatic operation becomes available (step S420, "Yes"), the process shifts to step S421. In step S421, for example, from each section included in the road environment static data 90, a section in which automatic driving becomes available when the conditions are met is extracted as an ODD.

The process proceeds to step S422, and at that time, the automatic driving control unit 10112 acquires the mounted device information 91, which is the information of the device mounted on the vehicle. The mounted device information 91 includes information indicating a response limit by self-diagnosis that can be detected by the mounted device of the vehicle.

In step S422, the automatic driving control unit 10112 determines, based on the mounted device information 91, whether or not there is a section in the section extracted in step S421 that the mounted device of the vehicle cannot handle. When the automatic driving control unit 10112 determines that all the sections extracted in step S421 are sections that cannot be dealt with by the on-board equipment of the vehicle (step S422, "all sections"), the process shifts to step S429.

When the automatic driving control unit 10112 determines in step S422 that there is no section that can be dealt with by the on-board equipment of the vehicle in the section extracted in step S421 or that it exists in a part of the section (step S422, "none or part". Section "), the process is shifted to step S423. In step S423, the automatic operation control unit 10112 limits the ODD to the section that the on-board device can handle with respect to the section extracted in step S421.

The process proceeds to step S424, and at that time, the automatic driving control unit 10112 acquires the road correspondence availability information 92 indicating whether or not the corresponding road can be supported for automatic driving based on the weather information and the like.

In step S424, the automatic driving control unit 10112 determines, based on the road compatibility information 92, whether or not there is a section in the section restricted in step S423 where the handling of the on-board equipment is restricted due to weather or the like. When the automatic operation control unit 10112 determines that the entire section of the section restricted in step S423 is a section in which the handling of the on-board equipment is restricted due to the weather or the like (step S424, "all sections"), the processing is performed in step S429. To migrate to.

When the automatic operation control unit 10112 determines in step S424 that there is no section in the section restricted in step S423 for which the handling of the on-board equipment is restricted due to weather or the like, or that it exists in a part of the section (step S424, "None". or a part of the section "), the process is shifted to step S425. In step S425, the automatic driving control unit 10112 imposes on the ODD the use of the ODD for events in which the handling of the on-board equipment is restricted due to the weather, such as reduced visibility, road surface freezing, and backlight.

The process proceeds to step S426, and at that time, the automatic driving control unit 10112 acquires the driver response availability information 93 indicating whether or not the driver can return from the automatic operation to the manual operation. The driver response availability information 93 is information based on the driver's condition such as the degree of fatigue.

In step S426, the automatic driving control unit 10112 determines whether or not the driver can deal with the return to manual driving as necessary, based on the driver support availability information 93. When the automatic operation control unit 10112 determines that the countermeasure cannot be taken (step S426, “No”), the process shifts to step S429.

When the automatic operation control unit 10112 determines in step S426 that the return to manual operation can be dealt with (step S426, "Yes"), the process shifts to step S427. In step S427, when the use of automatic driving is prohibited due to the state of the driver, the automatic driving control unit 10112 excludes it from the usage permission ODD of automatic driving. Examples of driver states where the use of autonomous driving is prohibited include wakefulness, poor health, and drinking alcohol.

At this time, the psychology of the driver who wants to continue driving with the support of automatic driving works because the wakefulness is lowered. In this case, if a driver whose arousal is lowered due to drinking or drug use guarantees a certain level of automatic driving of the system, the driver will be repeatedly used for the support function, and as a result, the risk psychology will be lowered.

This decline in risk psychology induces excessive dependence on the automatic driving function even though the automatic driving function is not universal, and the driver cannot return to manual driving in the event of an unforeseen situation. Therefore, in the end, it automatically shifts to the above-mentioned MRM, which may adversely affect the social transportation network, which is not desirable. In other words, since the use of ergonomics that may disturb the social order is progressing, such an operation is necessary as a preventive measure.

Further, the process of forcing the transition to step S429 by the determination of step S426 (step S426, "No") requires the driver to develop a psychology that the automatic driving function cannot be used depending on the response. Further, the determination notification may be explicitly presented.

On the other hand, it is easy to activate MRM when a driver with a physical disability uses an autonomous driving vehicle to assist his / her movement, or when he / she is using an autonomous driving vehicle in combination with remote assistance. Instead, there are some operational modes such as follow-up driving support by the preceding guided vehicle and steering support by remote monitoring, in which the driver's manual return is not the highest priority, so controls other than those shown in FIGS. 19A and 19B have been introduced. May be good.

In the next step S428, the automatic driving control unit 10112 displays a map of the ODD that permits the use of automatic driving, which is comprehensively judged based on each of the above-mentioned judgment results.

After the process of step S428, a series of processes according to the flowchart of FIG. 19B is completed, and the process is transferred to step S401 of FIG. 19A.

FIG. 20 is a schematic diagram for more specifically explaining an example of setting an ODD section applicable to the embodiment.

In FIG. 20, the chart (a) shows an example of an allowable section for automatic driving in an infrastructure, and assumes a predetermined section of a specific highway as an example. The chart (b) shows an example of the average vehicle speed due to the traffic flow of the vehicle group in the section shown in the chart (a). In this example, the average vehicle speed in the section from point U ₀ to point U ₁ is 90 [km / h]. For example, due to the influence of traffic congestion, the average vehicle speed from point U ₁ is 60 [km / h]. The average vehicle speed is 60 [km / h] or more at point U ₂ due to the elimination of traffic congestion.

In addition, chart (c) shows an example of a section within the performance limit of the equipment mounted on the target vehicle, which can be dealt with in a steady state during the daytime by judgment based on information acquired in advance. Further, the chart (d) shows an example of a case where the ODD is out of the applicable section.

As an example, consider a case where the conditions for using the automatic driving function are allowed only when there is a traffic jam of 60 [km / h] or less as stipulated by law. In this case, even if the section on the map where the use of the automatic driving function is permitted is a certain section of the expressway as shown in the chart (a), for example, the section where the use of the automatic driving is actually permitted. Is limited to the section (points U ₁ to U ₂ in the example of FIG. 20) and the timing in which the average vehicle speed of the vehicle group drops to 60 [km / h] or less due to, for example, the occurrence of traffic congestion within the fixed section. ..

Furthermore, there may be sections where safe driving cannot be expected during automatic driving due to conditions such as the equipment mounted on the vehicle and the weight of the vehicle. For example, there are sections where the use of automatic driving is restricted by loading heavy load differences in sections with steep curves, and automatic driving functions that the system can handle in normal weather, but due to rough weather. There are sections where misidentification of road boundaries may occur due to poor visibility or flooding of roads.

In the example of FIG. 20, as shown in the chart (d), the section from the points U ₀ to U ₁ is not applicable to the ODD because the average vehicle speed due to the traffic flow is 90 [km / h] (ODD). Outside). Since the average vehicle speed is 60 [km / h] in the section between points U ₁ and U ₂ , ODD is applied (within ODD). Here, at point _U11 , visibility is poor due to weather such as local heavy rain and snowfall, and even though the section is within the handling limit of the onboard equipment, it is a section outside the ODD to which the ODD is not applied. Therefore, the driver needs to return from the automatic operation to the manual operation during the period indicated by the diagonal line from the point U ₁₁ . For example, when the poor visibility started from the point U ₁₁ at the point U ₁₂ is resolved, the ODD is applied and the manual operation can be shifted to the automatic operation.

Next, the section to which the _ODD is applied is the section within points U ₁ to U ₂ , and starts from U ₁₂ which is the starting point of the handling limit of the onboard equipment. It is necessary to return from automatic operation to manual operation during the indicated period. In the example of FIG. 20, at the point U ₁₃ that has traveled for a predetermined time from the point U ₁₂ , it is determined that the driver cannot drive due to fatigue of the driver, etc., and in the subsequent sections, the on-board equipment is within the handling limit. No, ODD is not applied. After this point _U13 , MRM may be activated.

In this way, even in sections where the use of autonomous driving is permitted by the infrastructure, the ODD conditions change due to various causes such as the driving of the vehicle group, the weather, and the driver's condition, and the ODD application range changes accordingly. Changes can occur. Therefore, as described with reference to the flowcharts of FIGS. 19A and 19B, the actual ODD is determined by sequentially setting the conditions as the vehicle travels.

Note that the term "ODD" described in this specification is not necessarily used as a section defined as a design in the industry at the present time. That is, "ODD" in the present specification is used based on the idea of "a usable section in which automatic driving at each automatic driving level is allowed as a system in each social situation according to various conditions". It is possible to obtain the same effect even if this definition is extended and applied according to the application of automatic driving of vehicles implemented in order to aim for safe and comfortable use of society.

In FIGS. 19A, 19B and 20 described above, the description is limited to the operation of the ODD at the automatic operation level 3, but the application of the ODD is limited to the automatic operation level 3 and further to the automatic operation level 4. do not have. For example, even in the automatic driving level 2, if the support level becomes more sophisticated, it is considered that the same problems as those described in the above-mentioned automatic driving level 3 will occur. Moreover, even if the vehicle is the same, it is conceivable that different automatic driving modes will be switched depending on the situation of social infrastructure development.

Therefore, it is important for the driver to maintain a certain degree of involvement in the system control in system control, regardless of which automatic driving level the driving mode of the vehicle has transitioned to. It is to use automatic driving with behavioral judgment psychology. Therefore, the control sequence for each automatic operation mode described in the present embodiment is a use case, and is not limited to these.

<3-6. About DMS according to the embodiment>
Next, the DMS (Driver Monitoring System) according to the embodiment will be described.

<3-6-1. Outline of DMS according to the embodiment>
When the vehicle's automation mode is autonomous driving level 3 or autonomous driving level 4, the system needs to constantly monitor the driver's abilities.

If the system is unable to continue a planned task, such as following a route or continuing on a planned itinerary, and suddenly switches driving from autonomous driving to manual driving, the driver will take over to manual driving. Unprepared and panicked, situational awareness may be difficult and the system may run MRM for safety. MRM serves as a fallback function in the event of a failed migration. However, improper use of MRM in the social environment can often result in unnecessary rear collisions, traffic jams and other unwanted and unexpected consequences. Therefore, constant monitoring of the driver as described above is required.

The system is driver-specific in order to maximize the probability that the return will be successful and to minimize the probability that the driver will not be able to return to manual operation when the system requests the driver to return to manual operation. You must be able to make appropriate judgments about your ability.

On the other hand, if it is difficult for the system to assist the driver in giving the right notifications and warnings at the right time and returning to manual driving without hindrance, the driver's sense of duty to follow the system is reduced. there's a possibility that.

Notifying the driver of the request for return to manual operation early means that there is time to return to the timing and it is not necessary to immediately start the return procedure. Also, the slow notification to the driver may increase the risk and fail to recover, and the cause of the migration failure is the delay in notification by the system. It will give the driver the opportunity to take responsibility for the system.

A person's behavioral psychology is selfish and selfish. Therefore, if the system cannot issue notifications and warnings at appropriate times, the driver does not have much sense of risk for notifications from the system when using autonomous driving. Therefore, when taking over the actual manual operation, the driver does not recognize the importance of starting the takeover until the takeover limit is reached, and the situation awareness (Situation Awareness) including the confirmation of the surrounding situation required before the start of the manual operation is lowered. It creates the risk of reaching the limit of handing over.

That is, if the driver prioritizes the return to the requested manual operation and the driver is often unable to complete the return to the manual operation in time before the system initiates the MRM. It is up to the driver's choice and intention to accept that the system may perform MRM.

However, since such fallback processing is likely to induce new problems in social activities, it does not solve the problems from the viewpoint of social activities. For example, on a road under construction, if the vehicle is in one lane such as a road point during accident recovery, if the vehicle is controlled by MRM, the traffic may be completely blocked. Social benefits are prioritized over individual tastes and / or satisfaction, as no one has the right to interfere with social activity or put others into crisis.

As a solution to avoid such a negative side of automation of the automatic driving system, there is a method of promoting behavioral psychology to increase the degree of involvement of the driver depending on the situation.

However, the challenge is how to increase the driver's involvement and correctly recognize the state transitions required for the system to start taking over to manual operation. When the system requests the driver to return to manual operation, it is necessary for the driver to appropriately and voluntarily return to manual operation without delay. To do this, the system provides the driver with control equivalent to the benefits of proper return behavior by the driver, and if the driver is late in the start of the return or does not take the return action promptly, or if there is a violation. If this is the case, it is effective to impose a penalty on the driver and encourage the driver to change his / her behavior.

However, it is difficult to observe the psychological state of the driver in the brain while driving the vehicle with a practical device. For example, a head-mounted optical topography observation device such as a head gear for observing cerebral blood flow, a cerebral blood flow table device such as fNIR (functional Near-Infrared Spectroscopy), and a large-scale such as fMRI (functional Magnetic Resonance Imaging). The use of medical equipment makes it possible to visualize the driver's direct local thinking activity in the brain in a laboratory or the like. However, there is no prior art that allows the driver to observe the state in the brain without being restrained in the vehicle. Therefore, a direct measurement of the driver's thinking activity is not realistic.

On the other hand, by observing the behavior that reacts to the thinking activity and analyzing and evaluating the behavior, it is possible to estimate the activity status in the brain to some extent from the observed physical behavior of the driver. Then, by properly analyzing the measurement results, many hints that can be regarded as the reaction of thinking activity can be obtained, and the probability that the driver is appropriately following the instructions within a predetermined time frame can be obtained. Can be done. By quantifying the driver's activity in this way, it is possible to estimate the response when responding to various advance notifications, return requests, warnings, confirmation requests, and other instructions from the driver's system. It is possible to realize a method of giving appropriate rewards and penalties to the driver, and to encourage the driver's voluntary involvement.

In other words, the driver can enjoy the comfort of autonomous driving and not be content with overuse for autonomous driving by gaining the merit of executing appropriate recovery measures in response to the request from the system. It is expected that this will eliminate the harmful effects of the widespread introduction of autonomous driving functions in society.

In the embodiment of the present disclosure, DMS as a technique for quantifying the "Quality of Action" together with a plurality of assumed examples in which a driver-involved parameter group, which is a parameter group involved in the driver, can be obtained. We propose (Driver Monitoring System). In the following, "quality of action" may be referred to as "QoA".

<3-6-2. More specific description of DMS according to the embodiment>
Next, the DMS according to the embodiment will be described more specifically.

The following two targets are calculated and monitored by the DMS according to the embodiment.

(1) Estimated time required for the driver to return to an appropriate driving posture The DMS according to the embodiment calculates the estimated required time allocation required for the driver to return to an appropriate and safe driving posture from the current posture. Monitoring targets for this include, but are not limited to, deviations resulting from head, hand, eye, and postural movements in the driver's conscious or unconscious non-driving movements. Further, the DMS according to the embodiment estimates a real-time deviation by the driver from the driving posture that the driver deems appropriate.

(2) Preparation level at which the driver can completely take over the driving task from automatic driving to manual driving The DMS according to the embodiment is the boundary of the road division of automatic driving level 3 or can be automatically driven by automatic driving level 4. A certain level of awareness required to return to manual driving before the vehicle reaches the permitted ODD boundaries, whether at the boundaries of the road division (either dynamically applicable). Monitor your mental and physical condition.

If the system is capable of continuing autonomous driving without driver intervention, the driver can greatly enjoy the benefits of autonomous driving on certain road sections. On the other hand, such a road section is limited to a certain section of the road where traffic information is dynamically monitored and updated information can be provided in advance to vehicles approaching the road section. In these sections, the driver is not required to be fully responsive to immediate intervention requests requesting the driver to return to manual operation as a fallback. In the road section in these special situations, the driver of a particular vehicle with the ability can enjoy autonomous driving at autonomous driving level 4.

However, the road sections as described above, which are determined by ODD for each autonomous driving level, are not fixed and continue to change over time. There are an extremely large number of factors that can change the conditions for determining the ODD for automatic driving according to the automatic driving level 3 of the vehicle and the ODD for automatic driving according to the automatic driving level 4.

For example, when the driver departed from the starting point, a specific road section was an ODD capable of automatic driving at automatic driving level 4, but with the passage of time, a large amount of snow accumulated on the road section. It can occur and the vehicle system may not be able to handle such situations. In such a case, the system may change the automatic driving level allowed for the road section from the automatic driving level 4 to the automatic driving level 3 or a lower automatic driving level.

In such cases, the system will respond to the driver's request to return from autonomous driving to manual control before the vehicle reaches the boundary of the road section that allows the modified and updated autonomous driving level 4 autonomous driving. It is necessary to be able to predict the driver's readiness and the time it will take for the driver to return to the driving position so that the response is in time.

It is also expected that the sensitivity or threshold of the driver attention index required to warn the driver will also need to be adjusted according to environmental conditions (including risk factors or personalized controls).

By the way, in some sections of the road, if the driver can return to manual driving, automatic driving of automatic driving level 3 can be used, and a specific ODD to which automatic driving of automatic driving level 3 can be applied. It can be defined as a section. Further, in some sections of the road where the system can predict the scheduled driving operation task to continue the scheduled driving without the intervention of the driver, the automatic driving of the automatic driving level 4 is possible. The system can extend this section to a predictable range, and a specific ODD to which automatic driving of automatic driving level 4 can be applied is newly set.

The road section that specifies the ODD is dynamically specified regardless of which level of automatic driving is used. Since it is extremely difficult to control the external environment of the vehicle and the entire performance of the system functions of the vehicle system itself, the road section is constantly changing over time.

MRM is considered a safe fallback in current autonomous driving systems, but is used in certain situations (eg construction work, certain weather conditions, unexpected sudden behavior of nearby vehicles, etc.) When it is done, it is not suitable as a fallback and is not really practical. If the use of large-scale MRM cannot be minimized, various events that significantly reduce social activities such as traffic congestion, rear collisions, and obstacles on the road of vehicles will occur, depending on traffic conditions. The social impact will increase. From such a social point of view, it is strongly required in autonomous driving to find measures for drivers to at least not use MRM excessively as much as possible.

Therefore, it is extremely important to constantly monitor the driver. That is, the driver needs to be able to return to manual operation in advance for an appropriate time that can be estimated and determined from the extreme point in time that triggers the use of the MRM.

The time required to return to manual operation differs depending on the initial state of the driver at the time of notification of the return request by the system. Alternatively, if the system senses that the level of attention required of the driver is not sufficient, it must make a decision to stop immediately, regardless of the level of autonomous driving.

From this point of view, the system that controls automatic driving uses the behavior characteristics peculiar to the driver who learned the behavior by collecting the behavior of the driver so that the driver can return to the manual driving in response to the request for returning to the manual driving. Then, a sign indicating a decrease in the driver's ability to return to manual driving is estimated, and the time required for returning to manual driving is estimated based on the detected situation. When balancing this estimated time with the time it takes to launch the MRM, the system decides whether to notify, request intervention, or warn, depending on the situation. Can be decided.

In order to maximize the safety level at autonomous driving level 3 and autonomous driving level 4, the system can continue the safe and calm driving task as appropriate when safe driving is required, without the driver relying on MRM. To be able to do so, you need to make sure that the driver maintains a certain level of readiness / attention. In addition, when the driver is notified of the request to return to manual driving, the driver does not panic in response to the request to return, and takes over the situation awareness (Situation Awareness) for action judgment necessary for near future prediction. The analysis of the behavior secured by the start is also a judgment factor for the system to notify the return request.

The intent behind this hypothesis is that the driver can always fall back, and even if that is not possible, the driver should resume manual driving in a timely manner before reaching the boundaries of the ODD. In an emergency where it is difficult, the MRM function can completely stop the vehicle. Therefore, the system must be able to accurately detect the driver's attention level and readiness level when requesting intervention from the driver.

Thus, the ODD, which regulates autonomous driving at a particular level of autonomous driving, is the driver's estimated ability to respond to intervention requests and the creditworthiness of the return collected based on the driver's past use of autonomous driving. Score history needs to be considered.

In addition to existing ODD factors such as road conditions to the destination, vehicle performance, load dynamics, etc., the system has sufficient minimum information to continue driving without driver intervention for the time being. If it can be predicted that the vehicle is currently traveling on a certain road section, the driver can be made to use the automatic driving by the automatic driving level 4 of the system. Alternatively, if the performance of the automatic operation level deteriorates due to an insect strike on the front view camera, and it is necessary to shift to the automatic operation of the automatic operation level 3 until the automatic cleaning is completely completed, or the automatic cleaning takes a long time. In such a case, it may be difficult for the driver to keep his / her attention and wait, and it is necessary to set again to completely return to the manual operation.

In the question of how the driver should interact with the autonomous driving system in response to the ever-changing ODD, it is important that the autonomous driving system keeps track of the driver's performance appropriately from various perspectives. This is to ensure that the driver is still able to resume manual operation. In addition, the system may cause a delay from the time when the driver is notified of the request for return from automatic driving to manual driving until the driver completes the transition without any trouble with a success rate exceeding the target value without switching to MRM mode. Can be predicted.

To enable this method, the system constantly monitors (a) the driver's mental readiness and (b) the estimated time it takes for the driver to return to the driving position. There is a need.

All of these are unique to the driver. For drivers who are new to autonomous driving, the system will uniformly adapt to the behavior of the driver over a period of time, for example, from the time obtained from the statistically evaluated behavioral evaluation of the new driver to the time-consuming behavior of the driver. Give the driver permission to use. That is, the system first learns the behavior characteristics of a specific driver in an offline manner based on the return observation evaluation data regarding the driver. The system utilizes the dictionary generated from the repetition to obtain the recognized driver's return characteristics by real-time analysis for the driver's behavior observation, and estimates the driver's awakening state each time.

Whether or not the driver needs to be prepared to maintain his or her attention depends on whether forecast information can be obtained from the infrastructure or whether the embedded system can automatically navigate and predict road safety. As mentioned above, the time required for recovery is used by the system for recovery notification, so that warning processing and fallback processing are determined in a timely manner, and some time is required for the system to start MRM. Always secured.

More importantly, the system can be adapted to the online system by constantly monitoring the driver's behavior and parameterizing and storing the data. Pre-notification behavior monitoring, which monitors driver behavior in response to prior notification to the driver, captures trends and characteristics that correlate with different levels of behavioral quality (QoA) required for the process of returning to manual driving. can.

It should be noted that the operation related to the driver's individual behavioral characteristics is performed online using the latest learning dictionary without relying on the offline outside the vehicle, because the behavioral characteristics of the person are fatigue and physical condition at that time. This is because there are fluctuating factors, and if behavior analysis is performed uniformly depending on the dictionary generated by learning offline, offsets such as physical condition fluctuations cannot be corrected.

No methods or techniques available to accurately model and estimate the driver's ability to take over driving and manual control have been identified at this time. However, by modeling and predicting the driver's return process and feasible return levels, the driver's readiness, awareness, and response are essential for the driver to transition from autonomous driving to manual driving without hindrance. Several different techniques can be used to provide the potential with high reliability.

By continuing to use autonomous driving repeatedly, it is possible to obtain an evaluation value for the behavior quality according to the driver's behavior, and as a result, the subsequent hand-over request notification and the execution by the driver for it are found, and the behavior is Since the quality is obtained as a set, self-learning is repeated quickly in the learner.

Predicting the driver's readiness will help achieve a high level of transition between different autonomous driving levels. This allows the driver to stay away from the driving task for extended periods of time, allowing the driver to be further away from the driving position, an existing method for detecting arousal and fatigue. No mere monitoring of the driver's facial features or eye condition is required. By monitoring the driver's movements with posture tracking, the system determines whether the driver can resume manual driving and accurately estimates the driving position and time required to return to a suitable driving position and posture for safe driving. Many parameters can be accumulated.

<3-6-3. Quantification of behavioral quality (QoA) according to the embodiment>
In this disclosure, we propose the following to quantify the "Quality of Activity" as an index.

(Item # 1)
By directly sensing the following by the DMS according to the embodiment, the ready state of the driver is constantly monitored.

Monitor the following for the physical characteristics of the driver:
(1) 3D (three-dimensional) human body (joint position and posture) and head posture (position and orientation)
(2) 3D human body and head movements within the time cycle T (3) Eye condition and facial gesture monitoring (4) Gaze direction for driving task

Monitor the following regarding the behavior of the driver.
(1) Response to warnings and interventions by the system (2) Driver behavior in driving tasks (manual driving mode) and non-driving tasks (automatic driving mode)

Monitor the driver's alertness level and attention, as well as lesions, based on the extracted driver characteristics and driver behavior.

(Item # 2)
The DMS according to the embodiment monitors the driver and drives the driver by considering the input information regarding the driver's current position, the driver's movement (movement), and the conditions outside the vehicle (ODD and weather conditions). Estimate the time required to return to position.

From the driver's point of view, the driver needs some advantage in following the notification requesting that the manual operation be restarted with high accuracy, smoothly and promptly. As with many activities of daily living, it is very important to visualize the relevance of the return process. Therefore, visual and auditory feedback to the driver and / or feedback by tactile information has a learning effect and motivates the driver to improve his / her return performance.

This feedback loop is greatly involved in the development of human natural movements when using the automatic driving function. Ratings such as multiple levels of penalties imposed by delayed message feedback and, conversely, high performance good drivers for responding to highly accurate driving transitions, are punished by drivers for poor return performance or autonomous driving. Educate drivers little by little to raise awareness of the benefits of promptly requesting advance notice or notification, not because they are deprived of usage opportunities.

(Item # 3)
The data collected by the DMS sensing system according to the embodiment is parameterized and converted into motion values corresponding to different types of motion / motion in the vehicle as a QoA index (quantification of return quality performance).

The intermediate analysis of the movement corresponding to the correction of the body movement to return to the driving position is given a high score in consideration of the speed of correction in addition to the evaluation of the initial state.

The following examples can be considered as the above-mentioned QoA index.

(1) A foot return index that can be evaluated by the orientation vector of the foot movement.
For example, if the initial position of the driver's foot is off the pedal while engaged in autonomous driving level 4, the time it takes for the driver to return to the natural position of normal driving is estimated. There is a need. When assessing speed, not only the physical speed, but also how quickly and accurately the driver can return the foot to the proper position. At this time, since the behavioral habit differs depending on the driver, the detected return procedure flow is provided to the learning function and combined with the delay time when the transfer is successful.
(2) Posture return index that evaluates the body returning from the non-driving position of the backrest.
(3) The position of the hand in three dimensions with respect to the steering is tracked, and how far the hand is from the steering is estimated. At this time, it can be estimated whether both hands or one hand is free or closed.

The advantage of converting the raw data acquired as described above into indicators that characterize the return performance according to the return stage of manual operation is that the system accurately tracks the return performance and the driver's behavior is actual. The fact that they are categorized according to performance means that they can give immediate visual feedback to the driver. Note that this actual performance directly affects the permission level at which the system subsequently switches back to the automated driving level.

In the driving engagement behavior, excessive dependence of automatic driving and low quality of return are prohibited or suppressed. Such "visual feedback" works directly on the memory of the human brain and is prioritized in the driver's working memory depending on the difference in critical level determined by the indicators of improper return movement or timely good return movement. Ranking is given.

The novel features of the embodiments of the present disclosure are as follows.

(1) The driver's dynamic action in the return procedure is quantified by a quantification method and a parameterization method, and each time an RTI (operation change request) event occurs, the result of the return event is used to directly correlate with the driver return performance. ..

(2) Also, additional timely feedback on the detected and quantified quality of behavior is provided to the driver by visual methods that directly affect the visual cortex of the brain, thereby improper return processing. If the system detects that, the driver is always urged to give top priority to the return task. This is because it is a visual indication that a penalty will be imposed if the delay cannot be recovered in a timely manner.

(3) The positive side of the secondary effect of repeatedly giving feedback to each event is that the driver who has learned by himself / herself by this procedure intuitively and positively returns. At this time, if the feedback tends to provide incorrect information, self-learning is not performed normally, and it is necessary to provide accurate and appropriate feedback according to each person's performance. This adaptation process is realized by the following personalization process and learning function embedded in the system.

(4) The self-learning method is not a method of learning by forcing the required movement, but the driver attempts a quick and good recovery process using the analysis type given as an example here. It is a method of learning unconsciously by repeating use due to rewards for things and disadvantages and penalties imposed for improper return processing of the driver.

(Item # 4)
Driver identification and personalization-Adapt the driver monitoring system to a specific person (specific driver). This is a necessary process as the monitored features and parameters vary widely from person to person.

The driver's condition varies greatly depending on the driver's ethnicity, age, personal habits, physique, gender, etc. Therefore, for example, eye condition monitoring needs to be changed according to the person. Similarly, depending on the physique and camera position, the surveillance system should be customizable to the characteristics of the driver.

In the customization process, the driver should make adjustments and settings as the driver feels comfortable with personalization, such as setting the height and setting of the driver's seat position and adjusting the orientation of the rearview mirror. There is. On the other hand, in the customization process according to the embodiment, the system observes the driver's behavior through repeated use, calculates the return time distribution by learning based on the data of the observation result, and gives advance notice necessary for achieving the target RRR. It is a process to decide the notification timing. That is, the customization process according to the embodiment is a process performed by the system after observing the usage of the driver, and is not set according to the driver's preference.

At this time, some of the set values may be adjusted to add an offset to ensure safety in commercial vehicles such as large carpool vehicles, and a takeover request notification may be issued earlier than the system schedule notification. It is useful. Furthermore, for drivers and the like with impaired memory ability, auxiliary notification functions such as advance notification and re-notification may be further added to notifications such as a return request.

The system must be able to learn and adjust (fit) new drivers.
(1) The process of learning a part of the pipeline includes the process of acquiring reference data when the driver's information is first fetched into the system.
(2) A general (average) model is determined based on a plurality of drivers and their behavior. At this time, the model may include a plurality of base models.
(3) Based on the detection data from each sensor and the driver's situation, the driver's database is constructed, transformed, and adjusted for a new driver.
(4) Adapt the system to new case scenarios and driver changes by performing adaptations and adjustments in real time. For example, adjust the sensing system, adjust to a new case scenario, and so on.

(Item # 5)
Introduce a defined driver model consisting of case scenarios and n-dimensional data sets that correlate with a particular ODD. Driver models include head and face models with 3D data adapted to extract head and face features more accurately. In addition, high-level driver model definitions are parameterized to a group of driver descriptors for body shape, physique, behavior, and the like.

-Identify the driver by a face recognition method using depth information and brightness information of 3D sensing technology. The depth and luminance information generate a 3D model of the driver's head and face, which can be adapted to the driver as the depth and luminance information is input. As a result, a driver's 3D mesh model can be obtained with a limited number of control points. This makes it possible to detect rigid transformation (head position) and facial gestures and eye conditions due to non-rigid deformation with high accuracy.

Head position and orientation alignment for whole body skeletal tracking plays an important supportive role in driver motion monitoring used to assess driver behavior with respect to expected behavior in known ODDs. Fulfill.

-The driver is in a situation where it is necessary to dynamically change the expected ODD according to the weather, the condition of the in-vehicle device, other situations notified to the driver, and the driver's reaction to the notification. Helps improve the estimation process of the system for estimating the "situational awareness" level of. Here, the driver's "situational awareness" is dynamically that, for example, the manual driving must be restarted in a state different from the state agreed and approved by the driver before starting the automatic driving mode. Includes recognition that the vehicle is in a changed new state. The system may also detect that the driver has received the notification, for example by pointing to the notification menu or destination on the information panel.

<3-6-4. Configuration applicable to DMS according to the embodiment>
Next, a configuration applicable to the DMS according to the embodiment will be described. FIG. 21 is a functional block diagram of an example for explaining the function of the driver behavior evaluation unit 200 applicable to the DMS according to the embodiment. The driver behavior evaluation unit 200 shown in FIG. 21 is included in an automatic driving system of a vehicle such as an automatic driving control unit 10112.

In FIG. 21, the driver behavior evaluation unit 200 includes a learning unit 201a and an evaluation unit 202. For example, the learning unit 201a learns and parameterizes the characteristics and actions of a specific driver, and stores the parameterized driver information in a database. The evaluation unit 202 refers to the database based on the information obtained by monitoring the driver in the vehicle with various sensors, acquires the corresponding parameters, and obtains the quality of the driver's behavior based on the acquired parameters.

First, the learning unit 201a will be described. In FIG. 21, the learning unit 201a includes a driver information generation unit 2000, a parameter generation unit 2001, and a driver database (DB) 2002.

The driver information generation unit 2000 inputs static information and dynamic information about the driver. The static information input to the driver information generation unit 2000 is, for example, an image of the driver's head, face, and body at a fixed reference position facing the direction of the in-vehicle camera with the in-vehicle camera. It is a captured image (called a reference image) obtained by the above. On the other hand, the dynamic information input to the driver information generation unit 2000 is an captured image (referred to as an operation image) obtained by capturing an image of a driver performing a defined series of operations with the in-vehicle camera.

In the embodiment, since the ToF camera and the stereo camera are provided as the in-vehicle camera, these reference images and motion images can be acquired as information having depth information.

The driver information generation unit 2000 extracts the feature amount of the head or face from each of the input reference image and the motion image, and personalizes the driver based on the extracted feature amount, and the individualized driver. Generate information. Further, the driver information generation unit 2000 generates an N-dimensional model of the driver based on these reference images and motion images. The driver information generation unit 2000 further adjusts each generated information to reduce the dimension of the information.

The parameter generation unit 2001 generates parameters based on each information generated by the driver information generation unit 2000, and parameterizes the driver. Parameter generation unit The generated driver parameters are stored in the driver DB 2002.

Next, the evaluation unit 202 will be described. In FIG. 21, the evaluation unit 202 includes a conforming unit 2003, a monitoring / extraction / conversion unit 2004, a preparation state evaluation unit 2005, and a buffer memory 2006.

The matching unit 2003 is input with an image captured by a camera in the vehicle of a driver who performs a series of random movements. The driver who performs this series of random movements is, for example, a driver who is driving the vehicle. This captured image is also passed to the driver information generation unit 2000. The driver information generation unit 2000 further performs driver individualization and N-dimensional model generation using this captured image. The parameter generation unit 2001 parameterizes the N-dimensional model generated using this captured image, and adds the generated parameters to the driver DB 2002.

The matching unit 2003 identifies the driver by performing face recognition with reference to the driver DB 2002 based on the 3D information generated from the input captured image. Further, the matching unit 2003 passes the fitted 3D model to the monitoring / extraction / conversion unit 2004 based on the 3D information.

The monitoring / extraction / conversion unit 2004 refers to the driver DB 2002 based on the 3D model passed from the conforming unit 2003, and extracts parameters from the 3D data. The monitoring / extraction / conversion unit 2004 converts the extracted parameters into the format used by the preparation state evaluation unit 2005 and stores them in the buffer memory 2006.

In FIG. 21, the arrow on the right side of the buffer memory 2006 shows the transition of time inside the buffer memory 2006, and the lower end side in the figure shows the newer (latest stored) parameter. The processing in the evaluation unit 202 is updated, for example, every predetermined time cycle T, and the parameters stored in the buffer memory 2006 are sequentially moved to an earlier time domain (upper side in the figure) for each time cycle T. .. In this example, the buffer memory 2006 has a capacity for four cycles of the time cycle T, and when the capacity is filled by a new parameter input, for example, the first stored parameter is discarded.

Region 2006a schematically shows the parameters of the last, that is, the latest stored time period T. The preparation state evaluation unit 2005 evaluates the preparation state of the driver based on the parameters stored in this area 2006a. This preparatory state is, for example, a preparatory state for the driver's return operation from automatic driving to manual driving when the driving mode of the vehicle is switched from the automatic driving mode to the manual driving mode. As described above, the evaluation value indicating this evaluation is added or subtracted for each driver and serves as an index of reward or penalty for the driver.

The learning unit 201a shown in FIG. 21 is configured in the system, that is, in the automatic operation control unit 10112, but this is not limited to this example. That is, the function of the learning unit 201a may be realized outside the system, that is, offline.

FIG. 22 is an example functional block diagram for explaining the function of the learning unit configured offline, which is applicable to the embodiment. In FIG. 22, the learning unit 201b includes a 3D head model generation unit 2100, a face identification information extraction unit 2101, a body model generation unit 2102, a model expansion unit 2103, a parameter generation unit 2104, and a storage unit 2105. including.

The captured images captured by the in-vehicle camera of the driver looking at the direction of the in-vehicle camera are input to the 3D head model generation unit 2100 and the face identification information extraction unit 2101, respectively. The 3D head model generation unit 210 generates a 3D model of the driver's head based on the input captured image. The generated 3D model is passed to the model extension unit 2103. In addition, the face recognition information extraction unit 2101 extracts face identification information for identifying the driver's face from the input captured image. For example, the face recognition information extraction unit 2101 extracts a feature amount from the input captured image to specify a face, and acquires face identification information based on the feature amount related to the specified face. The acquired face recognition information is passed to the model extension unit 2103.

Fixed driver's body sensing information is input to the body model generation unit 2102. Here, for the body sensing information, for example, an image captured by a camera of a driver having a fixed posture can be applied from a certain direction. The body model generation unit 2102 generates a driver's body model (for example, a body model based on 3D information) based on the input body sensing information. The generated body model is passed to the model extension 2103.

Further, the model expansion unit 2103 is input with an captured image acquired by capturing an image of a driver performing a specified series of operations with the in-vehicle camera and information indicating different environmental conditions. The model expansion unit 2103 expands the driver's head and body model to an N-dimensional model based on each input information. Further, the model expansion unit 2103 personalizes and customizes the expanded N-dimensional model and passes it to the parameter generation unit 2104.

The parameter generation unit 2104 performs parameterization of the N-dimensional model passed from the model expansion unit 2103, and stores the generated parameters in the storage unit 2105. The parameters stored in the storage unit 2105 can be used, for example, as the initial data of the driver DB 2002 shown in FIG.

The generation of the 3D head model applicable to the

learning units

201a and 201b described above will be schematically described. FIG. 23A is a schematic diagram for schematically explaining the generation of the 3D head model applicable to the embodiment.

As shown in the chart (a) of FIG. 23A, the 3D head model generation process 220 uses the information of a plurality of different types of drivers, and is a database of the head models of a specific driver. With reference to the DB 2200 and the general head model 2201, a meshed head model 2202 for the driver is generated. The meshed head model 2202 generated by the 3D head model generation process 220 has 3D + 1D information. The 3D information is three-dimensional information in a three-dimensional space, and the 1D information is time-dimensional information. That is, the meshed head model 2202 is a head model based on 3D information with time information added.

The chart (b) of FIG. 23A is a schematic diagram showing the meshed head model 2202 more specifically. Model 2210 shows an example in which a 3D mesh is applied to a face image, and model 2211 shows an example in which a face image is removed from the model 2210 to make only a 3D mesh. In the 3D head

model generation process

220, 3D control points 2212a, 2212b, 2212c, ... Generate a 3D deformable head model that can be adapted to facial expressions and features.

The generation of the body model applicable to the

learning units

201a and 201b described above will be schematically described. FIG. 23B is a schematic diagram for schematically explaining the generation of a body model applicable to the embodiment.

As shown in the chart (a) of FIG. 23B, in the 3D body model generation process 230, the driver body model DB 2300, which is a database of a specific driver's body model, and the driver body model DB 2300, which is a database of a specific driver's body model, are used by using information of a plurality of different types of drivers. The driver's body model 2302 is generated with reference to the general body model 2301. The body model 2302 generated by the 3D body model generation process 230 has 3D + 1D information like the meshed head model 2202 described above. That is, the body model 2302 is a body model with time information added.

The chart (b) in FIG. 23B is a schematic diagram showing the body model more concretely. Model 2310 shows an example in which a skeletal model (skeleton) is used to determine

3D control points

2311a, 2311b, 2311c, ... ing. The

3D control points

2311a, 2311b, 2311c, ... Generate a 3D deformable head model that can be adapted to different people and their body gestures and characteristics.

The model 2320 in the chart (b) of FIG. 23B schematically shows a state in which the driver is seated in the driver's seat of the vehicle. In this case, for example, based on the positional relationship between the steering 2321 and the model 2320, it is determined whether or not the driver is in a state where he / she can return to manual driving by looking at the angle of the upper body centered on the waist position 2312. can. In the example of FIG. 23B, when the angle is α, the driver's seat is in the reclining (non-driving position) state, and it can be determined that immediate return is difficult. On the other hand, when the angle is the angle β, the reclining state of the driver's seat is released and the driver's seat is set to the driving position, and it can be determined that immediate return is possible. It should be noted that this determination method is an example for explanation and is not limited thereto.

Next, an example of a method for determining the driver's awakening state will be described. FIG. 24 is a schematic diagram for explaining a method for determining a driver's wakefulness, which is applicable to the embodiment. First, the driver's face 261 is extracted from the captured image 260 captured by the in-vehicle camera, and the right eye 262R and the left eye 262L are further extracted from the extracted face 261. Further, the captured image 260 is predeterminedly adjusted by the driver adjustment process 251.

For the right eye 262R and the left eye 262L extracted from the face 261, the feature amount is extracted by the feature extraction process 2500 by the user agnostic classification process 250, and the eye condition is classified by the eye condition classification process 2501. .. As the feature extraction process 2500, various methods such as combination of HoG (Histograms of Oriented Gradients) and principal component analysis, detection of EAR (Eyes Aspect Ratio), measurement of corneal reflex point, and the like can be applied. Similarly, various classification methods such as SVM (Support Vector Machine), K-means method, and classification using a neural network can be applied to the eye condition classification process 2501.

In addition to the feature quantities extracted by the feature extraction process 2500 by the fusion and arousal estimation process 252, the eye states classified by the eye condition classification process 2501, and the captured image adjusted by the driver adjustment process 251. The arousal degree of the driver is estimated by fusing the physical information acquired by the physical monitoring 2520 and the head information acquired by the head monitoring 2521.

<3-6-5. Specific example of evaluation of quality of behavior according to embodiment>
Next, the evaluation of the quality of behavior according to the embodiment will be described using a specific example.

FIG. 25 is an example flowchart showing a process for evaluating the quality of behavior according to an embodiment. When the driver sits in the driver's seat, the automatic driving control unit 10112 performs personal authentication of the driver in step S500. Personal authentication can be performed, for example, based on the driver's face image captured by the in-vehicle camera. Not limited to this, it may be performed based on biometric information that can be obtained from fingers or the like.

In the next step S501, the automatic driving control unit 10112 acquires, for example, a statistical standard model 95 regarding the head and body (posture) of the human body stored in advance in the storage unit 10111, and the acquired statistical standard model 95 is used by the driver. Applies as a standard model of head and posture.

In the next step S502, the automatic driving control unit 10112 estimates the state of the driver's head and the joint position of the body using the skeleton. In the next step S503, the automatic driving control unit 10112 uses the skeleton to estimate the position state of the lower body of the driver. In the next step S504, the automatic driving control unit 10112 estimates the driver's state by facial expressions using the captured image obtained by capturing the driver's face with the in-vehicle camera. More specifically, the automatic driving control unit 10112 estimates fixation, fatigue, emotions, etc. based on the facial expression of the driver.

Next, the automatic driving control unit 10112 acquires the driver's self-awakening / return behavior learning data 96 and the driver's normal driving posture data 97, and starts monitoring the driver's activity state in step S505. do. The automatic driving control unit 10112 monitors the activity state of the driver such as NDRA by this monitoring. The driver's self-awakening / return behavior learning data 96 and the driver's normal driving posture data 97 are, for example, the driver. It can be obtained from the personal return characteristic dictionary 81.

The above-mentioned processes of steps S500 to S504 are the processes for the driver who does not have the self-awakening / return behavior learning data 96. In the case of a driver who already has the self-awakening / return behavior learning data 96, information such as a body model included in the driver's individual return characteristic dictionary 81 can be used instead of the statistical standard model 95.

In the next step S506, the automatic operation control unit 10112 calculates the return notification timing for notifying the driver of the return from the automatic operation to the manual operation. The automatic driving control unit 10112 can calculate the return notification timing based on, for example, LDM, information acquired in the driver's activity state monitoring started in step S505, and the like. Then, the automatic driving control unit 10112 gives various notifications such as a return request to the driver according to the calculated timing. This notification is made, for example, in the processing of the flowchart of FIG. 17B, and in particular in step S230.

In the next step S507, the automatic operation control unit 10112 weights each evaluation item. That is, the automatic driving control unit 10112 is based on the driver's response to various notifications made in step S506, and based on the relationship with the monitoring result regarding the priority evaluation item according to the activity state of the driver such as NDRA. Weight the priority evaluation items.

The set of steps S508a and S508b, the set of steps S509a and S509b, ..., The set of steps S513a and S513b show specific examples of weighting to the evaluation items, respectively. Of these, the set of steps S508a and S508b has the highest importance, and the more to the right in the figure, the lower the importance, and the set of steps S513a and S513b has the lowest importance.

The automatic driving control unit 10112 tracks the position of the driver in the vehicle in step S508a. For example, in step S508a, the position where the driver leaves the driver's seat and moves is tracked. In step S508b, the automatic driving control unit 10112 compares the tracking result acquired in step S508a with the expected value for tracking by the driver, and obtains an evaluation value normalized by the expected value.

For example, the expected value can be the parameter value when the driver is in perfect condition. That is, the expected value is the unique characteristic of the driver in the target item. By comparing the driver's parameter value acquired by monitoring with the expected value, it is possible to obtain an evaluation value normalized by the driver's unique characteristics.

The automatic driving control unit 10112 evaluates the movement of the driver's foot in step S509a. This makes it possible to know, for example, the operating state of the accelerator pedal and the brake pedal by the driver. In step S509b, the automatic driving control unit 10112 compares the evaluation value obtained in step S509a with the expected value for the movement of the foot in the driver, and normalizes the evaluation value with the expected value as the driver's unique characteristic. Ask.

The automatic driving control unit 10112 evaluates the posture and posture of the driver based on the body model of the driver in step S510a. This makes it possible to know, for example, whether or not the driver is in the reclining state of the driver's seat, and whether or not the driver is facing the steering. In step S510b, the automatic driving control unit 10112 compares the evaluation value obtained in step S510a with the expected value as the driver's unique characteristics with respect to the posture and posture of the driver, and determines the evaluation value normalized by the expected value. Ask.

Further, in step S5102, the automatic driving control unit 10112 responds to various notifications such as a return request made in step S506 based on the driver's body model, especially the arms and fingers (“OK”]. The automatic driving control unit 10112 evaluates the evaluation value obtained in step S510a in step S510b and the expected value as the driver's unique characteristics for the arm and finger in the driver. Compare and find the evaluation value normalized by the expected value.

The automatic driving control unit 10112 evaluates the driver's facial expression based on the driver's facial expression model in step S511a. As the facial expression model, the driver's head model can be applied. This makes it possible to estimate, for example, the driver's current emotions (normal, drowsy, irritated, angry, etc.). In step S510b, the automatic driving control unit 10112 compares the evaluation value obtained in step S510a with the expected value as the driver's unique characteristic for the facial expression of the driver, and obtains the evaluation value normalized by the expected value. ..

The automatic driving control unit 10112 evaluates the details of the behavior of the driver's eyeball in step S512a. For example, as described with reference to FIG. 24, the automatic driving control unit 10112 performs eye condition classification processing 2501 on the right eye 262R and the left eye 262L extracted from the driver's face 261 and performs eyeballs such as PERCLOS and saccade. Behavior can be obtained. This makes it possible to estimate, for example, whether the driver is concentrated in the front of the driving direction or is in a mind wandering state. In step S512b, the automatic driving control unit 10112 compares the evaluation value obtained in step S512a with the expected value as the driver's unique characteristic for the eyeball behavior of the driver, and obtains the evaluation value normalized by the expected value. ..

The automatic operation control unit 10112 evaluates other items in step S513a. In step S513b, the automatic operation control unit 10112 compares the evaluation value obtained in step S513a with the expected value for the other items, and obtains an evaluation value normalized by the expected value.

Whether or not the automatic operation control unit 10112 needs to switch the notification to the driver performed in step S506 to an alarm in step S514 after processing each of the set of steps S508a and S508b to the set of steps S513a and 513b. To judge. That is, based on the normalized evaluation values obtained in each of the set of steps S508a and S508b to the set of steps S513a and 513b, it is said that the quality of the return operation of the driver is deteriorated due to the delay of the operation or the like. If so, escalate from notification to alert.

When the automatic operation control unit 10112 determines in step S514 that switching is necessary (step S514, "Yes"), the process shifts to step S516. In step S516, the automatic driving control unit 10112 issues an alarm urging the driver to return to manual driving. Further, the automatic driving control unit 10112 deducts the evaluation value of the driver and applies a penalty to the driver. The automatic operation control unit 10112 may also start MRM depending on the situation.

The automatic operation control unit 10112 returns the process to step S506 after the process of step S516.

On the other hand, when the automatic operation control unit 10112 determines in step S514 that switching is unnecessary (step S514, "No"), the process shifts to step S515. In step S515, the automatic operation control unit 10112 performs a comprehensive evaluation regarding whether or not to return to manual operation based on the normalized evaluation values in each of the set of steps S508a and S508b to the set of steps S513a and 513b.

The automatic operation control unit 10112 calculates the comprehensive evaluation value nt according to the following equation (2). In addition, in the formula (2), "Ev" indicates the evaluation value for each evaluation item normalized by the expected value as the driver's unique characteristic. Further, "W" indicates a weight for each evaluation item. As shown in the formula (2), the evaluation values for each evaluation item are weighted corresponding to the evaluation items, and the weighted evaluation values are added up for all the evaluation items to calculate the total evaluation value nt.
tun = Σ (Ev × W)… (2)

Here, a comprehensive evaluation based on the evaluation values calculated by the set of steps S508a and S508b to the set of steps S513a and 513b described above will be described with reference to specific examples. Here, the normalized evaluation value is represented by a percentage%. That is, assuming that the evaluation value to be weighted is X and the weight as a percentage is W (%), the weighted evaluation value X'is calculated by the following equation (3).
X'= X × (W / 100)… (3)

In the position tracking of step S508a, when the driver leaves the driver's seat and moves to the loading platform, moves to the nap space, teleworks away from the seat, etc. are detected, the evaluation value is, for example, 100%. It is subject to priority evaluation.

Although the position tracking in step S508a does not detect leaving the seat, a secondary task with an inappropriate posture for driving, such as the driver sitting in the driver's seat and rotating the direction of the driver's seat, is used. Execution may be detected. In this case, paying attention to the evaluation of the foot movement in step S509a, emphasis is placed on the evaluation of the return behavior such that the driver's foot can step on the pedal such as the accelerator pedal and the brake pedal, and weighting of, for example, 80% is performed. Further, the return behavior based on the lower body of the body model is evaluated by weighting, for example, 20%.

Here, when the driver who has taken his foot off the pedal and rotated the driver's seat and crossed his legs returns to the driver's posture, he disengages his legs, moves his legs sideways, and sits in the driver's seat in the lower body posture. It is necessary to correct the surface. In that evaluation, the return movement speed is calculated from the observation information dropped into the foot skeleton model, and it takes time for the posture and posture movement normally expected of the driver, or the expected movement is not performed. It can happen.

In such a case, since there are individual differences in the behavioral transition characteristics, the evaluation value in the driver's normally successful return behavior is used by referring to the learning dictionary generated through repeated use based on the absolute observation value. Use the normalized evaluation value.

In addition, even if the driver who was away from the seat receives a request to return and is in a normal sitting posture, it is possible that the wakefulness has not been fully restored. Therefore, from the return notification to the driver, the evaluation by steps S508a and S508b and steps S509a and S509b is weighted, and the return behavior of the driver is observed. Immediately before the transfer of driving steering to the driver after sitting, the weight for steps S508a and S508b is set to 0%, and instead, the evaluation by steps S511a and S511b and the evaluation by steps S512a and S512b are added up. The weight is 100%. In this way, each evaluation value is weighted and the total evaluation value nt is obtained.

The explanation returns to FIG. 25, and in the next step S517, the automatic operation control unit 10112 determines whether or not the procedure for returning to the manual operation by the driver has been completed. When the automatic operation control unit 10112 determines that the process has not been completed (step S517, “No”), the process returns to step S506. In this case, if the return procedure by the driver proceeds smoothly and there is no delay, as the loop processing according to this step S506 to S517 progresses, the priority point of monitoring shifts from the left to the right in the figure, that is, step S508a and The process shifts to the right one by one from the set of S508b, and the priority points are sequentially changed.

On the other hand, when the automatic operation control unit 10112 determines that the return procedure is completed in step S517 (step S517, "Yes"), the process shifts to step S518. In step S518, the automatic driving control unit 10112 calculates addition / subtraction points such as rewards and penalties for the driver based on the comprehensive evaluation value nt calculated in step S515.

(Specific example of weighting)
Next, the weighting in the set of steps S508a and S508b to the set of steps S513a to S513b in the flowchart of FIG. 25 described above will be described more specifically with reference to Tables 4 to 6. Table 4 shows an example of QoA evaluation based on a driver-specific head model. In Tables 4 to 6, the "coefficient of determination / relevance coefficient" is a "weight" for the evaluation value of the item, and is a coefficient indicating the relevance of the item to QoA, as well as the QoA of the item. It is also the coefficient of determination used for the judgment of.

For the head model, if the extracted variables indicate authentication and the application of the standard model to classification analysis, the target acquisition information is personal authentication and the extracted features and information are personal authentication information. And, it is the feature information of the face such as eyes, nose, and mouth. The weight is set to 10%.

Similarly, for the head model, if the variables extracted are the head position, orientation, and parallax direction, the desired acquisition information is the driver's duty of care fulfillment, confirmation implementation rate determination, forward carelessness factor, Inattentiveness, forward confirmation, operation confirmation of parallax designation, evaluation confirmation of confirmation operation to system notification instruction, etc. The extracted feature amounts and information are, for example, driver's attention direction analysis, forward gaze, road sign confirmation, inattentiveness, side mirror line-of-sight movement, use of brought-in terminal, terminal browsing, navigation screen operation, and the like. The weight is 25%.

Furthermore, regarding the head model, if the extracted variable is a polygon model of facial expressions, the target acquired information is emotional evaluation. Examples of emotional evaluation include aggressive driving psychology, calm driving psychology, and evaluation of the degree of immersion in NDRA by bringing in terminal devices (watching sports, watching movies, games, etc.).

It is possible to infer how much attention is directed to the browsing content from the state of directing the line of sight to the content image at the time of NDRA by the line of sight. Furthermore, it is possible to estimate the emotional transfer status from facial expressions and physical gestures, and the larger the emotional transfer, the more the driver's working memory is occupied by the information related to the browsed content, which is necessary when returning to manual driving. There is a risk of lowering awareness (Situation Awareness). Therefore, the evaluation of this facial expression is important because it is one of the great clues as a judgment index when the driver returns to normal from automatic driving to manual driving.

In addition, by evaluating the presence or absence of forward confirmation and notification confirmation during NDRA by evaluation linked with the movement of the line of sight, evaluation of the duty of care withdrawal status for driving steering (while using automatic driving level 3, the unconfirmed state continued for several minutes. In some cases, it is a violation of the duty of care, etc.), and in addition, it causes drowsiness and tiredness.

The extracted features and information are the condition of the eyes and the condition of other facial parts. As for the condition of the eyes, the extracted features and information are the condition where the eyes are open, the eyelids are lower than usual, the eye opening speed is slowed down (sleepiness index), blinking, PERCLOS, fatigue due to eye movement, and sleepiness estimation. , And so on. As for the state of facial parts other than the eyes, the extracted feature quantities and information include deficiency, fatigue estimation by facial expression analysis, emotion estimation, disease estimation such as seizures (open mouth situation, painful facial expression, etc.), calmness and calmness. Deposition psychology, aggressive driving psychology, a state where the situation is not grasped due to waking up, etc.

The weight is 25% regardless of whether the extracted feature amount and information are in the state of the eyes or the state of other facial parts.

Table 5 shows an example of QoA evaluation based on the driver's body model.

Regarding the body model, if the extracted variable is the posture evaluation of the upper body in the driver's seat (the coordinates of the skeletal model including the fingers, hands, arms, feet, etc. of the upper body), the target acquisition information is Indexing of behavioral evaluations that lead to violations of driving steering and duty of care, eating and drinking, reading, unacceptable behavior during driving (navigation operation, mail, searching for cigarettes, wallets, cameras, etc.), degree of hand blockage, etc. The extracted features and information include whether or not the steering is touched, eating and drinking while talking, smoking, what to look for, operation of the mobile terminal, and so on. The weight is said to be 15%.

Similarly, for the body model, by further analyzing the transition along the time axis, the NDRA immersiveness evaluation of the behavior is extracted as a variable, and the confirmation motion evaluation such as pointing and calling such as road front confirmation is also used as a variable. Be extracted.

The target acquisition information is normal behavior, return transition behavior evaluation after return request, notification content confirmation operation after notification, and so on. The return transition behavior evaluation includes a behavior quality evaluation that evaluates whether the driver has swiftly returned to the driver's seat and returned to the driving posture after the return notification from the movement analysis of the limbs.

Of course, the return request notification is designed to be issued at the required time due to the harmful effect of issuing it earlier than necessary, and the driver's behavior evaluation is incorporated as a return preparation from the advance notification sound that will be notified to the driver from now on. You may. In particular, once involved in NDRA for a long time, the situation in front of the road may change as the itinerary progresses, such as the scenery, the situation of surrounding vehicles, and even the weather immediately after receiving the return request. be. Information necessary for situation awareness in stages by the driver looking forward to check the road condition and checking the updated display information prior to the actual return notification with the notification sound. It is also an effective early recovery procedure to start taking in, and the evaluation for them may be extended and used as a reward target.

The target acquisition information also includes delays based on steady-state stable return and quick return quality evaluation by learning. Even if it is a fast movement, it is determined whether or not it is a hasty movement, and if it is determined to be a hasty movement, it is classified as a low quality return behavior. That is, if the start of the return is delayed and an attempt is made to recover in a hurry, the risk of impairing safety increases, so the purpose is to avoid such a rushed return.

The extracted features and information are the driving steering posture (whether driving steering is possible or not), and if it is outside the driving posture, the return delay prediction, etc. The weight is set to 20%.

Table 6 shows an example of QoA evaluation based on the driver's foot movement evaluation and an example of QoA evaluation based on the detailed evaluation of eyeball behavior.

Regarding the movement of the driver's foot, the extracted variable is the evaluation of the movement based on the skeleton model of the foot, and the target acquired information is whether or not the brake pedal and accelerator pedal can be operated immediately. If not, the time required for recovery is predicted. As for the time required for recovery, the delay time required from the initial state is estimated by learning. The extracted feature amount and information are the occurrence of a posture change that takes time to return to the driving posture, such as moving away from the driving steering behavior, crossing the legs, rotating the driver's seat, and the like. The weight is said to be 15%.

Regarding the driver's eye behavior, the extracted variables are the expression evaluation of saccade, microsaccade, and fixation. These are extracted based on the evaluation of the coordinates of the eyeball behavior and the polar coordinates. The target acquired information is whether or not the degree of arousal can be searched by visual information, whether or not the degree of attentiveness is high, and the like. The extracted feature amount and information include the presence / absence of visual information search, the presence / absence of reference to visual information storage, the presence / absence of notification information confirmation, and the like. The weight is an index of the degree of arousal depending on the presence or absence of the behavior of the information confirmation search, and is 0% for undetected eyeball behavior, 80% for continuous linear confirmation, and 100% for high-frequency confirmation by task response. ..

Further, regarding the driver's eye movement, even if the extracted variables and the target acquired information are the same as described above, the extracted feature amount and information are different from the above, and the visual information search for the content of the notification information is performed. For example, in the case of eyeball behavior of message confirmation of notification content, expression evaluation of macrosaccade behavior in the vicinity of fixation, etc., the weights are also different from the above. In this case, the weight is 0% without response to the transfer point change notification or update notification, and 100% when the notification content is promptly notified after the notification and the cause is visually confirmed.

(3-6-6. DMS summary according to the embodiment)
Techniques for assessing the driver's condition itself are known. On the other hand, later depending on whether the system decides whether the driver is actually ready to return and whether the driver is willing to follow the RTI (Driving Change Request) instructions issued by the system. Nothing is known about the quantification of "quality of return motion" that makes it difficult to assess motion performance for reward determination. Without a reward feedback method according to the severity of the RTI instructions, it is difficult to hold the driver responsible for prioritizing critical situations.

In this case, the driver's working memory remains at a monotonous level of attention that cannot attempt to increase attention even in situations where recovery is strongly required, resulting in MRM execution on busy roads and rear. There is an increased risk of collisions and traffic congestion on certain roads.

Therefore, in the DMS (Driver Monitoring System) according to the embodiment, as described above, the driver is monitored by using the method described below.

(1) Directly monitor the driver The driver's readiness for the driving task is monitored using the direct sensing technology.
(2) Estimate and monitor the driver's response from the viewpoint of the time required to return from the non-driving task in the automatic driving mode to the driving task (manual driving) and the appropriateness of the driver's position.
(3) By monitoring the driver's non-driving behavior, an index regarding the driver's distraction is estimated.
(4) Personalized monitoring of the driver is performed by monitoring the driver's response and preparation status based on the driving position that the driver deems appropriate.
(5) Parameterize the driver's characteristics related to the position and behavior of each part of the driver and record them for statistical analysis over time. It can be used to adapt the system in time and to refer to driver behavior.
(6) The detected driver readiness and response are used to design the optimal driver system with the best response to warnings and interventions at

automated driving levels

3 and 4.
(7) Obtain a 3D model of the driver's head or face, upper body, legs, arms, and hands, and adapt it to the conditions of a specific person and driver (for example, emotions, fatigue, illness, distraction, etc.). Let me.
(8) In order to estimate the driver's behavior with high accuracy, the movement of the entire body skeleton of the driver is tracked.
(9) Estimate driver attitudes and behaviors now and in the near future based on common driver behaviors, personalized general behaviors, and a series of behaviors in the near past in specific situations (such as ODD and environmental conditions). do.
(10) Build an HMI that gives feedback to the driver about the quality of the behavior related to the return that was detected and analyzed.
(11) Construct a feedback HMI that is composed of at least visual information and further includes an acoustic or tactile HMI.
(12) Provide first level feedback information that is timely without delay of several seconds or up to 10 seconds or more from detection.
(13) The quality of behavior (QoA), which is presented to the driver along with the credibility of the quality of the driver's past return by visual feedback, is indexed.
(14) In order to avoid data falsification, data is recorded and stored in a non-volatile data storage that can be searched by a predetermined method, and the data is analyzed.

As a technique for realizing each of these items, a 3D information acquisition technique for acquiring luminance (or RGB) information and depth information is required. As an example for realizing such a 3D information acquisition technology, a D-ToF (Direct-Time of Flight) camera can be applied. 3D information acquisition and processing accurately identifies and recognizes a particular driver's condition (eg, wakefulness, distraction, lesions, etc.) and dynamically characterizes the driver's posture, body and body parts. Enables.

Furthermore, it is possible to generate a 3D head / face model of the driver from the depth information and the brightness information, and adapt the driver to the 3D model according to the input of the depth information and the brightness information.

As a result, a 3D mesh model of the driver can be obtained with a limited number of control points. This makes it possible to perform rigid transformation (head position) and face gesture and eye condition monitoring by non-rigid deformation with high accuracy.

In addition to customizing the driver's personal system, it is also necessary to acquire the 3D position of the driver's body part. This mainly includes the skeleton of the body and the position and orientation of the head. That is, the position and orientation of the head play an important role in efficiently extracting facial features and facial gestures necessary for monitoring the condition of the eyes. Body skeleton tracking, on the other hand, plays an important role in monitoring the condition and posture of the driver's hands and in motion monitoring used to assess the driver's behavior with respect to the behavior expected in known ODDs. ..

Furthermore, the arms, legs and hands can be aligned with high accuracy in the 3D domain using 3D information acquisition technology. This not only makes it possible to predict future driver behavior with higher accuracy, but also makes it possible to greatly simplify the acquisition of body postures and movements related to driver behavior.

The above-mentioned DMS that can be customized for each driver needs to be learned for each driver about the features extracted from the luminance information and depth information of the 3D sensing technology. The following items can be considered as examples of the contents to be learned by the driver.

・ Sit in the normal driver's position and face the windshield.
・ Sit in the normal driver's position and face the camera.
・ Drive in accordance with the rules according to the variable outside and inside conditions.
-Monitoring behavior with specifically defined HMI interactions.

Also, if it is an efficient system, the number of sensors installed in the vehicle can be minimized. Therefore, camera position and camera technology with a sufficiently wide field of view are very important in designing a driver surveillance system.

It is desirable that the position of the camera is a position where the entire body of the driver can be imaged. Further, it is more preferable to arrange the camera so that the occlusion is minimized with respect to the body of the driver who is the subject. The field of view of the camera is preferably wide enough to simultaneously monitor different parts of the body and its surroundings (eg, passengers).

Furthermore, it is preferable to be able to apply surveillance camera technology that can simultaneously acquire 3D information necessary for accurate 3D model positioning and 2D images or texture contents. That is, in order to detect the driver's behavior with high accuracy, 3D information that is important for reliably monitoring all body parts of the driver, and 2D images and textures that are important for extracting facial features. You need to be able to get the content.

Furthermore, it is preferable that the compatibility with noise variables is high because it is easy to detect the behavior of the driver with high accuracy. For example, it is conceivable to improve the robustness against fluctuating lighting conditions in combination with the multiple exposure method. It is also effective to arrange the camera so that the occlusion is minimized as described above in terms of compatibility with noise variables. In addition, learning different types of drivers (age, ethnicity, etc.) and their behavior is also effective.

In the DMS according to the embodiment of the present disclosure, by quantifying the "quality of action" of the return action when the driver returns from the automatic driving to the manual driving, the automatic driving level 3 and the automatic driving level 4 are automatically performed. In driving, the system can evaluate and predict the driver's behavior when the system requires the driver to resume manual driving after automatic driving.

In the DMS according to the embodiment, the system can appropriately determine the evaluation value of the personalized reward and penalty for the driver's movement based on the result obtained from this series of evaluation processes. Thereby, by applying the DMS according to the embodiment, the driver is not forced by the system, but suspends or discontinues the continuation of the NDRA, and returns to the manual operation at an early stage according to the intervention request from the system. , It is possible to directly influence the prioritization of actions in the driver's consciousness.

In this specification, examples are described focusing on the applications of automatic driving, which are mainly called automatic driving level 3 and automatic driving level 4. On the other hand, general vehicles are not expected to be used specifically for a specific level of autonomous driving, and functionally, autonomous driving is a condition that enables more advanced autonomous driving without the intervention of the driver. It is only level 3 of the automatic driver or level 4 of the automatic driver.

Therefore, when the driver actually uses the vehicle with the performance, depending on the environment and other conditions, the assumed automatic driving level 1 or automatic driving level 2 limited to driving support may be used. Any combination can be used. These usage categories are merely definition terms that have been used in the process of discussing vehicle and system design perspectives and usage systems, and from the driver's usage perspective, the usage boundaries between them will continue to be defined as explicit. Usage is not always limited to the street.

Furthermore, even if the system is automatic driving level 2, if more advanced driving support is provided, the driver will provide it as automatic driving support, which is the "automatic driving function under specific conditions" of automatic driving level 2. Even in the ODD state, if the risk remains extremely small in a stable road driving section, the feeling of using "conditional automatic driving" of automatic driving level 3 will be the use of this more advanced automatic driving level 2. In addition to having little difference from the sensation, it is also required to monitor the decrease in the alert state of the driver for preventive safety caused by the decrease in attention that occurs when using the automatic driving level 2.

Therefore, in addition to the examples in the present embodiment, it is obvious that the functions described in the present embodiment can be extended and used in the driver monitoring at the time of use from the automatic driving level 0 to the automatic driving level 2. The automatic driving function of automatic driving level 3 or lower is not described separately as an example. That is, the scope of application of the embodiment is not limited to the categories described according to the definition of the level of automatic operation of SAE.

<Supplement>
In the above-described embodiment, when a person, that is, a driver uses an automatic driving or an advanced driving support function, the automatic driving function is a change in behavioral judgment psychology caused by a psychological sense of security that can occur by introducing these functions. It provides measures to reduce the attention assigned for safe steering behavior judgment required during existing manual driving due to excessive dependence on.

As a countermeasure, we provide a behavior evaluation method from the behavior observation of the driver when the driver uses the vehicle, and use the indexed numerical value to directly or indirectly tell the driver whether the driver's behavior is good or bad. Therefore, the system feeds back to the driver according to the conditions allowed by the driving conditions. More specifically, the influence of the driver's own behavior, how it is reflected on the behavioral result, one or more of visual, auditory, tactile, and odor. It is fed back to the driver. This feedback gives the driver the influence of his or her own behavior as a direct or indirect reward or penalty through physical, psychological and institutional means, thereby giving the driver a behavioral psychology. The present invention relates to a technique for promoting behavior change to prevent or reduce dependence on the above-mentioned automatic driving system through repeated use.

The driver may be overly dependent on the automatic driving function provided by the system through the use of automatic driving, and may not be able to expect the driver's manual driving response beyond the limit range that the system can provide the automatic driving function. In the present disclosure, even in such a case, the system provides an emergency stop of the vehicle in the road section of the vehicle, for example, by MRM as a risk minimization process according to the necessity of the road environment which may be the key to social activities. In order to prevent and to achieve the behavioral psychology for the driver to take appropriate early return behavior in response to the request for return to manual driving, the behavioral psychology works correctly and the hierarchy according to the importance. It relates to a technology that can provide information presentation that enables objective risk judgment with a change in circumstances, that is, an intuitive sense of approaching time with the passage of time.

Furthermore, the superiority or inferiority of the driver's response in line with the information presentation that enables accurate risk judgment provided by the system related to this disclosure, that is, the required behavioral quality evaluation at the time of returning to manual driving, promptly and accurately The return behavior is learned as a reward, benefit, and disadvantage that the driver can obtain. The automatic driving system according to the present disclosure is composed of the following series of technologies that provide a control method in line with human behavioral psychology while drawing out the advantages of the automatic driving function required by society from an ergonomic point of view. Will be done.

・ Driver's behavior quality evaluation technology ・ Driver's behavior evaluation result indexing technology ・ HMI that predicts risk judgment and its progress to be provided to the driver
・ HMI that provides non-monotonic risk information in chronological order by providing hierarchical risk-specific information to drivers
・ Visualization of future risks at the time of action judgment for the driver (future penalties for addition / subtraction display, feedback display of restrictions) Memory projection to working memory by HMI ・ Use of automatic driving such as ODD for the driver Intuitive and timely provision of tolerances

The autonomous driving system according to the present disclosure provides the driver with time-series risk fluctuations, risk importance, and options through HMI through these technologies, etc., so that the driver feels when using autonomous driving. It provides a function that enables the driver's behavior change and self-learning to maximize the merit on the risk balance.

Then, by using the technology related to this disclosure in the driver's natural sense of use, the influence of the actions taken by the driver himself when using automatic driving can be visualized in the near future. As a result, the autonomous driving system according to the present disclosure has negative social impacts such as traffic congestion that cannot be directly felt and may be caused by excessive dependence on MRM when using autonomous driving, and as a result, a rear-end collision. Can be projected onto the driver with the appropriate HMI. Thereby, the automatic driving system according to the present disclosure can provide a control suitable for use for the purpose of incorporating human behavioral psychology and not hindering social activities. Other effects and methods of implementation that are obvious to those skilled in the art can be achieved.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

The present technology can also have the following configurations.
(1)
The acquisition unit that acquires the state of the driver of the vehicle,
An automatic driving control unit that controls automatic driving to drive the vehicle autonomously,
Equipped with
The automatic operation control unit is
Based on the state of the driver acquired by the acquisition unit, the return quality, which is the quality of action when the driving of the vehicle returns from the automatic driving to the manual driving by the driver's operation, is obtained and obtained. By quantifying the return quality, the driver is monitored for the driver.
Information processing equipment.
(2)
The automatic operation control unit is
Control is performed to evacuate the vehicle according to the situation during the automatic driving of the vehicle, and the driver is monitored in order to return from the automatic driving to the manual driving before the evacuating driving.
The information processing apparatus according to (1) above.
(3)
The acquisition unit
Obtaining the skeleton information and face information of the driver,
The automatic operation control unit is
The driver monitoring is performed based on the skeleton information and the face information of the driver.
The information processing apparatus according to (1) or (2) above.
(4)
The automatic operation control unit is
The skeleton information and the face information acquired by the acquisition unit are parameterized, and the return quality is quantified based on the parameters generated by the parameterization.
The information processing apparatus according to (3) above.
(5)
The automatic operation control unit is
The position of the driver in the vehicle is tracked based on the skeletal information, and the return quality is obtained based on the tracked position.
The information processing apparatus according to (3) or (4) above.
(6)
The automatic operation control unit is
The movement of the driver's foot is detected based on the skeletal information, and the return quality is obtained based on the detected movement of the foot.
The information processing apparatus according to any one of (3) to (5).
(7)
The automatic operation control unit is
The position and orientation of the driver's head are detected based on the skeleton information and the face information, and the return quality is obtained based on the detected position and orientation of the head.
The information processing apparatus according to any one of (3) to (6).
(8)
The automatic operation control unit is
The behavior of the driver's eyeball is detected based on the facial information, and the return quality is obtained based on the detected behavior of the eyeball.
The information processing apparatus according to any one of (3) to (7).
(9)
The automatic operation control unit is
The driver's wakefulness is estimated based on the behavior of the eyeball, and the return quality is obtained based on the estimated wakefulness.
The information processing apparatus according to (8) above.
(10)
The automatic operation control unit is
The seated state of the driver in the driver's seat of the vehicle is detected based on the skeleton information, and the return quality is obtained based on the detected seated state.
The information processing apparatus according to any one of (3) to (9).
(11)
The automatic operation control unit is
Based on the skeleton information, it is detected as the seated state whether the driver's seat is in the non-driving position state or the driving position state.
The information processing apparatus according to (10) above.
(12)
The automatic operation control unit is
Based on the skeleton information, the response operation to the notification issued to the driver is detected, and the return quality is obtained based on the detected response operation.
The information processing apparatus according to any one of (3) to (11).
(13)
The skeleton information and the face information each include time information.
The information processing apparatus according to any one of (3) to (12).
(14)
The automatic operation control unit is
The quantified return quality is weighted for each evaluation item for which the return quality is evaluated, and the return quality for each of the quantified and weighted evaluation items is added up to obtain the driver. To obtain the comprehensive evaluation value of the return quality of
The information processing apparatus according to any one of (1) to (13).
(15)
Performed by the processor,
The acquisition process to acquire the state of the driver of the vehicle and
An automatic driving control process that controls automatic driving to drive the vehicle autonomously,
Including
The automatic operation control process is
Based on the state of the driver acquired in the acquisition process, the return quality, which is the quality of the action when the driving of the vehicle returns from the automatic driving to the manual driving by the driver's operation, is obtained and obtained. By quantifying the return quality, the driver is monitored for the driver.
Information processing method.
(16)
On the computer
The acquisition process to acquire the state of the driver of the vehicle and
An automatic driving control process that controls automatic driving to drive the vehicle autonomously,
To execute,
The automatic operation control process is
Based on the state of the driver acquired in the acquisition process, the return quality, which is the quality of the action when the driving of the vehicle returns from the automatic driving to the manual driving by the driver's operation, is obtained and obtained. By quantifying the return quality, the driver is monitored for the driver.
Information processing program for.

50, 50a, 50b, 50c Bird's-eye view display 51a Short-distance display section 51b Medium-distance display section 51c Long-distance display section 53a Automatic operation possible section 53b Return posture maintenance section 53c Operation return required section 80 LDM initial data 81 Driver's individual return characteristic dictionary 82 RRR
83 Vehicle dynamics characteristics 85 Road environment data 86 Own vehicle information 90 Road environment static data 91 Installed equipment information 92 Road compatibility information 93 Driver support availability information 95 Statistical standard model 96 Self-awakening / return behavior learning data 97 Driver's normal Driving posture data 100 HMI
101 Driver return delay evaluation unit 102 Driving road advance predictability acquisition range estimation unit 103 Remote support control / steering support availability monitoring unit 104 Driver behavior change achievement level estimation unit 105 Own vehicle driving road performance information provision unit 106 ODD application estimation Unit 107 Automatic driving permission integrated control unit 108 Driver behavior quality evaluation unit 132 Face / upper body / eyeball camera 133 Body posture / head camera 140 High freshness update LDM
200 Driver

behavior evaluation unit

201a, 201b Learning unit 202 Evaluation unit 220 3D head model generation processing 230 3D body model generation processing 250 User unknown classification processing 1010 Driver behavior response evaluation unit 1040 Excellent return steering behavior evaluation point addition unit 1041 Penalty action cumulative addition recording unit 2000 Driver information generation unit 2001, 2104 Parameter generation unit 2002 Driver DB
2003 Conformity unit 2004 Monitoring / extraction / conversion unit 2005 Preparation status evaluation unit 2006 Buffer memory 2100 3D head model generation unit 2101 Face identification information extraction unit 2102 Body model generation unit 2103 Model expansion unit 2105 Storage unit 2202 Meshed head model 2302 Body model 10101 Input unit 10112 Automatic operation control unit

Claims

The acquisition unit that acquires the state of the driver of the vehicle,
An automatic driving control unit that controls automatic driving to drive the vehicle autonomously,
Equipped with
The automatic operation control unit is
Based on the state of the driver acquired by the acquisition unit, the return quality, which is the quality of action when the driving of the vehicle returns from the automatic driving to the manual driving by the driver's operation, is obtained and obtained. By quantifying the return quality, the driver is monitored for the driver.
Information processing equipment.
The automatic operation control unit is
Control is performed to evacuate the vehicle according to the situation during the automatic driving of the vehicle, and the driver is monitored in order to return from the automatic driving to the manual driving before the evacuating driving.
The information processing apparatus according to claim 1.
The acquisition unit
Obtaining the skeleton information and face information of the driver,
The automatic operation control unit is
The driver monitoring is performed based on the skeleton information and the face information of the driver.
The information processing apparatus according to claim 1.
The automatic operation control unit is
The skeleton information and the face information acquired by the acquisition unit are parameterized, and the return quality is quantified based on the parameters generated by the parameterization.
The information processing apparatus according to claim 3.
The automatic operation control unit is
The position of the driver in the vehicle is tracked based on the skeletal information, and the return quality is obtained based on the tracked position.
The information processing apparatus according to claim 3.
The automatic operation control unit is
The movement of the driver's foot is detected based on the skeletal information, and the return quality is obtained based on the detected movement of the foot.
The information processing apparatus according to claim 3.
The automatic operation control unit is
The position and orientation of the driver's head are detected based on the skeleton information and the face information, and the return quality is obtained based on the detected position and orientation of the head.
The information processing apparatus according to claim 3.
The automatic operation control unit is
The behavior of the driver's eyeball is detected based on the facial information, and the return quality is obtained based on the detected behavior of the eyeball.
The information processing apparatus according to claim 3.
The automatic operation control unit is
The driver's wakefulness is estimated based on the behavior of the eyeball, and the return quality is obtained based on the estimated wakefulness.
The information processing apparatus according to claim 8.
The automatic operation control unit is
The seated state of the driver in the driver's seat of the vehicle is detected based on the skeleton information, and the return quality is obtained based on the detected seated state.
The information processing apparatus according to claim 3.
The automatic operation control unit is
Based on the skeleton information, it is detected as the seated state whether the driver's seat is in the non-driving position state or the driving position state.
The information processing apparatus according to claim 10.
The automatic operation control unit is
Based on the skeleton information, the response operation to the notification issued to the driver is detected, and the return quality is obtained based on the detected response operation.
The information processing apparatus according to claim 3.
The skeleton information and the face information each include time information.
The information processing apparatus according to claim 3.
The automatic operation control unit is
The quantified return quality is weighted for each evaluation item for which the return quality is evaluated, and the return quality for each of the quantified and weighted evaluation items is added up to obtain the driver. To obtain the comprehensive evaluation value of the return quality of
The information processing apparatus according to claim 1.
Performed by the processor,
The acquisition process to acquire the state of the driver of the vehicle and
An automatic driving control process that controls automatic driving to drive the vehicle autonomously,
Including
The automatic operation control process is
Based on the state of the driver acquired in the acquisition process, the return quality, which is the quality of the action when the driving of the vehicle returns from the automatic driving to the manual driving by the driver's operation, is obtained and obtained. By quantifying the return quality, the driver is monitored for the driver.
Information processing method.
On the computer
The acquisition process to acquire the state of the driver of the vehicle and
An automatic driving control process that controls automatic driving to drive the vehicle autonomously,
To execute,
The automatic operation control process is
Based on the state of the driver acquired in the acquisition process, the return quality, which is the quality of the action when the driving of the vehicle returns from the automatic driving to the manual driving by the driver's operation, is obtained and obtained. By quantifying the return quality, the driver is monitored for the driver.
Information processing program for.