JP6840035B2 - Vehicle control system - Google Patents

Description

The present invention relates to an imaging display unit.

Conventionally, there has been disclosed a device that superimposes a sound input by a microphone on an image captured by a camera to generate a call image of a videophone and transmits the call image to a other party's call device via a communication network. (See, for example, Patent Document 1). Further, there is disclosed a driver monitoring device that detects a driver's posture collapse based on an image of the driver's seat of the vehicle taken by a camera and notifies the driver of the posture collapse (for example, Patent Document 2). reference).

Japanese Unexamined Patent Publication No. 2015-201669 Japanese Unexamined Patent Publication No. 2016-38793

However, in the conventional technique, the mode of the image pickup unit when the use of the image captured as described above is different has not been taken into consideration.

The present invention has been made in consideration of such circumstances, and one of the objects of the present invention is to provide an image pickup display unit in which image pickup units for capturing images for different purposes are provided in an appropriate arrangement.

(1): An imaging display unit mounted on a vehicle, which is a display unit for displaying an image and a first imaging unit for imaging an occupant of the vehicle, and images used for monitoring the state of the occupant. A first imaging unit for imaging and a second imaging unit for imaging the occupants of the vehicle, the second imaging unit for capturing an image used for a service linked with the image displayed by the display unit. , Is an imaging display unit arranged in the unit.

(2): The image pickup display unit according to (1), wherein the recognition degree by the occupant is made different between the first image pickup unit and the second image pickup unit.

(3): The imaging display unit according to (1) or (2), wherein the second imaging unit is provided at a position where the occupant can see the display unit. Is.

(4): The imaging display unit according to (1) to (3), wherein the second imaging unit is provided at a position where the occupant can see the display unit. Is.

According to (1) to (4), imaging units for capturing images for different purposes can be provided in an appropriate arrangement.

It is a block diagram of the vehicle system 1 of an embodiment. It is a figure which shows a mode that the relative position and posture of the own vehicle M with respect to the traveling lane L1 are recognized by the own vehicle position recognition unit 322. It is a figure which shows how the target track is generated based on the recommended lane. It is a figure for demonstrating the process at the time of lane change. It is a figure for demonstrating the process at the time of lane change. It is a figure which shows an example of HMI 400 in own vehicle M. It is a figure which shows an example of the appearance of the 2nd display unit 460. It is sectional drawing of the 2nd display unit 460 of FIG. 7 in the 8-8 direction. It is a figure which shows various scenes from manual driving to automatic driving start of own vehicle M, and lane change by automatic driving is executed. It is a figure which shows an example of the 1st screen IM1-1 and the 2nd screen IM2-1 displayed at the time of manual operation. It is a figure which shows an example of the 3rd screen IM3-1 and the 4th screen IM4-1 which are displayed when the main switch 412 is operated. It is a figure which shows an example of the screens IM3-2 and IM4-2 which are displayed on the 1st display part 450 and HUD470 when the auto switch 414 is operated. It is a figure which shows an example of the screen which is displayed on the 1st display part 450 and HUD470 during the driving support of the 2nd degree. It is a figure which shows an example of the 3rd screen IM3-4 and the 4th screen IM4-4 which are displayed at the 1st timing before the behavior of the own vehicle M changes. It is a figure which shows various scenes from executing the driving support in the 3rd degree to the own vehicle M, and then executing the driving support in the 3rd degree to the 2nd degree. It is a figure which shows an example of the 3rd screen IM3-5 and the 4th screen IM4-5 which are displayed at the time of low-speed follow-up running. It is a figure which shows an example of the 3rd screen IM3-6 and the 4th screen IM4-6 which are displayed in order to make an occupant perform peripheral monitoring. It is a figure which shows the switching timing related to the driving support.

Hereinafter, embodiments of the imaging display unit of the present invention will be described with reference to the drawings. In the embodiment, it is assumed that the vehicle system is applied to an autonomous driving vehicle capable of autonomous driving (autonomous driving). Autonomous driving means that the vehicle runs in a state that does not require the operation by the occupant in principle, and is considered to be a kind of driving support. The vehicle can also be driven by manual driving. In the following description, the "occupant" shall mean an occupant seated in a driver's seat, that is, a seat provided with a driver's operator.

The degree of driving support in the present embodiment includes, for example, a first degree, a second degree, and a third degree. The first degree of driving support is executed by operating a driving support device such as ACC (Adaptive Cruise Control System) or LKAS (Lane Keeping Assistance System). The second degree of driving assistance has a higher degree of control than the first degree and automatically controls at least one of acceleration / deceleration or steering of the vehicle without the occupant performing any operation on the driving operator of the vehicle. It is a driving support that imposes an obligation to monitor the surroundings of the occupants, although it executes automatic driving. The third degree of driving support has a higher degree of control than the second degree of driving support and does not impose an obligation to monitor the surroundings of the occupants. In the present embodiment, the second degree and the third degree of driving support correspond to automatic driving.

[overall structure]
FIG. 1 is a configuration diagram of a vehicle system 1 including the vehicle control system of the embodiment. The vehicle on which the vehicle system 1 is mounted (hereinafter referred to as the own vehicle M) is, for example, a vehicle such as a two-wheeled vehicle, a three-wheeled vehicle, or a four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or the like. Alternatively, it is a combination of these. The electric motor operates by using the power generated by the generator connected to the internal combustion engine or the discharge power of the secondary battery or the fuel cell.

The vehicle system 1 includes, for example, a camera 10, a radar device 12, a finder 14, an object recognition device 16, a communication device 20, a navigation device 50, an MPU (Map Positioning Unit) 60, a vehicle sensor 70, and the like. A driving operator 80, a master control unit 100, a driving support control unit 200, an automatic driving control unit 300, an HMI (Human Machine Interface) 400, a traveling driving force output device 500, a braking device 510, and a steering device. 520 and. These devices and devices are connected to each other by a multiplex communication line such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, or the like. The configuration shown in FIG. 1 is merely an example, and a part of the configuration may be omitted or another configuration may be added.

The camera 10 is a digital camera that uses a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). One or a plurality of cameras 10 are attached to an arbitrary position of the own vehicle M on which the vehicle system 1 is mounted. When photographing the front, the camera 10 is attached to the upper part of the front windshield, the back surface of the rearview mirror, and the like. When photographing the rear, the camera 10 is attached to the upper part of the rear windshield, the back door, or the like. When taking a side image, the camera 10 is attached to a door mirror or the like. The camera 10 periodically and repeatedly images the periphery of the own vehicle M, for example. The camera 10 may be a stereo camera.

The radar device 12 radiates radio waves such as millimeter waves around the own vehicle M, and detects radio waves (reflected waves) reflected by the object to detect at least the position (distance and orientation) of the object. One or a plurality of radar devices 12 may be attached to any position of the own vehicle M. The radar device 12 may detect the position and speed of the object by the FMCW (Frequency Modulated Continuous Wave) method.

The finder 14 is a lidar (Light Detection and Ranging or Laser Imaging Detection and Ranging) that measures scattered light with respect to the irradiation light and detects the distance to the target. One or a plurality of finder 14s may be attached to any position of the own vehicle M.

The object recognition device 16 performs sensor fusion processing on the detection results of a part or all of the camera 10, the radar device 12, and the finder 14, and recognizes the position, type, speed, and the like of the object. The object recognition device 16 outputs the recognition result to the driving support control unit 200 and the automatic driving control unit 300.

The communication device 20 communicates with another vehicle existing in the vicinity of the own vehicle M by using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or wirelessly. Communicates with various server devices via the base station. Further, the communication device 20 communicates with a terminal device owned by a person outside the vehicle.

The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a routing unit 53, and the first map information 54 is stored in a storage device such as an HDD (Hard Disk Drive) or a flash memory. Holds. The GNSS receiver identifies the position of the own vehicle M based on the signal received from the GNSS satellite. The position of the own vehicle M may be specified or complemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 70. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, and the like. The navigation HMI 52 may be partially or wholly shared with the HMI 400 described later. The route determination unit 53, for example, has a route from the position of the own vehicle M (or an arbitrary position input) specified by the GNSS receiver 51 to the destination input by the occupant using the navigation HMI 52 (for example,). (Including information on waypoints when traveling to the destination) is determined with reference to the first map information 54. The first map information 54 is, for example, information in which a road shape is expressed by a link indicating a road and a node connected by the link. The first map information 54 may include road curvature, POI (Point Of Interest) information, and the like. The route determined by the route determination unit 53 is output to the MPU 60. Further, the navigation device 50 may perform route guidance using the navigation HMI 52 based on the route determined by the route determination unit 53. The navigation device 50 may be realized by, for example, the function of a terminal device such as a smartphone or a tablet terminal owned by the user. Further, the navigation device 50 may transmit the current position and the destination to the navigation server via the communication device 20 and acquire the route returned from the navigation server.

The MPU 60 functions as, for example, a recommended lane determination unit 61, and holds the second map information 62 in a storage device such as an HDD or a flash memory. The recommended lane determination unit 61 divides the route provided by the navigation device 50 into a plurality of blocks (for example, divides the route every 100 [m] with respect to the vehicle traveling direction), and refers to the second map information 62 for each block. Determine the recommended lane. The recommended lane determination unit 61 determines which lane to drive from the left. The recommended lane determination unit 61 determines the recommended lane so that the own vehicle M can travel on a reasonable traveling route to proceed to the branch destination when there is a branching point, a merging point, or the like on the route.

The second map information 62 is more accurate map information than the first map information 54. The second map information 62 includes, for example, information on the center of the lane, information on the boundary of the lane, and the like. Further, the second map information 62 may include road information, traffic regulation information, address information (address / zip code), facility information, telephone number information, and the like. Road information includes information indicating the type of road such as highways, toll roads, national roads, and prefectural roads, the number of lanes on the road, the area of the emergency parking zone, the width of each lane, the slope of the road, and the position of the road (longitudinal). , Latitude, three-dimensional coordinates including height), curvature of lane curves, positions of lane confluences and branch points, signs provided on roads, and the like. The second map information 62 may be updated at any time by accessing another device using the communication device 20.

The vehicle sensor 70 includes a vehicle speed sensor that detects the speed of the own vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around the vertical axis, an orientation sensor that detects the direction of the own vehicle M, and the like.

The driving operator 80 may include, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel and other operators. A sensor for detecting the amount of operation or the presence or absence of operation is attached to the operation operator 80, and the detection result is the master control unit 100, the operation support control unit 200, the automatic operation control unit 300, or the driving force. The output is output to any one or more of the output device 500, the brake device 510, and the steering device 520.

[Various control devices]
The vehicle system 1 includes, for example, a master control unit 100, a driving support control unit 200, and an automatic driving control unit 300 as a part of the configuration of the control system. The master control unit 100 may be integrated into either the operation support control unit 200 or the automatic operation control unit 300.

[Master control unit]
The master control unit 100 switches the degree of driving support and controls the HMI 400 related thereto. The master control unit 100 includes, for example, a switching control unit 110, an HMI control unit 120, an operator state determination unit 130, and an occupant condition monitoring unit 140. The switching control unit 110, the HMI control unit 120, the operator state determination unit 130, and the occupant condition monitoring unit 140 are each realized by executing a program by a processor such as a CPU (Central Processing Unit). Further, some or all of these functional parts may be realized by hardware such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or software. And may be realized by the cooperation of hardware.

The switching control unit 110 switches the driving support state based on, for example, an operation signal input from a predetermined switch (for example, a main switch and an auto switch described later) included in the HMI 400. Further, the switching control unit 110 may cancel the driving support and switch to manual driving based on, for example, an operation of instructing the driving controller 80 of the access pedal, the brake pedal, the steering wheel, or the like to accelerate, decelerate, or steer. Good. The details of the function of the switching control unit 110 will be described later.

The switching control unit 110 may switch the degree of driving support based on the action plan generated by the action plan generation unit 323. For example, the switching control unit 110 may end the driving support at the scheduled end point of the automatic driving.

The HMI control unit 120 causes the HMI 400 to output a notification or the like related to switching the degree of driving support. Further, the HMI control unit 120 may output the information regarding the determination result by one or both of the operator state determination unit 130 and the occupant condition monitoring unit 170 to the HMI 400. Further, the HMI control unit 120 may output the information received by the HMI 400 to the automatic operation control unit 300. Details of the function of the HMI control unit 120 will be described later.

The operator state determination unit 130 is, for example, in a state in which the steering wheel included in the driving operator 80 is being operated (specifically, when an intentional operation is actually performed, a state in which the steering wheel can be immediately operated or is gripped. It is determined whether or not it is a state). The details of the function of the operator state determination unit 130 will be described later.

The occupant condition monitoring unit 140 monitors the condition of the occupant seated at least in the driver's seat of the own vehicle M based on the captured image of the first imaging unit 464 included in the second display unit 460 described later. The occupant condition monitoring unit 140 determines, for example, whether or not the occupant is monitoring the surroundings of the own vehicle M. The details of the function of the occupant condition monitoring unit 140 will be described later.

[Driving support control unit]
The driving support control unit 200 executes the first degree of driving support. The driving support control unit 200 executes, for example, ACC, LKAS, or other driving support control. For example, when the driving support control unit 200 executes the ACC, the driving support control unit 200 moves forward with the own vehicle M based on the information input from the camera 10, the radar device 12, and the finder 14 via the object recognition device 16. The traveling driving force output device 500 and the braking device 510 are controlled so as to travel while keeping the distance between the vehicle and the vehicle constant. That is, the driving support control unit 200 performs acceleration / deceleration control (speed control) based on the inter-vehicle distance from the vehicle in front. Further, when executing the LKAS, the driving support control unit 200 controls the steering device 520 so that the own vehicle M travels while maintaining (lane keeping) the traveling lane in which the vehicle is currently traveling. That is, the driving support control unit 200 performs steering control for maintaining the lane. The type of driving support of the first degree may include various controls other than automatic driving (second degree and third degree) that do not require an operation on the driving operator 80.

[Automatic operation control unit]
The automatic operation control unit 300 executes the operation support of the second degree and the third degree. The automatic operation control unit 300 includes, for example, a first control unit 320 and a second control unit 340. The first control unit 320 and the second control unit 340 are each realized by executing a program by a processor such as a CPU. Further, a part or all of each of these functional units may be realized by hardware such as LSI, ASIC, FPGA, or may be realized by collaboration between software and hardware.

The first control unit 320 includes, for example, an outside world recognition unit 321, a vehicle position recognition unit 322, and an action plan generation unit 323.

The outside world recognition unit 321 recognizes states such as the position, speed, and acceleration of surrounding vehicles based on the information input from the camera 10, the radar device 12, and the finder 14 via the object recognition device 16. The position of the peripheral vehicle may be represented by a representative point such as the center of gravity or a corner of the peripheral vehicle, or may be represented by an area represented by the outline of the peripheral vehicle. The "state" of the peripheral vehicle may include the acceleration or jerk of the peripheral vehicle, or the "behavioral state" (eg, whether or not the vehicle is changing lanes or is about to change lanes).

Further, the outside world recognition unit 321 may recognize at least one of the above-mentioned peripheral vehicles, obstacles (for example, guardrails, utility poles, parked vehicles, people such as pedestrians), road shapes, and other objects. ..

The own vehicle position recognition unit 322 recognizes, for example, the lane (traveling lane) in which the own vehicle M is traveling, and the relative position and posture of the own vehicle M with respect to the traveling lane. The own vehicle position recognition unit 322 is, for example, around the own vehicle M recognized from the pattern of the road marking line (for example, the arrangement of the solid line and the broken line) obtained from the second map information 62 and the image captured by the camera 10. The driving lane is recognized by comparing with the pattern of the road marking line. In this recognition, the position of the own vehicle M acquired from the navigation device 50 and the processing result by the INS may be added.

Then, the own vehicle position recognition unit 322 recognizes, for example, the position and posture of the own vehicle M with respect to the traveling lane. FIG. 2 is a diagram showing how the own vehicle position recognition unit 322 recognizes the relative position and posture of the own vehicle M with respect to the traveling lane L1. The own vehicle position recognition unit 322 is formed on, for example, a line connecting the deviation OS of the reference point (for example, the center of gravity) of the own vehicle M from the central CL of the traveling lane and the central CL of the traveling lane in the traveling direction of the own vehicle M. The angle θ is recognized as the relative position and orientation of the own vehicle M with respect to the traveling lane L1. Instead of this, the own vehicle position recognition unit 322 recognizes the position of the reference point of the own vehicle M with respect to any side end of the traveling lane L1 as the relative position of the own vehicle M with respect to the traveling lane. May be good. The relative position of the own vehicle M recognized by the own vehicle position recognition unit 322 is provided to the recommended lane determination unit 61 and the action plan generation unit 323.

The action plan generation unit 323 generates an action plan for the own vehicle M to automatically drive to the destination or the like. For example, the action plan generation unit 323 sequentially executes events in the automatic driving control so as to drive in the recommended lane determined by the recommended lane determination unit 61 and to respond to the surrounding conditions of the own vehicle M. decide. The event in the automatic driving of the embodiment includes, for example, a constant-speed driving event in which the vehicle travels in the same lane at a constant speed, and low-speed tracking that follows the vehicle in front under the condition of a low speed (for example, 60 [km / h] or less). Event, lane change event to change the driving lane of own vehicle M, overtaking event to overtake the preceding vehicle, merging event to join vehicles at the merging point, branching event to drive own vehicle M in the desired direction at the branch point of the road , There is an emergency stop event that makes the own vehicle M stop in an emergency. In addition, during the execution of these events, actions for avoidance may be planned based on the surrounding conditions of the own vehicle M (existence of surrounding vehicles and pedestrians, lane narrowing due to road construction, etc.).

The action plan generation unit 123 generates a target track on which the own vehicle M will travel in the future. The target track is expressed as a sequence of points (track points) to be reached by the own vehicle M. The track point is a point to be reached by the own vehicle M for each predetermined mileage, and separately, the target speed and target acceleration for each predetermined sampling time (for example, about 0 commas [sec]) are set as the target track. Is generated as part of. Further, the track point may be a position to be reached by the own vehicle M at the sampling time for each predetermined sampling time. In this case, the information of the target velocity and the target acceleration is expressed by the interval of the orbital points.

FIG. 3 is a diagram showing how a target trajectory is generated based on the recommended lane. As shown, the recommended lanes are set to be convenient for driving along the route to the destination. When the action plan generation unit 323 approaches a predetermined distance before the recommended lane switching point (may be determined according to the type of event), the action plan generation unit 323 activates a lane change event, a branch event, a merging event, and the like. If it becomes necessary to avoid an obstacle during the execution of each event, an avoidance trajectory is generated as shown in the figure.

Further, when the action plan generation unit 323 activates the lane change event, the action plan generation unit 323 generates a target trajectory for lane change. 4 and 5 are diagrams for explaining a process when changing lanes. First, the action plan generation unit 323 is an adjacent lane adjacent to the lane (own lane) L1 in which the own vehicle M is traveling, and is two peripheral vehicles from the peripheral vehicle traveling in the adjacent lane L2 to which the lane is changed. Select to set the lane change target position TAs between these surrounding vehicles. Hereinafter, a peripheral vehicle traveling immediately before the lane change target position TAs in the adjacent lane will be referred to as a front reference vehicle mB, and a peripheral vehicle traveling immediately after the lane change target position TAs in the adjacent lane will be referred to as a rear reference vehicle mC. .. The lane change target position TAs is a relative position based on the positional relationship between the own vehicle M, the front reference vehicle mB, and the rear reference vehicle mC.

In the example of FIG. 4, the action plan generation unit 323 sets the lane change target position TAs. In the figure, mA represents a vehicle in front, mB represents a front reference vehicle, and mC represents a rear reference vehicle. Further, the arrow d indicates the traveling (running) direction of the own vehicle M. In the case of the example of FIG. 4, the action plan generation unit 323 sets the lane change target position TAs between the front reference vehicle mB and the rear reference vehicle mC on the adjacent lane L2.

Next, the action plan generation unit 323 satisfies the primary condition for determining whether or not the lane change target position TAs (that is, between the front reference vehicle mB and the rear reference vehicle mC) can be changed. Is determined.

The primary condition is that, for example, there are no peripheral vehicles in the prohibited area RA provided in the adjacent lane, and the TTCs of the own vehicle M and the front reference vehicle mB and the rear reference vehicle mC are each larger than the threshold value. That is. This determination condition is an example when the lane change target position TAs is set on the side of the own vehicle M. If the primary condition is not satisfied, the action plan generation unit 323 resets the lane change target position TAs. At this time, the speed control for waiting until the timing at which the lane change target position TAs satisfying the primary condition can be set, or changing the lane change target position TAs and moving to the side of the lane change target position TAs is performed. It may be done.

As shown in FIG. 4, the action plan generation unit 323 projects, for example, the own vehicle M onto the lane L2 of the lane change destination, and sets a prohibited area RA having a slight margin in the front-rear direction. The prohibited area RA is set as an area extending from one end to the other end in the lateral direction of the lane L2.

When there are no peripheral vehicles in the prohibited area RA, the action plan generation unit 323, for example, extends the front end and the rear end of the own vehicle M to the lane L2 side of the lane change destination, and the extension line FM. The extension line RM is assumed. The action plan generation unit 323 calculates the collision margin time TTC (B) between the extension line FM and the front reference vehicle mB, and the collision margin time TTC (C) between the extension line RM and the rear reference vehicle mC. The collision margin time TTC (B) is a time derived by dividing the distance between the extension line FM and the front reference vehicle mB by the relative speeds of the own vehicle M and the front reference vehicle mB. The collision margin time TTC (C) is a time derived by dividing the distance between the extension line RM and the rear reference vehicle mC by the relative speeds of the own vehicle M and the rear reference vehicle mC. The trajectory generation unit 118 determines that the primary condition is satisfied when the collision margin time TTC (B) is larger than the threshold Th (B) and the collision margin time TTC (C) is larger than the threshold Th (C). .. The threshold values Th (B) and Th (C) may be the same value or may be different values.

When the primary condition is satisfied, the action plan generation unit 323 generates a track candidate for changing lanes. In the example of FIG. 5, the action plan generation unit 323 assumes that the front vehicle mA, the front reference vehicle mB, and the rear reference vehicle mC travel at a predetermined speed model, and the speed models of these three vehicles and themselves. Based on the speed of the vehicle M, a track candidate is generated so that the own vehicle M does not interfere with the preceding vehicle mA and is located between the front reference vehicle mB and the rear reference vehicle mC at a certain time in the future. To do. For example, the action plan generation unit 323 has a spline curve or the like from the current position of the own vehicle M to the position of the forward reference vehicle mB at a certain time in the future, the center of the lane to which the lane is changed, and the end point of the lane change. It is smoothly connected using the polynomial curve of, and a predetermined number of orbital points K are arranged on this curve at equal or unequal intervals. At this time, the action plan generation unit 323 generates a track so that at least one of the track points K is arranged in the lane change target position TAs.

In various situations, the action plan generation unit 323 generates a plurality of target trajectory candidates, and at that time, selects the optimum target trajectory that matches the route to the destination.

The second control unit 340 includes, for example, a travel control unit 342. The travel control unit 342 controls the travel driving force output device 500, the brake device 510, and the steering device 520 so that the own vehicle M passes the target trajectory generated by the action plan generation unit 323 at the scheduled time. To do.

The HMI 400 presents various information to the occupants in the vehicle and accepts input operations by the occupants. The HMI 400 includes, for example, a part or all of various display devices, a light emitting unit, a speaker, a buzzer, a touch panel, various operation switches, keys, and the like. Further, the HMI 400 may include a part of a seatbelt device that holds an occupant in a seated state by a seatbelt. Details of the functions of the HMI 400 will be described later.

The traveling driving force output device 500 outputs a traveling driving force (torque) for traveling the vehicle to the drive wheels. The traveling driving force output device 500 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, and the like, and an ECU (Electronic Control Unit) that controls them. The ECU controls the above configuration according to the information input from the travel control unit 342 or the information input from the operation controller 80.

The brake device 510 includes, for example, a brake caliper, a cylinder that transmits flood pressure to the brake caliper, an electric motor that generates flood pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to the information input from the travel control unit 342 or the information input from the operation controller 80 so that the brake torque corresponding to the braking operation is output to each wheel. The brake device 510 may include a mechanism for transmitting the hydraulic pressure generated by the operation of the brake pedal included in the operation operator 80 to the cylinder via the master cylinder as a backup. The brake device 510 is not limited to the configuration described above, and controls the actuator according to the information input from the travel control unit 342 or the information input from the operation operator 80 to transmit the oil pressure of the master cylinder to the cylinder. It may be an electronically controlled hydraulic braking device. Further, the brake device 510 may include a plurality of systems of brake devices.

The steering device 520 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies a force to the rack and pinion mechanism to change the direction of the steering wheel. The steering ECU drives the electric motor according to the information input from the traveling control unit 342 or the information input from the driving controller 80, and changes the direction of the steering wheel.

During manual operation, the input information from the driving operator 80 is directly output to the traveling driving force output device 500, the braking device 510, and the steering device 520. Further, the input information from the driving operator 80 may be output to the traveling driving force output device 500, the braking device 510, and the steering device 520 via the automatic driving control unit 300. Each ECU of the traveling driving force output device 500, the brake device 510, and the steering device 520 performs its own operation based on the input information from the driving operator 80 and the like.

[Structure of HMI400]
Hereinafter, a configuration example of the HMI 400 of the embodiment will be described. FIG. 6 is a diagram showing an example of the HMI 400 in the own vehicle M. The HMI 400 includes, for example, a first operation unit 410, a second operation unit 420, a light emitting unit 430R, 430L, a third operation unit 440, a first display unit 450, and a second display unit 460. And a HUD (Head Up Display) 470.

The first operation unit 410, the second operation unit 420, and the light emitting units 430R and 430L are provided on the steering wheel 82, which is one of the operation controls 80. Further, the steering wheel 82 is provided with a grip sensor 82A. The grip sensor 82A is, for example, a capacitance sensor provided along the circumferential direction of the steering wheel 82. The grip sensor 82A detects that an object is close to or touches the area to be detected as a change in capacitance. When the degree of change in capacitance is equal to or greater than the threshold value, the grip sensor 82A outputs a predetermined detection signal to the operator state determination unit 130 of the master control unit 100. This threshold is set to a value lower than the capacitance that occurs when the occupant is holding the steering wheel 82, for example. Further, the gripping sensor 82A may output a detection signal indicating the capacitance to the operator state determination unit 130 regardless of whether or not it is equal to or higher than the threshold value.

The first operation unit 410 includes, for example, a main switch 412 and an auto switch 414. The main switch 412 is a switch for making the driving support startable. In other words, a switch for starting processing (internal processing) in the preparatory stage before executing driving support, or a switch for making it possible to determine whether or not driving support can be started, which will be described later. Is.

When the main switch 412 is operated, the own vehicle M does not immediately start the execution of the automatic driving, but performs the processing in the preparatory stage before executing the automatic driving. The processing in the preparatory stage is, for example, the start of the processing of object recognition (specifically, the start of operation of the Kalman filter, etc.). When the auto switch 414 is operated while the main switch 412 is operated and the own vehicle M is in a state where the automatic operation can be started (that is, after a certain amount of time has passed since the operation), the operation is performed. Control for assistance begins. That is, the auto switch 414 is a switch for actually starting the automatic operation in a state where the automatic operation can be started.

The second operation unit 420 includes an operation switch 422 for starting to provide the telephone function with an image (television telephone function). The light emitting portions 430R and 430L are arranged, for example, in the spoke portions extending from the central boss portion of the steering wheel 82 toward the annular rim portion. The lighting state of the light emitting unit 330R is controlled by the control of the HMI control unit 120.

The third operation unit 440 includes, for example, a rotation operation unit 442 protruding toward the front side when viewed from the occupant, and a switch operation unit 444. The rotation operation unit 442 is formed in a substantially cylindrical shape and can be rotated around an axis. The switch operation unit 444 is provided around the rotation operation unit 442 and on the top surface of the rotation operation unit 442. The third operation unit 440 includes a rotation sensor (not shown) such as an encoder that detects the rotation angle and rotation speed of the rotation operation unit 442, and a displacement sensor (not shown) that detects the displacement of the switch operation unit 444. The detected value output from the sensor is output to the master control unit 100. The detected value output to the master control unit 100 is used for operations such as arrows, selection buttons, and confirmation buttons output on the screen of the second display unit 463, which will be described later, and for selecting and confirming input characters.

Further, the third operation unit 440 may be a so-called touch panel type operation unit in which a selection or confirmation operation is performed by touching the display screen with a fingertip. Further, the third operation unit 440 is provided with a light emitting unit 446 capable of emitting light in a predetermined color.

The first display unit 450 is, for example, a display device provided near the front of the driver's seat on the instrument panel and visible to the occupant from the gap of the steering wheel 82 or through the steering wheel 82. The first display unit 450 is, for example, an LCD (Liquid Crystal Display), an organic EL (Electro Luminescence) display device, or the like. The first display unit 450 displays information necessary for traveling of the own vehicle M during manual driving or automatic driving, or information regarding instructions to the occupants. The information necessary for the running of the own vehicle M during manual operation is, for example, the speed of the own vehicle M, the engine speed, the remaining fuel amount, the radiator water temperature, the mileage, and other information. On the other hand, the information necessary for the running of the own vehicle M during automatic driving is, for example, information such as the future track of the own vehicle M, the degree of driving support, and instructions to the occupants.

The second display unit 460 is provided in the center of the instrument panel. FIG. 7 is a diagram showing an example of the appearance of the second display unit 460. Hereinafter, it will be described using XYZ coordinates as needed. For example, the X direction is the central axis direction of the own vehicle M, the Y direction is the width direction of the own vehicle M, and the Z direction is the vertical direction of the own vehicle M.

The second display unit 460 includes, for example, a panel 461, a light emitting unit 462, a second display unit 463, a visor unit 463A, a left side wall portion 463BL, a right side wall portion 463BR, a bottom portion 463C, and a first display unit 460. The image pickup unit 464, the second image pickup section 465, and the microphone 466 are provided. The second display unit 460 is an example of the "imaging display unit" in the claims. The second display unit 460 has, for example, a space partitioned by four wall portions (visor portion 463A, left wall portion 463BL, right wall portion 463BR, bottom portion 463C).

The panel 461 is made of, for example, transparent glass or resin. A second display unit 463 is provided near the center of the panel 461 and on the back side. The second display unit 463 is, for example, an LCD, an organic EL display device, or the like. On the second display unit 463, for example, route information and map information output by the navigation device 50 are displayed, and an image of a call partner is displayed when a videophone call is made. In addition, the second display unit 463 may display contents such as a TV program, a video played on a DVD, and a video of a downloaded movie. Videophone is an example of "service linked with images".

Hereinafter, in the panel 461, the area on the left side (minus Y direction side) of the areas that deviate from the display area of the second display unit 463 to the left and right is referred to as a left side area 461L, and the area on the right side (plus Y direction side) is referred to as a right side area. It is called 461R. The left side region 461L is a region in which the interval in the vertical direction (plus or minus Z direction) becomes narrower toward the left side. Further, the right side region 461R is a region in which the vertical interval becomes narrower toward the right side.

The visor portion 463A, the left side wall portion 463BL, the right side wall portion 463BR, and the bottom portion 463C are each formed so as to project from the vicinity of the outer periphery of the panel 461 toward the occupant side. The visor portion 463A is formed so as to cover the panel 461 from above. The visor unit 463A tilts downward when it deviates from the display area of the second display unit 463 in the left-right direction.

The left side wall portion 463BL and the right side wall portion 463BR are formed in a shape that fits the portion of the left side region 461L and the right side region 461R that extends from the bottom of the panel 461, respectively.

The first imaging unit 464 is provided on the back side of the left side region 461L of the panel 461, and images an occupant seated in the driver's seat through the panel 461. The degree of recognition of the first imaging unit 464 and the second imaging unit 465 by the occupant is different. The first imaging unit 464 is provided in a form in which the degree of recognition by the occupant of the own vehicle M is smaller than that in the second imaging unit 465 (a form in which the occupant is less likely to recognize it).

FIG. 8 is a cross-sectional view of the second display unit 460 of FIG. 7 in the 8-8 direction. FIG. 8 partially includes something other than a cross section. A film 461F is provided on the traveling direction side of the left side region 461L. Hereinafter, the traveling direction side of the left side region 461L is referred to as an inner region AR. The film 461F is a film that adjusts the transmission of light so that the occupant cannot easily recognize the state of the inner region AR, and makes it possible to recognize the state of the occupant from the inner region AR. In addition, instead of the film, in the panel 461, a reflective film may be formed in the left side region 461L, or a process for adjusting the light transmission may be performed. As described above, since the first imaging unit 464 is provided on the traveling direction side of the left side region 461L, it is difficult for the occupant to recognize the first imaging unit 464, but the first imaging unit 464 is the state of the occupant. Can be imaged.

The first imaging unit 464 mainly images the face of an occupant (particularly, an occupant seated in the driver's seat) seated in a seat installed in the vehicle interior. The first image pickup unit 464 is a digital camera using a solid-state image pickup element such as a CCD or CMOS. The first imaging unit 464, for example, periodically images the occupant. The captured image captured by the first imaging unit 464 is output to the master control unit 100. The image captured by the first imaging unit 464 is an image used for monitoring the state of the occupant.

The second imaging unit 465 is provided on the visor unit 463A, for example, on the upper side of the left side region 461L. The second imaging unit 465 is provided at a position where the occupant can see the second display unit 463. The second imaging unit 465 is provided, for example, in a state in which the lens 465l of the second imaging unit 465 can be recognized by the occupant. The second image pickup unit 465 takes an image centering on the face of the occupant. The image captured by the second imaging unit 465 is an image transmitted to a device located at a position distant from the own vehicle M. Specifically, the image captured by the second imaging unit 465 is an image used for a videophone or the like, and is an image displayed on a display unit used by a other party of the videophone.

The microphone 466 is above the right side region 461R and is provided in the visor portion 463A. The information indicating the sound acquired by the microphone 466 is an image that is associated with the image captured by the second imaging unit 465 and transmitted to a device located at a position distant from the own vehicle M. Specifically, the information indicating the voice acquired by the microphone 466 is information used for a videophone or the like, and is information output to the voice output unit of the other party of the videophone.

The light emitting unit 462 is above the second display unit 463 and is provided on the visor unit 463A. Further, the light emission of the light emitting unit 462 is visible to the occupant without being blocked by the visor unit 463A. By adopting this form, the visor unit 463A shields external light such as sunlight entering the light emitting unit 462, so that the visibility of the light emitted by the occupant can be improved.

The light emitting unit 462 is controlled by the HMI control unit 120 so as to emit light when the second display unit 463 is available. Usable means that, for example, a screen related to the videophone function is displayed on the second display unit 463 by operating the second operation unit 420, or a movie or a television program is displayed by operating the third operation unit 440. The image related to the above is displayed on the second display unit 463, so that the occupant can use the image.

As described above, since the first imaging unit 464 is provided so as not to be easily recognized by the occupant, the occupant does not excessively be aware of the first imaging unit 464. Therefore, the first imaging unit 464 can image the natural state of the occupant. As a result, the occupant condition monitoring unit 140 can monitor the occupant's condition more accurately. Further, since the second imaging unit 465 is provided so that the occupant can easily recognize it, when using a videophone, the direction in which the line of sight is directed can be easily recognized. Therefore, the vehicle system 1 can promote communication with the other party in the videophone.

As described above, the second display unit 460 is the first image pickup unit 464 that images the occupant of the own vehicle M, and the first image pickup unit 464 that captures the image used for the condition monitoring of the occupant and the self. The second display unit 465 that captures the occupant of the vehicle M and the second image pickup unit 465 that captures the image used for the service linked with the image displayed by the second display unit 463 is the second display. By arranging in the unit 460, it is possible to provide an image pickup display unit in which image pickup units for capturing images for different purposes are provided in an appropriate arrangement.

Further, as described above, since the left side region 461L is formed so that the distance in the vertical direction becomes shorter toward the left side, the region where the first imaging unit 464 is provided is secured and the left side region is secured. The size of 461L can be suppressed. Further, the right side region 461R is a target region in which the left side region 461L is inverted, and such a design improves the design.

The HUD 470 is arranged at a position higher than, for example, the first display unit 450. The HUD470 projects an image onto a predetermined imaging unit. For example, the HUD 470 projects an image onto a part of the front windshield in front of the driver's seat so that the eyes of the occupant seated in the driver's seat can see the virtual image. The display area of the image projected by the HUD 470 is smaller than the display area of the image in the first display unit 450. This is to prevent the occupant from overlooking the real object ahead of the image projected by the HUD470. In the embodiment, the front windshield of the own vehicle M may be used as the second display unit instead of the HUD470. In this case, for example, the LED (Light Emitting Diode) incorporated in the instrument panel may be made to emit light, and the light emission of the LED may be reflected by the front windshield.

[Display control of HMI400 related to automatic driving]
Next, the display control of the HMI 400 related to the automatic operation will be described. The layout of the display screen shown below is just an example and can be changed arbitrarily. Layout refers to placement, color, scale, and more.

FIG. 9 is a diagram showing various scenes from the manual driving to the automatic driving of the own vehicle M to the execution of the lane change by the automatic driving. In the example of FIG. 9, the scene (1) is a scene in which the own vehicle M enters the expressway from the general road by manual driving. The scene (2) is a scene in which the own vehicle M switches from manual driving to automatic driving. The scene (3) is a scene in which the own vehicle M executes a lane change by automatic driving. The display control corresponding to each of the scenes (1) to (3) will be described below.

<Scene (1)>
The scene (1) is, for example, a scene before entering the highway. In this scene, since the main switch 412 and the auto switch 414 of the first operation unit 410 are not operated, the driving support is not executed and the manual operation is performed. When manual driving is performed, the HMI control unit 120 causes the first display unit 450 to display information necessary for the occupant in the driver's seat to manually drive the own vehicle M using the driving operator 80. Further, the HMI control unit 120 causes the HUD 470 to display a part of the information displayed on the first display unit 450. The screen in this case is shown in FIG.

FIG. 10 is a diagram showing an example of the first screen IM1-1 and the second screen IM2-1 displayed during manual operation. The first screen IM1-1 is a screen displayed by the first display unit 450, and the second screen IM2-1 is a screen that is projected by the HUD 470 and is reflected in the eyes of the occupant. The HMI control unit 120 displays, for example, the remaining battery level, the rotation speed, the shift position, the indoor temperature, and the mileage of the own vehicle M on the first screen IM1-1 as information necessary for the running of the own vehicle M during manual operation. , Running speed, remaining fuel, etc. are displayed. Further, the HMI control unit 120 causes the second screen IM2-1 to display the speed information of the images displayed on the first screen IM1-1 smaller than that of the first screen IM1-1. As described above, the recognition area of the image projected by the HUD 470 to the eyes of the occupant is smaller than the display area of the image by the first display unit 450. Therefore, the HMI control unit 120 causes the first display unit 450 to display relatively detailed first information (detailed information) regarding the driving support of the own vehicle M, and the HUD470 is simpler than the detailed information. It is information, and the second information (simple information) about the driving support is displayed. The simple information is, for example, information whose amount of information is smaller than that of detailed information. Further, the simple information may be information in which the type and number of items to be displayed are smaller than the type and number of items to be displayed as detailed information. Further, the simple information may be an image whose resolution is lowered, simplified or deformed with respect to the image displayed as detailed information. Further, the second information may be highly important information or highly urgent information among the first information.

<Scene (2)>
In scene (2), the own vehicle M is entering the highway. Upon receiving that the main switch 412 has been operated by the occupant, the HMI control unit 120 changes the screens displayed on the first display unit 450 and the HUD 470. The screen after the change is shown in FIG.

FIG. 11 is a diagram showing an example of the third screen IM3-1 and the fourth screen IM4-1 displayed when the main switch 412 is operated. The third screen IM3-1 is a screen displayed by the first display unit 450, and the fourth screen IM4-1 is a screen that is projected by the HUD 470 and is reflected in the eyes of the occupant. The same applies to the third screen IM3-X (X is an arbitrary natural number) and the fourth screen IM4-X shown in the drawings below. The third screen IM3-X and the fourth screen IM4-X are continuously displayed in a state in which driving support can be executed and a state in which driving support is being executed.

The third screen IM3-1 includes a peripheral detection information display area 600-1, a driving support status display area 620-1, and a driving support start operation guide area 640-1 as areas for displaying information related to driving support. ing. Hereinafter, each area on the third screen IM3-X will be referred to as a peripheral detection information display area 600-X, a driving support status display area 620-X, and a driving support start operation guide area 640-X.

The HMI control unit 120 has an image showing the road shape in front of the own vehicle M acquired from the second map information 62 in the peripheral detection information display area 600-1, and the own vehicle M recognized by the own vehicle position recognition unit 322. And an image showing the peripheral vehicle m recognized by the outside world recognition unit 321 are displayed.

Further, the HMI control unit 120 causes the driving support status display area 620-1 to display all the information indicating the candidates of the driving support (including automatic driving) status that the own vehicle M can execute. In the example of FIG. 11, image 621 showing three indicators of "Assist", "Hands Off", and "Eyes Off" is shown as information indicating the candidate of the driving support state. For example, the degree of driving assistance is expressed by each indicator alone or by a combination of a plurality of indicators.

The indicator "Assist" indicates that the own vehicle M is executing the driving support at the first degree such as ACC or LKAS, or is in a state where it can transition to the driving support at the first degree. Whether the driving support is executed in the first degree or the state in which the transition to the driving support in the first degree is possible can be grasped by the request operation notification image 622 described later.

The indicator "Hands Off" is a state in which the own vehicle M is executing driving support to the second degree in which the occupant does not have to operate the driving controller 80, but the occupant is obliged to monitor the surroundings. Alternatively, it indicates that it is in a state where it is possible to transition to driving support in the second degree. Whether the driving support is executed in the second degree or the state in which the transition to the driving support in the second degree is possible can be grasped by the requested operation notification image 622.

The indicator "Eyes Off" indicates that the vehicle M is executing driving support to a third degree in which the occupant does not have to operate the driving controller 80 and the occupant is not obliged to monitor the surroundings. , Indicates a state in which it is possible to transition to driving support in the third degree. Whether the driving support is executed in the third degree or the state in which the transition to the driving support in the third degree is possible can be grasped by the requested operation notification image 622. In the example of FIG. 11, the driving support of the own vehicle M is not executed (manual driving state).

Further, the HMI control unit 120 has a display position corresponding to an image 621 showing three indicators of "Assist", "Hands Off", and "Eyes Off" in the driving support state display area 620-1, and is a request operation notification image. Display 622. "Corresponding" means that the correspondence can be recognized by a person, such as a guideline indicating horizontal arrangement, vertical arrangement, and association. As an example, the "display position corresponding to the image 621" is a display position adjacent to the image 621, and is a few [cm] or less (for example, 3 [cm]) in at least one of the top, bottom, left, and right with respect to the display position of the image 621. The display position of) is shown below. The requested operation notification image 622 is, for example, an image showing a predetermined operation performed by the occupant on the driving operator 80. The requested operation notification image 622 includes, for example, an image showing the driving operator 80 and an image showing a predetermined portion of the occupant. The requested operation notification image 622 is, for example, an image schematically showing the positional relationship between the steering wheel 82 and the occupant's hand.

The HMI control unit 120 causes the fourth screen IM4-1 of the HUD470 to display simple information extracted from a part of the detailed information with respect to the detailed information displayed on the third screen IM3-1. On the fourth screen IM4-1 of the HUD470, among the information on the driving support displayed on the screen IM3-1 of the first display unit 450, the information on the road shape in front of the own vehicle M and the speed of the own vehicle M are displayed. The information shown is displayed.

When it is detected that the auto switch 414 is operated by the occupant in the state shown in FIG. 11, the master control unit 100 causes the driving support control unit 200 to execute the first degree of driving support. Further, the HMI control unit 120 changes the screen displayed on the first display unit 450 and the HUD 470 to, for example, the screen shown in FIG.

FIG. 12 is a diagram showing an example of screens IM3-2 and IM4-2 displayed on the first display unit 450 and HUD470 when the auto switch 414 is operated. The HMI control unit 120 displays an image showing the degree of driving assistance being executed so as to be distinguishable from other images showing the degree of driving assistance. For example, the HMI control unit 120 highlights and displays an image of the driving support mode "Assist" of the own vehicle M in the driving support status display area 620-2 of the third screen IM3-2. As a result, the occupant can grasp that the first degree of driving support is being provided.

Here, the HMI control unit 120 requests, as the requested operation notification image 622, a moving image that requests the occupant to perform the operation necessary for shifting to the degree of driving support (automatic driving) corresponding to "Hands Off". It is displayed as an image 622. The moving image is, for example, an image including a dynamic object in which a predetermined object dynamically moves with the passage of time. In addition, the moving image may include an animation.

The same information as that of the fourth screen IM4-1 is displayed on the fourth screen IM4-2 of the HUD470.

Further, the switching control unit 110 satisfies the conditions for shifting to the second degree of driving support in the state where the auto switch 414 is operated, and the steering wheel 82 is not gripped by the occupant. When it is determined by the operator state determination unit 130 that there is, the automatic operation control unit 300 is made to execute the second degree of operation support (that is, automatic operation).

Further, the HMI control unit 120 changes the screen displayed on the first display unit 450 and the HUD 470 to, for example, the screen shown in FIG. 13 when the automatic operation control unit 300 executes the second degree of driving support. ..

FIG. 13 is a diagram showing an example of a screen displayed on the first display unit 450 and the HUD 470 during the second degree of driving support. The HMI control unit 120 highlights and displays an image of “Hands Off” corresponding to the second degree of driving support in the driving support status display area 620-2. This allows the occupant to know that the second degree of driving assistance is being provided.

Further, the HMI control unit 120 is recognized in the peripheral detection information display area 600-3 by, for example, an image showing the road shape in front of the own vehicle M acquired from the second map information 62 and the own vehicle position recognition unit 322. The image showing the own vehicle M, the image showing the peripheral vehicle m acquired by the outside world recognition unit 321 and the future track image 602 showing the future track of the own vehicle M generated by the action plan generation unit 323 are displayed. .. Further, the HMI control unit 120 starts driving support (automatic driving in the figure) in the second degree in the peripheral detection information display area 600-3, but causes the occupant to continuously monitor the traffic conditions in the surrounding area. Display information.

The occupant condition monitoring unit 140 of the master control unit 100 monitors that the occupant continuously monitors the traffic conditions in the surrounding area. For example, the occupant condition monitoring unit 140 acquires a face image of an occupant sitting in the driver's seat from the captured image of the first imaging unit 464, and acquires the line-of-sight direction from the acquired face image. For example, the occupant condition monitoring unit 140 may acquire the line-of-sight direction of the occupant from the captured image of the first imaging unit 464 by deep learning using a neural network or the like. For example, a neural network trained to output the line-of-sight direction is constructed in advance by inputting feature information such as eyes, nose, and mouth obtained by analyzing an unspecified number of face images and the position of the black eye of the eyeball. I will do it. Then, the occupant condition monitoring unit 140 acquires the line-of-sight direction of the occupant by inputting the face image of the occupant of the own vehicle M into this neural network.

Further, the occupant condition monitoring unit 140 determines whether or not the occupant is monitoring the periphery of the own vehicle M based on whether or not the occupant's line-of-sight direction is included in the range of the direction in which the occupant's line of sight can be monitored. To do. The occupant condition monitoring unit 140 does not monitor the surroundings when the line-of-sight direction of the occupant is not included in the range of the direction in which the occupant's line of sight can be monitored, or when the line-of-sight direction of the occupant cannot be acquired. Is determined. When it is determined that the peripheral monitoring is not performed, the HMI control unit 120 may give a warning by voice or the like in order to monitor the surroundings of the occupants.

Further, the occupant condition monitoring unit 140 determines that the occupant is performing peripheral monitoring when the line-of-sight direction of the occupant is included in the range of the direction in which the occupant's line of sight can be monitored. In this case, the automatic operation control unit 300 continues the operation support of the second degree. When the driving support of the own vehicle M is starting, nothing is displayed in the driving support start operation guide area 640-3.

The HMI control unit 120 causes the fourth screen IM4-3 of the HUD470 to display a future track image 602 showing the future track of the own vehicle M in addition to the same information as the fourth screen IM4-2.

<Scene (3)>
In the scene (3), the automatic driving control unit 300 changes the lane of the own vehicle M by the second degree of driving support. In this case, the HMI control unit 120 causes one or both of the first display unit 450 and the HUD 470 to display a screen corresponding to the driving support.

FIG. 14 is a diagram showing an example of the third screen IM3-4 and the fourth screen IM4-4 displayed at the first timing before the behavior of the own vehicle M changes. The HMI control unit 120, in addition to the contents displayed in the peripheral detection information display area 600-3 in the peripheral detection information display area 600-4 of the third screen IM3-4, for example, the direction in which the own vehicle M changes lanes. An image 606 indicating that the above is displayed is displayed. In the example of FIG. 14, the image 606 in which the own vehicle M changes lanes to the right lane adjacent to the traveling lane is displayed.

Next, scenes (4) to (6) will be described. FIG. 15 is a diagram showing various scenes from executing driving support at a third degree to the own vehicle M and then executing driving support at a third degree to a second degree. In the example of FIG. 15, the scene (4) is a scene in which the driving support of the own vehicle M is switched from the second degree to the third degree by following the surrounding vehicle m in the traffic jam. The scene (5) is a scene in which the own vehicle M is executing low-speed follow-up running, which is an example of driving support of the third degree. Low-speed follow-up travel (TJP; Traffic Jam Pilot) is a control mode for following a vehicle in front at a predetermined speed or lower. The predetermined speed is (for example, 60 [km / h] or less). The low-speed follow-up running is activated when the speed of M of the own vehicle is equal to or less than the predetermined speed and the distance between the vehicle and the preceding vehicle m is within the predetermined distance (confirmation). In low-speed follow-up driving, highly reliable automatic driving can be realized by continuing relatively easy control of following the vehicle in front on a congested road. The scene (6) is a scene in which the driving support of the own vehicle M is switched from the third degree to the second degree. The display control corresponding to each of the scenes (4) to (6) will be described below.

<Scene (4)>
In the scene (4), the automatic driving control unit 300 has not yet reached the low-speed follow-up running, and is performing acceleration control of the own vehicle M. In this case, the HMI control unit 120 displays a screen corresponding to the driving support on one or both of the first display unit 450 and the HUD 470.
<Scene (5)>
In scene (5), low-speed follow-up running is executed. In this case, the HMI control unit 120 displays a screen corresponding to the low-speed follow-up running on the first display unit 450 and the HUD 470.

FIG. 16 is a diagram showing an example of the third screen IM3-5 and the fourth screen IM4-5 displayed during low-speed follow-up running. The HMI control unit 120 causes the peripheral detection information display area 600-5 to display the peripheral detection image 610A indicating that the driving support of the third degree is being executed.

The peripheral detection image 610A is an image showing that the surroundings of the own vehicle M are being monitored by, for example, the camera 10, the radar device 12, the finder 14, the object recognition device 16, and the outside world recognition unit 321. The peripheral detection image 610A is, for example, an animation in which ripples spread from the center of the own vehicle M toward the outside.

In addition, the HMI control unit 120 has an indicator "Eyes Off" indicating that the own vehicle M does not impose an obligation to monitor the surroundings of the occupants in the driving support status display area 620-5 of the third screen IM3-5. The image of the indicator "Hands OFF" indicating that the operation of the operation operator 80 is not required is highlighted. Further, the HMI control unit 120 monitors the surroundings of the own vehicle M in the driving support state display area 620-5 by the camera 10, the radar device 12, the finder 14, the object recognition device 16, and the outside world recognition unit 321. An image 626 showing that the image is displayed is displayed.

Further, the HMI control unit 120 displays a peripheral detection image indicating that the fourth screen IM4-5 of the HUD470 is provided with the same information as that of the fourth screen IM4-4 and that the driving support of the third degree is executed. Display 610B. The peripheral detection image 610B is, for example, an animation in which ripples spread from the center of the own vehicle M toward the outside.

Further, in a state where the occupant is not obliged to monitor the surroundings, the HMI control unit 120 uses the light emitting unit 462 provided on the second display unit 463 when the second display unit 463 becomes available. It is controlled so that it emits light in a predetermined color. In the state where the occupant is not obliged to monitor the surroundings, the functions that can be used include, for example, a videophone and entertainment content (content stored in a movie, television, or DVD) as a second display unit 463. It is a function displayed in. When a videophone is used, an image of the other party is displayed on the second display unit 463.

For example, when the third operation unit 440 is operated while the second display unit 463 is available, the HMI control unit 120 causes the second display unit 463 to display a screen corresponding to the operation content. Further, the HMI control unit 120 is an image of a party to talk to the second display unit 463 when the operation switch 422 of the second operation unit 420 is operated while the second display unit 463 is available. Is displayed. As a result, the occupant can enjoy the call while looking at the other party displayed on the second display unit 463. That is, the occupant can use the videophone.

<Scene (6)>
In the scene (6), the automatic driving control unit 300 switches the own vehicle M from the third degree to the second degree of driving support because the preceding vehicle m following the low speed does not exist. In this case, the HMI control unit 120 provides information suggesting the occupant's monitoring target or operation target requested to the occupant based on the change in the degree of driving assistance, as shown in FIG. Display on one or both of the HUD470.

FIG. 17 is a diagram showing an example of the third screen IM3-6 and the fourth screen IM4-6 displayed for the occupant to perform peripheral monitoring. The HMI control unit 120 has information in the peripheral detection information display area 600-6 that low-speed follow-up driving (“traffic jam follow-up automatic driving” in the figure) is completed and information that the occupant is to confirm the traffic conditions in the surrounding area. Is displayed.

Further, the HMI control unit 120 causes the fourth screen IM4-6 to display a forward gaze request image 650 that requires the occupant to gaze at the front of the own vehicle M. The forward gaze request image 650 is an image including an elliptical region showing a predetermined region in front of the own vehicle M. Further, the forward gaze request image 650 may have a predetermined shape such as a circular shape or a rectangular shape, or may be information such as a mark or a symbol character that calls attention to the occupant. Further, the HMI control unit 120 lights or blinks the forward gaze request image 650 in a predetermined color. Further, the HMI control unit 120 may urge the occupant to gaze forward by causing the LED incorporated in the instrument panel to emit light and reflecting the light emission of the LED by the front windshield.

The occupant condition monitoring unit 140 determines whether or not the occupant is performing peripheral monitoring based on the captured image of the first imaging unit 464. When it is determined that the occupant is monitoring the surroundings, the switching control unit 110 causes the automatic driving control unit 300 to switch the driving support of the own vehicle M from the third degree to the second degree. The HMI control unit 120 causes one or both of the first display unit 450 and the HUD 470 to display a screen corresponding to the driving support according to the second degree.

<Timing of switching various devices or controls related to driving support>
Here, the switching timing of various devices or controls related to the driving support of the own vehicle M will be described with reference to the drawings. FIG. 18 is a diagram for explaining switching timing of various devices or controls related to driving support.

In FIG. 18, as switching related to driving support, (A) main switch 412 on / off, (B) auto switch 414 on / off, (C) manual operation mode display on / off, and (D) operation. Assist mode display on / off, (E) first degree driving support on / off, (F) steering wheel 82 gripped / not gripped, (G) second degree driving support On / off, (H) on / off of the third degree of driving support, and (I) switching timing for the passage of time when the driver's operation monitoring is required / unnecessary is shown.

At time T0, the own vehicle M is traveling by the manual operation of the occupant. In this case, the main switch 412 and the auto switch 414 are not operated, and the screens in the manual operation mode (first screen IM1, second screen IM2) are displayed on the first display unit 450 and HUD470. Further, at time T0, the driving support (first degree to third degree) for the own vehicle M is not implemented, and the occupant needs to grasp the steering wheel 82 and monitor the surroundings.

At time T1, an operation of turning on the main switch 412 is executed by the occupant. In this case, the screens in the driving support mode (third screen IM3, fourth screen IM4) are displayed on the first display unit 450 and the HUD470. In the states from time T1 to T2, the driving control by the driving support is not performed, and the manual driving is continued.

At time T2, the occupant is executing an operation of turning on the auto switch 414.
In this case, the master control unit 100 causes the driving support control unit 200 to execute the first degree of driving support. In the driving support mode display, the HMI control unit 120 displays an image showing that the second degree of driving support is executed when the occupant releases the steering wheel 82.

At time T3, the occupant has released his hand from the steering wheel 82 while the own vehicle M is capable of providing a second degree of driving support. In this case, the switching control unit 110 causes the automatic driving control unit 300 to execute the second degree of driving support from the first degree of driving support by the driving support control unit 200.

At time T4, for example, the own vehicle M performs low-speed follow-up travel to execute a third degree of driving support. In this case, it is not necessary to monitor the surroundings of the occupants. At this time, when the videophone becomes available and the use of the videophone is started, the second imaging unit 465 starts imaging the occupant.

At time T5, the third degree of driving support ends, and the second degree of driving support is switched to. Therefore, it is necessary to monitor the surroundings of the occupants. Further, at time T5, a display for switching from the second degree of driving support to manual driving is performed. In this case, the HMI control unit 120 displays information for causing the occupant to grip the steering wheel 82 in the driving support mode display.

At time T6, the occupant grips the steering wheel 82. In this case, the switching control unit 110 switches from the second degree of driving support by the automatic driving control unit 300 to the first degree of driving support by the driving support control unit 200. Further, the switching control unit 110 switches to manual operation after a predetermined time has elapsed for the first degree of driving support.

At time T7, the own vehicle M is switched to manual operation. In this case, the main switch 412 and the auto switch 414 are switched off in accordance with the timing when the own vehicle M is switched to manual operation.

According to the embodiment described above, the image pickup display unit mounted on the own vehicle M, the second display unit 463 for displaying an image, and the first image pickup unit 464 for taking an image of the occupant of the own vehicle M. A first imaging unit 464 that captures an image used for occupant state monitoring and a second imaging unit 465 that captures an image of the occupant of the own vehicle M, which are displayed by the second display unit 463. By providing a second display unit 460 in which a second image pickup unit 465 for capturing an image used for a service linked with an image is arranged in the unit, an image pickup unit for capturing images for different purposes can be arranged appropriately. Can be provided.

Although the embodiments for carrying out the present invention have been described above using the embodiments, the present invention is not limited to these embodiments, and various modifications and substitutions are made without departing from the gist of the present invention. Can be added.

1 ... Vehicle system, 10 ... Camera, 12 ... Radar device, 14 ... Finder, 16 ... Object recognition device, 20 ... Communication device, 30 ... HMI, 50 ... Navigation device, 60 ... MPU, 70 ... Vehicle sensor, 80 ... Driving Operator, 460 ... second display unit, 463 ... second display unit, 464 ... first imaging unit, 465 ... second imaging unit

Claims (5)

An imaging display unit mounted on a vehicle
A display unit that displays images and
A first imaging unit that captures an image of the occupant of the vehicle, and a first imaging unit that captures an image used for monitoring the state of the occupant.
It is a second imaging unit that is provided in such a form that the recognition degree by the occupant is higher than that of the first imaging unit and that images the occupant of the vehicle, and is displayed by the display unit. A second imaging unit that captures images used for services linked to images and used for videophones, an imaging display unit that is arranged inside the unit, and a recognition unit that recognizes the surrounding conditions of the vehicle.
A monitoring unit that determines whether or not the occupant is monitoring the surroundings of the vehicle based on the image captured by the first imaging unit.
A control unit that executes driving support that controls at least one or both of the steering and speed of the vehicle based on the surrounding conditions, and determines whether or not the driving support can be executed based on the monitoring result of the monitoring unit. Driving support device equipped with a control unit
Vehicle control system including. The second imaging unit is provided at a position where the occupant can see the display unit when he / she is visually recognizing the display unit.
The vehicle control system according to claim 1. The image pickup display unit includes a panel provided in the image pickup direction of the first image pickup unit.
The first imaging unit images the occupant through the panel.
The second imaging unit is arranged at a position visible to the occupant.
The vehicle control system according to claim 1 or 2. The display unit is provided in front of the driver's seat of the vehicle and in the center of the vehicle in the vehicle width direction.
The first imaging unit and the second imaging unit are arranged adjacent to the display unit.
The vehicle control system according to claim 3. The first imaging unit and the second imaging unit are arranged in an area of the display unit opposite to the driver's seat side in the vehicle width direction.
The vehicle control system according to claim 4.

Images