WO2020105685A1

WO2020105685A1 - Display control device, method, and computer program

Info

Publication number: WO2020105685A1
Application number: PCT/JP2019/045494
Authority: WO
Inventors: 誠秦
Original assignee: 日本精機株式会社
Priority date: 2018-11-23
Filing date: 2019-11-20
Publication date: 2020-05-28
Also published as: DE112019005849T5; CN113165510B; JP7255608B2; JPWO2020105685A1; CN113165510A

Abstract

The present invention makes an actual view in front of a driver's field of view, and an image displayed while overlapping the actual view more easily visible. A display control device which controls an image display unit for displaying an image in a region that overlaps a foreground when viewed from a driver of a vehicle displays a first information image 210 and a second information image 220 on the image display unit, detects the line-of-sight direction of the driver of the vehicle, when the first information image 210 is determined to be visually recognized, decreases the visibility thereof, and when the second information image 220 is determined to be visually recognized, prevents the visibility thereof from becoming lower than that of the first information image 210.

Description

Display control device, method, and computer program

The present disclosure relates to a display control device, a method, and a computer program that are used in a vehicle and that superimpose an image on the foreground of the vehicle for visual recognition.

Patent Document 1 discloses a vehicle image display system that relatively enhances the perceptibility of an image that is not perceived by the driver by reducing the conspicuousness of the image that matches the line of sight of the driver of the vehicle. It is disclosed.

JP, 2017-39373, A

However, in the image display system for a vehicle of Patent Document 1, it is possible to relatively facilitate visual attention to other images whose visual condition is not reduced by reducing the visual condition of the perceived image. Although possible, a situation in which attention should be paid to the image after being perceived and a potential driver's potential desire to pay attention are also assumed, and a display control device with further improved convenience is desired.

A summary of the specific embodiments disclosed herein is provided below. It should be understood that these aspects are presented only to provide an overview of the reader to these particular embodiments and are not intended to limit the scope of this disclosure. Indeed, the present disclosure may include various aspects not described below.

The outline of the present disclosure relates to improving the visibility of a driver's visual field in front of the driver's field of view and an image displayed in an overlapping manner with the actual scene. More specifically, the present invention relates to facilitating the transmission of information to the driver while suppressing the visual stimulus of the image displayed in an overlapping manner with the real scene.

Therefore, the display control device described in the present specification does not lower the visibility than the first information image that reduces the visibility when the line of sight is facing, and the degree to which the visibility of the first information image decreases when the line of sight is oriented. And a second information image. In some embodiments, the degree of change in the visibility of the information image when the line of sight is directed may be determined according to the magnitude of the risk potential of the information indicated by the information image.

It is a figure which shows the application example of the display system for vehicles which concerns on some embodiment. FIG. 3 is a block diagram of a vehicular display system according to some embodiments. FIG. 6 is a flow diagram of a process for reducing image visibility according to some embodiments. FIG. 6 is a flow diagram of a process for increasing image visibility according to some embodiments. It is a figure which shows the example of the image which the display system for vehicles displays according to some embodiments. It is a figure which shows the example of the image which the display system for vehicles displays according to some embodiments. FIG. 6 is a flow diagram of a process for reducing image visibility according to some embodiments. It is a figure which shows the example of the image which the display system for vehicles displays according to some embodiments. It is a figure which shows the example of the image which the display system for vehicles displays according to some embodiments.

1 and 2, below, a description of the configuration of an exemplary head-up display device is provided. An exemplary display control process flow will be described with reference to FIGS. 3, 4, and 6. 5A, 5B, and 7A, 7B provide example displays. The present invention is not limited to the following embodiments (including the contents of the drawings). Of course, changes (including deletion of components) can be added to the following embodiments. Further, in the following description, in order to facilitate understanding of the present invention, description of known technical matters will be appropriately omitted.

Refer to Figure 1. The image display unit 11 in the vehicle display system 10 is a head-up display (HUD: Head-Up Display) device provided in the dashboard 5 of the vehicle 1. The HUD device emits the display light 11a toward the front windshield 2 (which is an example of a member to be projected) and displays the image 200 in the virtual display area 100, so that the front windshield 2 is used. The image 200 is visually recognized by being superimposed on the foreground 300 which is the visually recognized real space.

Further, the image display unit 11 may be a head mounted display (hereinafter, HMD) device. The driver 4 mounts the HMD device on his / her head and sits in the seat of the host vehicle 1 to visually recognize the displayed image 200 by superimposing it on the foreground 300 through the front windshield 2 of the host vehicle 1. To do. The display area 100 in which the vehicle display system 10 displays the predetermined image 200 is fixed at a specific position with the coordinate system of the host vehicle 1 as a reference, and when the driver 4 turns in that direction, the display area 100 is displayed at the specific position. The image 200 displayed in the fixed display area 100 can be visually recognized.

Under the control of the display control device 13, the image display unit 11 includes obstacles (pedestrians, bicycles) existing in the foreground 300, which is a real space (real scene) visually recognized through the front windshield 2 of the vehicle 1. , A motorcycle, another vehicle, etc., a road surface, a road sign, the vicinity of the real object 310 such as a feature (building, bridge, etc.) (an example of the positional relationship between the image and the real object), and the position overlapping the real object 310 ( By displaying the image 200 at a position (an example of the positional relationship between the image and the real object) or a position (an example of the positional relationship between the image and the real object) set with the real object 310 as a reference, the visual augmented reality ( AR: Augmented Reality) can also be formed. The image display unit 11 displays a first information image 210 (AR image) and a second information image 220 (AR image) that differ according to the type of information to be provided (described in detail later).

FIG. 2 is a block diagram of a vehicle display system 10 according to some embodiments. The vehicle display system 10 includes an image display unit 11 and a display control device 13 that controls the image display unit 11. The display controller 13 includes one or more I / O interfaces 14, one or more processors 16, one or more storage units 18, and one or more image processing circuits 20. .. The various functional blocks depicted in FIG. 2 may be implemented in hardware, software, or a combination of both. FIG. 2 is only one embodiment of an implementation, and the illustrated components may be combined into fewer components or there may be additional components. For example, image processing circuitry 20 (eg, a graphics processing unit) may be included in one or more processors 16.

As illustrated, the processor 16 and the image processing circuit 20 are operably connected to the storage unit 18. More specifically, the processor 16 and the image processing circuit 20 execute the program stored in the storage unit 18 to operate the vehicle display system 10, such as generating or / and transmitting image data. It can be carried out. The processor 16 or / and the image processing circuit 20 may be at least one general purpose microprocessor (eg, central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA). , Or any combination thereof. The storage unit 18 includes any type of magnetic medium such as a hard disk, any type of optical medium such as CD and DVD, any type of semiconductor memory such as volatile memory, and non-volatile memory. Volatile memory may include DRAM and SRAM, and non-volatile memory may include ROM and NVROM.

As shown, the processor 16 is operably connected to the I / O interface 14. For example, the I / O interface 14 uses the vehicle display system 10 as a personal area network (PAN) such as a Bluetooth (registered trademark) network or a local area network (LAN) such as an 802.11x Wi-Fi (registered trademark) network. A wireless communication interface for connecting to a wide area network (WAN) such as a 4G or LTE cellular network can be included. The I / O interface 14 may also include a wired communication interface such as, for example, a USB port, a serial port, a parallel port, an OBDII, and / or any other suitable wired communication port.

As shown in the figure, the processor 16 is interoperably connected to the I / O interface 14 so as to be informed of various other electronic devices connected to the vehicle display system 10 (I / O interface 14). Can be given and received. The I / O interface 14 includes, for example, a vehicle ECU 401 provided in the host vehicle 1, a road information database 403, a host vehicle position detector 405, a vehicle exterior sensor 407, a line-of-sight direction detector 409, an eye position detector 411, and portable information. The terminal 413, the communication connection device 420 outside the vehicle, and the like are operably connected. The image display unit 11 is operably connected to the processor 16 and the image processing circuit 20. Therefore, the image displayed by the image display unit 11 may be based on the image data received from the processor 16 and / or the image processing circuit 20. The processor 16 and the image processing circuit 20 control the image displayed by the image display unit 11 based on the information obtained from the I / O interface 14. The I / O interface 14 may include a function of processing (converting, calculating, analyzing) information received from another electronic device or the like connected to the vehicle display system 10.

The host vehicle 1 is in the state of the host vehicle 1 (for example, mileage, vehicle speed, accelerator pedal opening, engine throttle opening, injector fuel injection amount, engine speed, motor speed, steering angle, shift position, drive mode). , Various warning states) are included in the vehicle ECU 401. The vehicle ECU 401 controls each unit of the host vehicle 1, and can transmit vehicle speed information indicating the current vehicle speed of the host vehicle 1 to the processor 16, for example. It should be noted that the vehicle ECU 401 can send the determination result of the data detected by the sensor and / or the analysis result to the processor 16 in addition to or instead of simply sending the data detected by the sensor to the processor 16. For example, information indicating whether the host vehicle 1 is traveling at a low speed or is stopped may be transmitted to the processor 16. Further, the vehicle ECU 401 may transmit an instruction signal for instructing the image 200 displayed by the vehicle display system 10 to the I / O interface 14, and at this time, the coordinates of the image 200, the notification necessity degree of the image 200, or / And the necessity degree related information which is a basis for determining the notification necessity degree may be added to the instruction signal and transmitted.

The own vehicle 1 may include a road information database 403 including a navigation system and the like. The road information database 403 is based on the position of the own vehicle 1 acquired from the own vehicle position detection unit 405 described later, and is road information (lane, white line, stop line, Crosswalk, width of road, number of lanes, intersection, curve, branch road, traffic regulation, etc.), presence / absence of feature information (buildings, bridges, rivers, etc.), position (including distance to own vehicle 1), direction , Shape, type, detailed information, etc. may be read and transmitted to the processor 16. Further, the road information database 403 may calculate an appropriate route from the starting point to the destination and send it to the processor 16 as navigation information.

The host vehicle 1 may include a host vehicle position detection unit 405 such as a GNSS (Global Navigation Satellite System). The road information database 403, the portable information terminal 413, which will be described later, and / or the external communication connecting device 420 acquires the position information of the own vehicle 1 from the own vehicle position detection unit 405 continuously, intermittently, or at every predetermined event. Thus, the information around the host vehicle 1 can be selected / generated and transmitted to the processor 16.

The host vehicle 1 may include one or more vehicle exterior sensors 407 that detect a real object existing in the vicinity of the host vehicle 1 (particularly the foreground 300 in this embodiment). The real object detected by the vehicle exterior sensor 407 includes, for example, a pedestrian, a bicycle, a motorcycle, another vehicle (a preceding vehicle, etc.), a road surface, a marking line, a roadside object, and / or a feature (a building etc.). Good. As the vehicle exterior sensor, for example, a millimeter wave radar, an ultrasonic radar, a radar sensor such as a laser radar, there is a camera sensor consisting of a camera and an image processing device, may be configured by a combination of both the radar sensor, the camera sensor, It may be configured with only one of them. Conventionally known methods are applied to the object detection by the radar sensor and the camera sensor. By detecting an object by these sensors, the presence or absence of a real object in the three-dimensional space, and the position of the real object (relative distance from the own vehicle 1 and the traveling direction of the own vehicle 1 when the real object exists) are detected. Position in the left-right direction in the front-back direction, vertical position, etc.), size (size in the horizontal direction (horizontal direction), height direction (vertical direction), etc.), movement direction (lateral direction (horizontal direction)) , Depth direction (front-back direction), movement speed (lateral direction (left-right direction), depth direction (front-back direction)), and / or type may be detected. One or more out-of-vehicle sensors 407 detect a real object in front of the host vehicle 1 for each detection cycle of the sensors and detect real object related information (presence or absence of real object, which is an example of real object related information). When a real object exists, information such as the position, size, and / or type of each real object) can be transmitted to the processor 16. The real object related information may be transmitted to the processor 16 via another device (for example, the vehicle ECU 401). Further, when a camera is used as a sensor, an infrared camera or a near infrared camera is desirable so that a real object can be detected even when the surroundings are dark such as at night. When using a camera as a sensor, a stereo camera that can acquire a distance and the like by parallax is desirable.

The host vehicle 1 may include a line-of-sight direction detection unit 409 including an infrared camera that detects the gaze direction of the driver 4 (hereinafter, also referred to as “line-of-sight direction”) that captures the face of the driver 4. The processor 16 can specify the line-of-sight direction of the driver 4 by acquiring an image captured by the infrared camera (an example of information that can estimate the line-of-sight direction) and analyzing the captured image. Note that the processor 16 may acquire, from the I / O interface 14, the line-of-sight direction of the driver 4 specified by the line-of-sight direction detection unit 409 (or another analysis unit) from the image captured by the infrared camera. Further, the method for acquiring the driver's 4 line-of-sight direction of the vehicle 1 or the information capable of estimating the driver's 4 line-of-sight direction is not limited to these, and the EOG (Electro-oculogram) method, corneal reflex Method, scleral reflection method, Purkinje image detection method, search coil method, infrared fundus camera method, and other known gaze direction detection (estimation) techniques may be used.

The host vehicle 1 may include an eye position detection unit 411 including an infrared camera that detects the position of the eyes of the driver 4. The processor 16 can specify the eye position of the driver 4 by acquiring an image captured by the infrared camera (an example of information that can estimate the eye position) and analyzing the captured image. The processor 16 may acquire the information on the position of the eyes of the driver 4 identified from the image captured by the infrared camera from the I / O interface 14. The method for acquiring the position of the eyes of the driver 4 of the vehicle 1 or the information capable of estimating the position of the eyes of the driver 4 is not limited to these, and known eye position detection (estimation) It may be acquired using a technology. The processor 16 adjusts at least the position of the image 200 based on the position of the eyes of the driver 4 so that the viewer who has detected the eye position of the image 200 superimposed on the desired position of the foreground 300 (driver 4). May be visually confirmed.

The mobile information terminal 413 is a smartphone, a laptop computer, a smart watch, or other information device that can be carried by the driver 4 (or another occupant of the vehicle 1). The I / O interface 14 can communicate with the mobile information terminal 413 by pairing with the mobile information terminal 413, and data recorded in the mobile information terminal 413 (or a server via the mobile information terminal). To get. The mobile information terminal 413 has, for example, the same function as the road information database 403 and the vehicle position detection unit 405 described above, acquires the road information (an example of the real object related information), and transmits it to the processor 16. Good. The mobile information terminal 413 may also acquire commercial information (an example of real object-related information) related to a commercial facility in the vicinity of the vehicle 1, and transmit the commercial information to the processor 16. The mobile information terminal 413 transmits schedule information of the owner (for example, the driver 4) of the mobile information terminal 413, incoming information at the mobile information terminal 413, mail reception information, etc. to the processor 16, The image processing circuit 20 may generate or / and transmit image data regarding these.

The outside-vehicle communication connection device 420 is a communication device for exchanging information with the own vehicle 1, and, for example, another vehicle connected to the own vehicle 1 via vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), pedestrian-to-vehicle communication (V2P: It is a network communication device connected by a pedestrian (a portable information terminal carried by a pedestrian) and a road-to-vehicle communication (V2I: Vehicle To road Infrastructure) connected by a Vehicle To Pedestrian. Includes everything connected by V2X (Vehicle To Everything). The extra-vehicle communication connection device 420 acquires the position of, for example, a pedestrian, a bicycle, a motorcycle, another vehicle (such as a preceding vehicle), a road surface, a marking line, a roadside object, and / or a feature (such as a building), and the processor 16 May be sent to. In addition, the vehicle exterior communication connection device 420 may have the same function as the own vehicle position detection unit 405 described above, may acquire the position information of the own vehicle 1 and may transmit the position information to the processor 16, and further, the road information database described above. It also has the function of 403, and may acquire the road information (an example of real object-related information) and send it to the processor 16. The information acquired from the vehicle exterior communication connection device 420 is not limited to the above.

The software components stored in the storage unit 18 are the real object related information detection module 502, the notification necessity degree detection module 504, the image type determination module 506, the image position determination module 508, the image size determination module 510, and the eye position detection module 512. A visual recognition module 514, an action determination module 516, and a graphic module 518.

The real object related information detection module 502 detects the position and size of the real object existing in the foreground 300 of the own vehicle 1, which is the basis for determining the coordinates and size of the image 200 described later. The real object-related information detection module 502 uses, for example, the road information database 403, the vehicle exterior sensor 407, or the vehicle exterior communication connection device 420 to identify the real object 310 existing in the foreground 300 of the vehicle 1 (for example, the road surface shown in FIG. 5A). 311, a preceding vehicle 312, a pedestrian 313, a building 314, etc.) (height direction (vertical direction) when the driver 4 of the own vehicle 1 visually recognizes the traveling direction (front) of the own vehicle 1 from the driver's seat). , The position in the horizontal direction (horizontal direction), and the position in the depth direction (forward direction) may be added thereto, and the size of the real object 310 (height direction, horizontal size) is acquired. You may.

In addition, the real object related information detection module 502 may determine whether or not the environment (the bad weather such as rain, fog, or snow) in which the detection accuracy decreases when detecting the position or size of the real object by the vehicle exterior sensor 407. Good. For example, the position detection accuracy of the real object is calculated, and the determination result of the degree of decrease in the detection accuracy, the position where the detection accuracy is decreased, and the environment (bad weather) in which the detection accuracy is decreased are determined by the processor. 16 may be transmitted. The bad weather may be determined by acquiring the weather information of the position where the host vehicle 1 travels from the portable information terminal 413, the vehicle exterior communication connection device 420, or the like.

Further, the real object related information detection module 502 is a source of determining the content of the image 200 described below (hereinafter, also referred to as “image type” as appropriate), which is information about the real object existing in the foreground 300 of the host vehicle 1. (Real object related information) may be detected. The real object related information is, for example, type information indicating the type of the real object such as a pedestrian or another vehicle, moving direction information indicating the moving direction of the real object, distance to the real object, or the like. It is distance time information indicating the arrival time, or individual detailed information of a real object such as a charge of a parking lot (real object) (but not limited to these). For example, the real object related information detection module 502 obtains the type information, the distance time information, and / or the individual detailed information from the road information database 403 or the portable information terminal 413, and the type information, the moving direction information, or the out-of-vehicle sensor 407. Alternatively, the type information, the moving direction information, the distance time information, and / or the individual detailed information may be detected from the vehicle exterior communication connection device 420 by acquiring the / and distance time information.

The notification necessity level detection module 504 notifies the driver 4 of the real object position information detected by the real object related information detection module 502 and the real object related information detected by the real object related information detection module 502. Necessity) is detected. The notification necessity degree detection module 504 may detect the notification necessity degree from various other electronic devices connected to the I / O interface 14. In addition, the electronic device connected to the I / O interface 14 in FIG. 2 transmits information to the vehicle ECU 401, and the notification necessity detection module 504 detects (acquires) the notification necessity degree determined by the vehicle ECU 401 based on the received information. ) May be. The “information need level” is, for example, a risk level derived from the degree of seriousness of the possibility itself, an urgency level derived from the length of the reaction time required to take a reaction action, the own vehicle 1 or the driver 4 ( Alternatively, it may be determined based on the effectiveness derived from the situation of other occupants of the vehicle 1 or a combination thereof (the indicator of the notification necessity degree is not limited to these).

Further, the notification necessity detection module 504 may detect the necessity degree related information that is the basis for estimating the notification necessity degree of the image 200, and estimate the notification necessity degree from this. Necessity degree related information that is a basis for estimating the notification necessity degree of the image 200 may be estimated, for example, by the position and type of a real object or traffic regulation (an example of road information), and is connected to the I / O interface 14. It may be estimated based on other information input from various other electronic devices described above, or in consideration of other information. The vehicle display system 10 may not have the function of estimating the notification necessity degree, and a part or all of the function of estimating the notification necessity degree may be included in the display control device of the vehicle display system 10. It may be provided separately from 13.

The image type determination module 506, for example, the type and position of the real object detected by the real object related information detection module 502, the type, number, or / and of the real object related information detected by the real object related information detection module 502. The type of the image 200 to be displayed on the real object can be determined based on the degree of the (necessary) notification need detected by the notification need detection module 504. Further, the image type determination module 506 may increase or decrease the type of the image 200 to be displayed according to the determination result by the visual recognition detection module 514 described later. Specifically, when the real object 310 is in a state where it is difficult for the driver 4 to visually recognize it, the types of the image 200 visually recognized by the driver 4 in the vicinity of the real object may be increased.

The image position determination module 508 is based on the position of the real object detected by the real object-related information detection module 502, and the coordinates of the image 200 (when the driver 4 views the direction of the display area 100 from the driver's seat of the own vehicle 1). The horizontal direction (X-axis direction) and the vertical direction (Y-axis direction) are determined. When displaying the image 200 related to a specific real object, the image position determination module 508 determines the coordinates of the image 200 so as to have a predetermined positional relationship with the specific real object. For example, the positions of the image 200 in the left-right direction and the vertical direction are determined so that the center of the image 200 is visually recognized so as to overlap with the center of the real object. Further, the image position determination module 508 can determine the coordinates of the image 200 so that the image 200 has a predetermined positional relationship with reference to a real object that is not directly related. For example, as shown in FIG. 5A, the lane markings based on the left lane marking 311a (an example of a real object) and the right lane marking 311b (an example of a real object) of the lane (road surface 310) on which the vehicle 1 is traveling are used. The coordinates of the first FCW image 221 described later that is related to the preceding vehicle 312 (an example of a specific real object) that is not directly related to may be determined (or corrected). The "predetermined positional relationship" can be adjusted depending on the situation of the real object or the host vehicle 1, the type of the real object, the type of the displayed image, and the like.

The image size determination module 510 associates and displays the image 200 detected by the real object related information detection module 502 with the type and position of the real object and the type of real object related information detected by the real object related information detection module 502. , Number, and / or the size of the (necessary) notification need detected by the notification need detection module 504, the size of the image 200 can be determined. Also, the image size determination module 510 can change the image size according to the number of types of the image 200. For example, the image size may be reduced as the number of types of the image 200 increases.

The eye position detection module 512 detects the position of the eyes of the driver 4 of the vehicle 1. The eye position detection module 512 determines where the eye height of the driver 4 is in the height region provided in a plurality of stages, detects the eye height of the driver 4 (position in the Y-axis direction), Detection of the eye height (position in the Y axis direction) and position in the depth direction (position in the Z axis direction) of the driver 4, and / or the position of the eye of the driver 4 (position in the X, Y, Z axis directions) ) Detection, and various software components for performing various operations associated therewith. The eye position detection module 512 can acquire the eye position of the driver 4 from the eye position detection unit 411, or can estimate the eye position including the eye height of the driver 4 from the eye position detection unit 411, for example. Information is received and the eye position including the eye height of the driver 4 is estimated. The information capable of estimating the position of the eyes includes, for example, the position of the driver's seat of the vehicle 1, the position of the driver's 4 face, the height of the sitting height, and the input value by the driver 4 on the operation unit (not shown). Good.

The visual recognition detection module 514 detects whether the driver 4 of the vehicle 1 visually recognizes the predetermined image 200. The visual recognition detection module 514 is for executing various operations regarding whether the driver 4 visually recognizes the predetermined image 200 and whether the driver 4 visually recognizes the periphery (vicinity) of the predetermined image 200. Includes various software components. The visual recognition detection module 514 compares the gaze position GZ of the driver 4, which will be described later, acquired from the line-of-sight direction detection unit 409 with the position of the image 200 acquired from the graphic module 518, and the driver 4 visually recognizes the image 200. It may be determined whether or not the image has been visually recognized, and the determination result as to whether or not the image is visually recognized, and information for identifying the visually recognized image 200 may be transmitted to the processor 16.

In addition, the visual recognition detection module 514 determines an area of a predetermined width that is preset from the outer edge of the image 200 to the outside in order to determine whether the driver 4 visually recognizes the periphery (vicinity) of the predetermined image 200. It may be set as the periphery of the image 200, and when a gaze position GZ described later enters the periphery of the image 200, it may be determined that the driver 4 of the own vehicle 1 visually recognizes the predetermined image 200. The visual recognition determination is not limited to these means.

Further, the visual recognition detection module 514 may detect what the driver 4 is visually recognizing other than the image 200. For example, the visual recognition detection module 514 detects the position of the real object 310 existing in the foreground 300 of the vehicle 1 detected by the real object-related information detection module 502 and the gaze position of the driver 4 to be described later acquired from the line-of-sight direction detection unit 409. Information that identifies the real object 310 that is gazing and that identifies the real object 310 that has been visually recognized may be transmitted to the processor 16 by comparing GZ and GZ.

The behavior determination module 516 detects the behavior of the driver 4 that is not appropriate for the information shown in the first information image 210 described later. In some of the first information images 210, the inappropriate behavior of the driver 4 is stored in the storage unit 18 in association with each other. The behavior determination module 516 particularly determines whether or not an inappropriate behavior of the driver 4 associated with the first information image 210 with reduced visibility is detected. For example, when the first information image 210 includes route guidance information, if the driver 4 is gazing at a branch road different from the direction indicated by the route guidance, it is determined that an inappropriate behavior of the driver 4 has been detected. You may. In addition, when the first information image 210 includes traffic regulation information, it may be determined that an inappropriate behavior of the driver 4 has been detected when behavior that is likely to violate this traffic regulation is performed. The information collected by the action determination module 516 to determine the action is the state of the vehicle 1 input from the vehicle ECU 401 (for example, mileage, vehicle speed, accelerator pedal opening, engine throttle opening, injector fuel injection amount, The engine rotation speed, the motor rotation speed, the steering angle, the shift position, the drive mode, various warning states), the line-of-sight direction input from the line-of-sight direction detection unit 409, and the like, are not limited to these.

The graphics module 518 allows the displayed image 200 to have a visual effect (eg, brightness, transparency, saturation, contrast, or other visual characteristic), size, display position, display distance (from driver 4 to image 200). Various known software components for changing the distance). The graphic module 518 displays the type set by the image type determination module 506 and the coordinates set by the image position determination module 508 (the left-right direction when the driver 4 views the display area 100 from the driver's seat of the vehicle 1 (X The image 200 is displayed so that the driver 4 can visually recognize it in the image size set by the image size determination module 510 in the axial direction) and the vertical direction (including at least the Y-axis direction).

The graphic module 518 has a first information image 210 and a second information image 220, which are augmented reality images (AR images) arranged so as to have a predetermined positional relationship with the real object 310 of the foreground 300 of the vehicle 1. , At least. When the visual recognition detection module 514 determines that the first information image 210 is visually recognized, the first information image 210 reduces the visibility (including non-display). On the other hand, when the visual recognition detection module 514 determines that the second information image 220 is visually recognized, the second information image 220 does not have lower visibility than the first information image 210 (smaller than the degree of reduction in visibility of the first information image 210 ( (To a lesser extent) including lowering the visibility, or not changing the visibility, or increasing the visibility).

"Reduce visibility" means to reduce brightness, increase transparency, decrease saturation, reduce contrast, reduce size, reduce image types, combinations of these, or It may include a combination of other elements. Conversely, "increasing visibility" means increasing brightness, decreasing transparency, increasing saturation, increasing contrast, increasing size, increasing the number of image types, combinations of these, Alternatively, a combination of these and other elements may be included.

The first information image 210 is information with a relatively low risk derived from the degree of seriousness that can occur, and for example, an arrow image (an example of navigation information) displayed on a road surface to indicate a route, a destination. Text image (an example of navigation information), an image showing the distance to the next turning point (an example of navigation information), and a POI (Point of Interest) image (a feature information of the feature information) indicating a store or facility existing in the foreground 300. Example), images related to road signs (guidance signs, warning signs, regulation signs, instruction signs, auxiliary signs), and ACC (Adaptive Cruise Control) that displays the inter-vehicle distance set when following a preceding vehicle on the road surface. Such as images.

The second information image 220 is information having a relatively high degree of danger derived from the degree of seriousness of the vehicle itself, and for example, a forward collision prediction visually recognized near an obstacle existing in the foreground 300 of the vehicle 1. It is an image of a warning (FCW: Forward Collision Warning).

FIG. 3 is a flow diagram of a process for reducing the visibility of an image according to some embodiments. First, the line-of-sight direction (gaze position GZ described later) of the driver 4 of the vehicle 1 is acquired (step S11), and the position of the displayed image 200 is acquired (step S12).

Next, the processor 16 identifies the target visually recognized by the driver 4 by comparing the line-of-sight direction acquired in step S11 with the position of the image 200 acquired in step S12 (step S13). Specifically, the processor 16 determines whether the first information image 210 is visually recognized, the second information image 220 is visually recognized, or the first information image 210 or the second information image 220 is not visually recognized. To do. If the first information image 210 is visually recognized, which image of the first information image 210 is visually recognized is specified.

When it is determined that the driver 4 visually recognizes the first information image 210 in step S13, the processor 16 reduces the visibility of the visually recognized first information image 210. At this time, the processor 16 may hide the first information image 210. Further, when determining that the driver 4 visually recognizes the second information image 220 in step S <b> 13, the processor 16 does not reduce the visibility of the second information image 220 or makes the visibility of the second information image 220 first. The information image 210 is made smaller (smaller) than the degree of visibility deterioration.

FIG. 4 is a flow diagram of a process of increasing the visibility of the second information image according to some embodiments. When the visibility of at least one first information image 210 is reduced by the process of step S14, the process is started.

First, the processor 16 acquires the behavior of the driver 4 (step S21), and the behavior of the driver 4 that is not appropriate for the information indicated by the first information image 210 whose visibility is reduced by the processing of step S14. Is detected (step S22), and when it is determined that the inappropriate behavior of the driver 4 is made, the visibility of the second information image 220, which has been reduced in visibility, is increased (step S23). ..

The operation of the processing process described above can be performed by executing one or more functional modules of an information processing device such as a general-purpose processor or an application-specific chip. All of these modules, a combination of these modules, and / or a combination with general hardware capable of substituting their functions are included in the scope of protection of the present invention.

The functional blocks of the vehicular display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various described embodiments. The functional blocks described in FIG. 2 may optionally be combined or one functional block may be separated into two or more sub-blocks to implement the principles of the described embodiments. As will be appreciated by those skilled in the art. Thus, the description herein optionally supports any possible combination or division of the functional blocks described herein.

5A and 5B are diagrams showing changes in the image 200 displayed by the vehicular display system 10 when it is determined that the first information image 210 is visually recognized. In FIG. 5A, the first information image 210 includes a navigation image 211 that is visually recognized as being superimposed on a road surface 311 and that indicates a guide route, and a notation “P” that indicates a parking lot that instructs a building 314 (actual object 310). And a POI image 212 that is an illustration. The second information image 220 is a first FCW image 221 visually recognized linearly (in a line) on the road surface 311 behind the preceding vehicle 312 traveling in front of the host vehicle 1, and the opposite lane side of the traveling lane of the host vehicle 1. The second FCW image 222 visually recognized in an arc shape on the road surface 311 around the pedestrian 313 who walks on the sidewalk. In addition to this, as a third information image that is not an augmented reality image (AR image) that is arranged so as to have a predetermined positional relationship with the real object 310 of the foreground 300 of the vehicle 1, notation of "80" indicating the speed limit A road information image 231 that is an illustration including "," and a speed image 232 that indicates the speed of the host vehicle 1 and is displayed as "35 km / h" are displayed. The display area 100 is a first display area 110 and a second display area 120 that is arranged vertically below the first display area 110 (Y axis negative direction) when the front is visually recognized from the driver's seat of the vehicle 1. The first information image 210 and the second information image 220, which are AR images, are displayed in the first display area 110, and the

third information images

231 and 232 are displayed in the second display area 120.

As shown in FIG. 5A, when the gaze position GZ is in the navigation image 211 that is the first information image 210, the processor 16 executes the instruction of step S14 in FIG. 3 and is visually recognized as shown in FIG. 5B. The navigation image 211 (first information image 210) is hidden (an example of reduced visibility). At this time, instead of the navigation image 211, which is the augmented reality image (AR image) whose displayed position is related to the position of the real object 310, the displayed position is a non-AR image which is not related to the position of the real object 310. The first navigation image 213 (an example of a related image) and the second navigation image 214 (an example of a related image) are displayed in the second display area 120. The first navigation image 213 is a simplified image showing the approximate direction of the next fork. In addition, the second navigation image 214 is a text described as "200 m ahead" indicating the distance to the next branch road.

In addition, in the situation where the visibility of the navigation image 211, which is the first information image 210 shown in FIG. 5B, has deteriorated, when the gaze position GZ is on the branch road of the nearest intersection, the driver 4 is at the nearest intersection. You can infer that you are about to turn. Assuming that the branch road presented by the navigation image 211 is not the branch road of the latest intersection, the behavior of the driver 4 who can guess that the vehicle will turn at the latest intersection (the behavior of gazing at the latest intersection of the intersection) is appropriate. Not an action. Therefore, when such an inappropriate action is detected, the processor 16 increases the visibility of the navigation image 211, which is the first information image 210 that has been reduced in visibility. That is, the state shown in FIG. 5B is changed to the state shown in FIG. 5A. This allows the driver 4 to recognize that the vehicle should not turn at the nearest intersection.

FIG. 6 is a flow diagram of a process for reducing the visibility of an image according to some embodiments. The flow chart of FIG. 6 corresponds to the flow chart of FIG. 3, and steps S31, 32, and 33 of FIG. 6 correspond to steps S11, S12, and S13 of FIG. 3, respectively. Here, step S34 which is a change point from FIG. 3 will be described.

When it is determined that the driver 4 visually recognizes the first information image 210 in step S13, the processor 16 reduces the visibility of the visually recognized first information image 210. Further, when it is determined that the driver 4 visually recognizes the second information image 220 in step S33, the processor 16 changes the visually recognized second information image 220 from a still image to a moving image, or the visually recognized second information image. 220 and other nearby second information image 220 may be changed from a still image to a moving image.

In some embodiments, the processor 16 may continuously change the number of second information images 220 that are viewed and other second information images 220 that are nearby. Specifically, when the gaze position GZ is in the vicinity of the first FCW image 221 which is the second information image 220, the processor 16 displays the first FCW image 221 that has been visually recognized and the other second FCW image 222 that is close to the first FCW image 221. 7B may be changed continuously and / or intermittently between the state where the number of images shown in FIG. 7B is small and the state where the number of images shown in FIG. 7B is large. The moving image is not particularly defined, but the shape of the image repeatedly and / or intermittently changes, the number of images repeatedly changes, the position of the image repeatedly changes, repeatedly blinks, and repeatedly. The size may be changed, and the like.

As described above, the display control device of the present embodiment controls the image display unit 11 (12) that displays the image 200 in the area overlapping the foreground 300 when viewed from the driver 4 of the vehicle 1. 13, one or more I / O interfaces 14, one or more processors 16, a memory 18, and stored in memory 18 and executed by one or more processors 16. And one or more computer programs configured as described above, the one or more processors 16 obtaining the line-of-sight direction of the driver 4 from the one or more I / O interfaces 14. However, based on the line-of-sight direction, a first information image 210 that reduces visibility when it is determined to be viewed, and a second information image 220 that does not reduce visibility when compared to the first information image 210 when determined to be viewed. , Is displayed, the command is executed. As a result, the change in visibility after visual recognition differs depending on the type of image, and in the case of the first information image, the visibility is reduced, making it easier to see the real scene in front of the driver's field of view. If it is the second information image, the visibility is not lowered so much, so that the image can be made easy to see even after the visual recognition.

Further, in some embodiments, the degree of change in the visibility of the second information image 220 when the line of sight is directed may be determined according to the magnitude of the risk potential of the information indicated by the second information image 220. .. The processor 16 may reduce the degree of reduction in visibility when the line of sight is directed, as the risk potential of the information indicated by the second information image 220 is higher. That is, if the risk potential is low, the visibility of the second information image 220 when the line of sight is directed is greatly reduced (the degree of reduction is increased). Note that the processor 16 may change the degree of reduction in visibility according to a risk potential that is predetermined according to the type of the second information image 220, and the second information image 220 is being displayed (I / I). It may be changed according to the risk potential calculated according to the information obtained from the O interface 14).

Further, in some embodiments, after the line of sight turns toward the second information image 220 or after the line of sight turns toward the second information image 220 to reduce the visibility, the risk of the information indicated by the second information image 220. If the potential does not increase, the visibility of the second information image 220 may be reduced. After the line of sight of the second information image 220 or the line of sight of the second information image 220 reduces the visibility, the processor 16 sets the risk potential of the information indicated by the second information image 220 for a predetermined period. If the risk potential does not become higher than the predetermined threshold value, does not significantly increase, or decreases, it is determined that it is not necessary to maintain the visibility as it is, and the second information image 220 is visually recognized. The sex may be reduced.

Also, in some embodiments, the second information image 220 may be displayed as a moving image when the line of sight turns toward the second information image 220. The processor 16 normally displays the second information image 220 as a still image and, when the line of sight is turned, makes part or all of the second information image 220 a moving image. It is recognized that the first information image 210 has a lower visibility when visually recognized, but becomes a moving image when the second information image 220 is visually recognized, and thus is different from the information indicated by the first information image 210. Thus, the attention can be directed to the second information image 220. The second information image 220 may be changed to a still image after being displayed as a moving image for a certain period. Further, the processor 16 may reduce the visibility of the second information image 220 that is visually recognized at the same timing that the second information image 220 is displayed as a moving image, or the visibility is reduced after the moving image is displayed. You may let me.

Further, in some embodiments, when the line of sight is directed to the predetermined second information image 220 when the plurality of second information images 220 are displayed, the predetermined second information image 220 and other second information images are displayed. 220 may be displayed as a moving image. For example, in FIG. 5A, when the driver 4 visually recognizes the second FCW image 222 displayed corresponding to the pedestrian 313, the second FCW image 222 and the first FCW image displayed corresponding to the other preceding vehicle 312. 221 may be a moving image. In this manner, by displaying the other information images in the same manner, the visual attention can be directed not only to the visually recognized image but also to the similar image.

Further, in some embodiments, when the line of sight is directed to the predetermined second information image 220 when the plurality of second information images 220 are displayed, the predetermined second information image 220 and the predetermined second information image are displayed. Another second information image 220 near 220 may be displayed as a moving image.

Further, in some embodiments, when the line of sight is directed to the predetermined second information image 220 when the plurality of second information images 220 are displayed, the predetermined second information image 220 and other second information images are displayed. 220 may be displayed as a moving image at the same cycle. This makes it easy to identify where an image similar to the image with the field of view is displayed.

Further, in some embodiments, after the information regarding the behavior that is not appropriate for the information indicated by the first information image 210 is acquired and the visibility of the first information image 210 is reduced, the first information image 210 is displayed. When it is possible to determine that the driver 4 performs an action that is not appropriate for the information shown, the visibility of the first information image 210 may be increased.

Also, in some embodiments, after reducing the visibility of the first information image 210, the related images (213, 214) related to the first information image 210 are changed to the first information image 210 or the second information image. 220 may be displayed in the second display area 102 different from the first display area 101 in which it is displayed. That is, the area of the foreground 300 that overlaps the first information image 210 can be easily seen, and the information indicated by the first information image 210 can be confirmed in the other second display area 102. Note that the processor 16 may reduce the visibility of one first information image 210 and newly display two or more related images. As a result, more information can be recognized in the related image, and it is possible to suppress deterioration in information recognition due to deterioration in visibility of the first information image 210 that is an AR image.

DESCRIPTION OF SYMBOLS 1 ... Own vehicle, 2 ... Front windshield, 4 ... Driver, 5 ... Dashboard, 10 ... Vehicle display system, 11 ... Image display part, 11a ... Display light, 13 ... Display control device, 14 ... I / O Interface, 16 ... Processor, 18 ... Storage unit, 20 ... Image processing circuit, 22 ... Memory, 100 ... Display area, 110 ... First display area, 120 ... Second display area, 200 ... Image, 210 ... First information image , 211 ... Navigation image, 212 ... POI image, 213 ... First navigation image, 214 ... Second navigation image, 220 ... Second information image, 221 ... First FCW image, 222 ... Second FCW image, 231 ... Road information image ( Third information image), 232 ... Velocity image (third information image), 300 ... Foreground, 310 ... Real object, 311 ... Road surface, 311a ... Marking line, 311b ... Marking line, 312 ... Leading vehicle, 313 ... Pedestrian, 314 ... Building, 320 ... Lane, 401 ... Vehicle ECU, 403 ... Road information database, 405 ... Own vehicle position detection unit, 407 ... Exterior sensor, 409 ... Line-of-sight direction detection unit, 411 ... Eye position detection unit, 413 ... Mobile information Terminal, 420 ... External communication connection device, 502 ... Real object related information detection module, 504 ... Notification necessity detection module, 506 ... Image type determination module, 508 ... Image position determination module, 510 ... Image size determination module, 512 ... Eyes Position detection module, 514 ... Visual detection module, 516 ... Behavior determination module, 518 ... Graphic module, GZ ... Gaze position

Claims

In a display control device (13) for controlling an image display unit (11, 12) that displays an image (200) in a region overlapping the foreground (300) when viewed from the driver of the vehicle (1),
One or more I / O interfaces (14),
One or more processors (16),
A memory (18),
One or more computer programs stored in the memory (18) and configured to be executed by the one or more processors (16);
Said one or more processors (16)
Obtaining the driver's gaze direction from the one or more I / O interfaces (14),
Based on the line-of-sight direction, a first information image (210) that reduces visibility when it is determined to be viewed, and a second information image that does not reduce visibility when compared to the first information image (210) when determined to be viewed The information image (220) is displayed,
A display controller that executes instructions.
Said one or more processors (16)
The degree of change in the visibility of the second information image (220) when the line of sight is directed is determined according to the magnitude of the risk potential of the information indicated by the second information image (220).
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
Risk of information indicated by the second information image (220) after the line of sight of the second information image (220) or after the line of sight of the second information image (220) reduces visibility. If the potential does not increase, the visibility of the second information image (220) is reduced.
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
When the line of sight is directed to the second information image (220), the second information image (220) is displayed as a moving image.
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
When the line of sight is directed toward the predetermined second information image (221) when the plurality of second information images (220) are displayed, the predetermined second information image (221) and other second information images Display (222 ~) as a video,
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
When the line of sight is directed toward the predetermined second information image (221) when the plurality of second information images (220) are displayed, the predetermined second information image (221) and the predetermined second information are displayed. The other second information image (222-) near the image (221) is displayed as a moving image.
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
When the line of sight is directed toward the predetermined second information image (221) when the plurality of second information images (220) are displayed, the predetermined second information image (221) and other second information images Display (222 ~) in the same cycle,
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
Acquiring inference information capable of inferring that the driver will behave in an inappropriate manner for the information indicated by the first information image (210),
Corresponding to the inferred information when it is possible to determine that the driver performs an action that is not appropriate for the information indicated by the first information image (210) after reducing the visibility of the first information image (210) Increase the visibility of the first information image (210)
The display controller according to claim 1, which executes an instruction.
Said one or more processors (16)
After reducing the visibility of the first information image (210), the related images (213, 214) related to the first information image (210) are changed to the first information image (210) or the second information. Displaying in a second display area (102) different from the first display area (101) in which the image (220) is displayed,
The display controller according to claim 1, which executes an instruction.
The second information image (220) is an image displayed in the vicinity so as to emphasize at least one of an obstacle, a pedestrian, and another vehicle existing in the foreground (300),
The display control device according to claim 1.
A method for controlling an image display unit (11, 12) that displays an image (200) in a region overlapping a foreground (300) when viewed from a driver of a vehicle (1),
Obtaining the driver's gaze direction from the one or more I / O interfaces (14);
A first information image (210) that reduces visibility when it is determined to have been visually recognized, and a second information image (220) that does not reduce visibility more than the first information image (210) even when the line of sight is facing, Is displayed,
Including the method.
Displaying the second information image (220) as a moving image when the line of sight is directed to the second information image (220),
The method of claim 11, comprising:
When the line of sight is directed toward the predetermined second information image (221) when the plurality of second information images (220) are displayed, the predetermined second information image (221) and other second information images Display (222 ~) as a moving image,
The method of claim 11, comprising:
When the line of sight is directed toward the predetermined second information image (221) when the plurality of second information images (220) are displayed, the predetermined second information image (221) and the predetermined second information are displayed. Displaying another second information image (222-) near the image (221) as a moving image,
The method of claim 11, comprising:
When the line of sight is directed toward the predetermined second information image (221) when the plurality of second information images (220) are displayed, the predetermined second information image (221) and other second information images Displaying moving images from (222 on) at the same cycle,
The method of claim 11, comprising:
Acquiring inference information with which it is possible to infer that the driver is performing an action that is not appropriate for the information indicated by the first information image (210)
Corresponding to the inferred information when it is possible to determine that the driver performs an action that is not appropriate for the information indicated by the first information image (210) after reducing the visibility of the first information image (210) Increasing the visibility of the first information image (210)
The method of claim 11, comprising:
After reducing the visibility of the first information image (210), the related images (213, 214) related to the first information image (210) are changed to the first information image (210) or the second information. Displaying in a second display area (102) different from the first display area (101) in which the image (220) is displayed;
The method of claim 11, comprising:
A computer program comprising instructions for performing the method according to any one of claims 11 to 17.